01research projectOngoing
Evaluating LLM Antidepressant-Prescribing Predictions Against Real EHR Data
Author · Shikhar Sisodia
Undergraduate research at Santa Clara University testing whether three open models reproduce real antidepressant prescribing distributions. Model output is mapped into six drug classes and compared against OCHIN electronic health record data across demographic subgroups.
model output vs. real prescribing distributions
450 groups · 25 runs each · 3 prompt tiers
- LLaMA 3.1 8B
- Mistral 7B
- Mistral-Nemo 12B
- LLaMA 3.1 8B
- top-1 90.4% · mae 0.159 · jsd 0.427
- Mistral 7B
- top-1 90.4% · mae 0.169 · jsd 0.467
- Mistral-Nemo 12B
- top-1 90.2% · mae 0.159 · jsd 0.429
Plotted with better upward in every panel. The top two tie on Top-1 and swap order on MAE and JSD — the lines cross — so no single model leads outright. Bias is analysed separately across race, gender, federal poverty level.
Spearman correlation · Top-1 hit rate · MAE · Jensen-Shannon divergence · benchmarked against OCHIN electronic health records
This evaluates prediction of prescribing patterns in aggregate. It is not medical advice and not a prescribing system.