case study
LLM Clinical Evaluation
Testing whether LLMs replicate real-world antidepressant-prescribing patterns across demographics.
overview
An SCU research project evaluating LLaMA 3.1 8B, Mistral 7B, and Mistral-Nemo 12B against real electronic health record (EHR) data on antidepressant prescribing, checking whether model outputs match real clinical patterns across demographic subgroups.
what was built
Prompted three open-weight models across 450 demographic-condition groups, 25 runs each, using a three-tier prompt framework, then compared outputs to real EHR prescribing data using correlation and Jensen-Shannon divergence.
outcome
Per the project's own reporting: roughly 90% top-drug-class accuracy, mean absolute error under 0.17, and Jensen-Shannon divergence around 0.43–0.47 versus real prescribing data. Tied to an ORCID research identity.