011: Evaluation of Prompt Engineering Strategies for Pharmacokinetic Data Analysis With the ChatGPT Large Language Model

Statement of Purpose, Innovation or Hypothesis: To systematically assess the ChatGPT large language model on diverse tasks relevant to pharmacokinetic data analysis.

Description of Methods & Materials: ChatGPT was evaluated with prototypical tasks related to report writing, code generation, non-compartmental analysis, and pharmacokinetic word problems. The writing task consisted of writing an introduction for this paper from a draft title. The coding tasks consisted of generating R code for semi-logarithmic graphing of concentration-time profiles and calculating area under the curve and area under the moment curve from time zero to infinity. Pharmacokinetics word problems on single intravenous, extravascular bolus and multiple dosing were taken from a pharmacokinetics textbook. Chain-of-thought and problem separation were assessed as prompt engineering strategies when errors occurred.

Data & Results: ChatGPT has the potential to become a powerful productivity tool for writing, knowledge encapsulation, and coding tasks in pharmacokinetic data analysis. The poor accuracy of ChatGPT in numerical calculations require resolution before it can be reliably used for PK and pharmacometrics data analysis.

Interpretation, Conclusion or Significance: Our results suggest that ChatGPT could be a useful productivity aid for writing, knowledge encapsulation and programming tasks. However, the high rate of errors in arithmetic calculations could limit its utility for more complex tasks and diverse data analysis scenarios in PK data analysis and pharmacometrics. If the limitations are overcome, ChatGPT could become a valuable tool for automating all aspects of PK and pharmacometric data analysis. Prompt protocol research and rigorous performance evaluation, and validation studies on benchmark data sets will be required to build the confidence and certainty expected in the pharmaceutical regulatory environment.

Disclosures: Euibeom Shin has no conflicts. Dr. Murali Ramanathan received research funding from the National Multiple Sclerosis Society, Department of Defense, National Science Fdn and National Inst of Neurological Diseases and Stroke. He receives royalty from a self-published textbook.

Citations/References: OpenAI (2023) ChatGPT (June 26 version) Large language model

White J, Fu Q, Hays S, Sandborn M, Olea C, Gilbert H, Elnashar A, Spencer-Smith J, Schmidt DC (2023) A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv:2302:11382

Rowland M, Tozer TN (1995) Clinical pharmacokinetics: concepts and applications, 3rd edn. Williams & Wilkins, Baltimore