\documentclass[10pt]{article}

\usepackage{fullpage}
\usepackage{setspace}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage[section]{placeins}
\usepackage{xcolor}
\usepackage{breakcites}
\usepackage{lineno}
\usepackage{hyphenat}





\PassOptionsToPackage{hyphens}{url}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother


\usepackage{natbib}




\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}


\usepackage{authblk}


\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[ngerman,english]{babel}



\usepackage{float}








\begin{document}

\title{Evaluating the performance of ChatGPT in clinical pharmacy}



\author[1]{Xiaoru Huang}%
\author[1]{Dannya  Estau}%
\author[1]{Xuening Liu}%
\author[1]{Yang Yu}%
\author[1]{Jiguang Qin}%
\author[1]{Zijian Li}%
\affil[1]{Affiliation not available}%


\vspace{-1em}



  
  \date{May 27, 2024}


\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup





\selectlanguage{english}
\begin{abstract}
Aim To evaluate the performance of ChatGPT in key areas of clinical
pharmacy practice, including prescription review, patient medication
education, adverse drug reaction (ADR) recognition, ADR causality
assessment, and drug counseling. Methods Quantitative and qualitative
analyses were conducted to assess the accuracy and quality of ChatGPT in
comparison to those of the clinical pharmacist. Results The results
indicate that ChatGPT is excellent in drug counseling and weak in
prescription review, patient medication education, ADR recognition, and
ADR causality. Conclusions While ChatGPT holds promise in clinical
pharmacy practice as a supplementary tool, the ability of ChatGPT to
handle complex problems needs further improvement and refinement.%
\end{abstract}%



\sloppy


\textbf{Title: Evaluating the performance of~ChatGPT~in clinical
pharmacy}

Running title: ChatGPT~in clinical pharmacy

Xiaoru Huang\textsuperscript{1,2}; Dannya Estau\textsuperscript{1,2};
Xuening Liu\textsuperscript{1,2}; Yang Yu\textsuperscript{1,2}; Jiguang
Qin\textsuperscript{1,2}; Zijian Li\textsuperscript{1,2,3*}

\textsuperscript{1}Department of Pharmacy, Peking University Third
Hospital, Beijing 100191, China

\textsuperscript{2}Department of Pharmaceutical Management and Clinical
Pharmacy, College of Pharmacy, Peking University, Beijing 100191, China

\textsuperscript{3}Department of Cardiology and Institute of Vascular
Medicine, Peking University Third Hospital, Beijing Key Laboratory of
Cardiovascular Receptors Research, Key Laboratory of Cardiovascular
Molecular Biology and Regulatory Peptides, Ministry of Health, State Key
Laboratory of Vascular Homeostasis and Remodeling, Peking University,
Beijing 100191, China

*Corresponding author:

Zijian Li

Ph.D.

Department of Pharmacy,

Peking University Third Hospital

No.49 Huayuan North Road

Haidian District, Beijing

P.R.China 100191

Email: \emph{lizijian@bjmu.edu.cn}, Tel: 86-10-87132335

\textbf{Keywords:} ChatGPT; clinical pharmacy

\textbf{Word count} :3600 approximately

\textbf{Table count:} 2

\textbf{Figure count:} 1

\textbf{What is already known about this subject?}

\begin{itemize}
\tightlist
\item
  In healthcare, ChatGPT, an AI-powered conversational large language
  model, represents both potential and challenge.
\item
  AI-powered systems have shown promise in assisting clinical
  pharmacists in their practice.
\item
  The performance of~ChatGPT~in clinical pharmacy remains unknown.
\end{itemize}

\textbf{What this study adds?}

\begin{itemize}
\tightlist
\item
  ChatGPT is excellent in drug counseling and weak in prescription
  review, patient medication education, ADR recognition, and ADR
  recognition causality assessment.
\item
  This study highlights ChatGPT's capabilities and limitations in
  clinical pharmacy practice.
\item
  ChatGPT holds promise in clinical pharmacy practice as a supplementary
  tool.
\end{itemize}

\par\null

Abstract

\textbf{Aim}

To evaluate the performance of ChatGPT in key areas of clinical pharmacy
practice, including prescription review, patient medication education,
adverse drug reaction (ADR) recognition, ADR causality assessment, and
drug counseling.

\textbf{Methods}

Quantitative and qualitative analyses were conducted to assess the
accuracy and quality of ChatGPT in comparison to those of the clinical
pharmacist.

\textbf{Results}

The results indicate that ChatGPT is excellent in drug counseling and
weak in prescription review, patient medication education, ADR
recognition, and ADR causality.

\textbf{Conclusions}

While ChatGPT holds promise in clinical pharmacy practice as a
supplementary tool, the ability of ChatGPT to handle complex problems
needs further improvement and refinement.

Main text

Introduction

In November 2022, OpenAI launched chat generative pre-trained
transformer (ChatGPT) that evolved from artificial intelligence (AI)
large language models (LLMs). It was designed to simulate human
conversation in response to prompts or questions based on the context of
input text\textsuperscript{1,2}. As a transformative and disruptive
technology, ChatGPT presents both promise and challenge for the
scientific community, particularly in healthcare \textsuperscript{3-5}.
Despite limited access to medical data, ChatGPT has achieved scores
equivalent to a third-year undergraduate medical student on the United
States Medical Licensing Examination (USMLE)\textsuperscript{6}.
Meanwhile, another study outlined ChatGPT's ability to generate formal
discharge summaries in a matter of seconds following brief patient
profile prompts\textsuperscript{7}. Moreover, ChatGPT has potential
applications in enhancing radiological decision-making, risk prediction
for various ailments, and drug discovery processes\textsuperscript{8,9}.
Beyond its technological impact, ChatGPT offers medical information to
patients and may improve personalized healthcare across various clinical
practices, including endocrinology\textsuperscript{10},
hepatology\textsuperscript{11}, cardiology\textsuperscript{12},
anti-infective medicine\textsuperscript{13}, obstetrics \&
gynecology\textsuperscript{14}, and neurosurgery\textsuperscript{15}.

Clinical pharmacists play a crucial role in ensuring patient safety,
optimizing medication therapy, and providing patient-centered
care\textsuperscript{16}. They possess the skills and knowledge required
to offer clinical pharmacy services to healthcare teams and
patients\textsuperscript{17}. In clinical pharmacy practice, AI-powered
systems have demonstrated the potential to aid clinical pharmacists, as
previously discussed\textsuperscript{18}. The development of AI-powered
apps and tools is primarily focused on prescription
review\textsuperscript{19,20}. Furthermore, the AI platform assists
clinical pharmacists in drug counseling, thus reducing costs and medical
utilization\textsuperscript{21}. AI-powered tools based on neural
network models have been developed to detect self-administration errors,
allowing for tailored patient medication education on device
techniques\textsuperscript{22}. In adverse drug reaction (ADR)
monitoring, AI employs big data from hospital information systems to
intelligently screen for adverse drug reactions/events through
electronic medical record analysis\textsuperscript{23}. With increasing
workloads and growing demands on clinical pharmacists, AI-driven support
tools like ChatGPT could potentially enhance efficiency,
decision-making, and patient care in clinical settings. However, the
accuracy, reliability, and real-world applicability of ChatGPT in
clinical settings have not been thoroughly assessed.

This study aims to evaluate the performance of ChatGPT in various
aspects of clinical pharmacy practice including prescription review,
patient medication education, ADR recognition, ADR causality assessment,
and drug counseling. Quantitative and qualitative methods were both used
to compare the accuracy and quality of ChatGPT's responses to those of
the clinical pharmacist. Our findings will offer valuable insights into
ChatGPT's strengths and limitations in clinical pharmacy practice.

\par\null

Methods

ChatGPT

``ChatGPT Mar 23 Version'' was used in this study, which is the latest
version of ChatGPT released on March 23, 2023\textsuperscript{24}.

Data source

From the practical cases of clinical pharmacists and assessment
questions for resident pharmacists, twenty-five questions were chosen to
evaluate the performance of ChatGPT in prescription review, patient
medication education, adverse drug reaction (ADR) recognition, ADR
causality assessment, and drug counseling. Each aspect had five
questions that covered the main content of the corresponding aspect. In
the prescription review, verbal descriptions of patient demographic
information, diagnoses, and medication details recorded in prescriptions
were extracted and submitted to ChatGPT. In the clinical setting,
according to the WHO-UMC system for standardized case causality
assessment\textsuperscript{25}, the evaluation process and results of
the causality assessment of ADR have been formatted as a table, termed
``the adverse drug reaction/event reporting forms''. The information
that is necessary to determine the correlation was extracted from these
forms submitted by clinical pharmacists. Then ChatGPT performed the
correlation assessment according to that information.

Quantitative and qualitative evaluation methods

Each question was separately input for ChatGPT by utilizing the ''New
Chat'' function\textsuperscript{11}.~The same set of questions was also
presented to~occupational clinical pharmacists. All the questions and
answers were documented. Five clinical pharmacy professionals reviewed
these answers and rated their accuracy~on a scale of 0 (completely
incorrect) to 10 (comprehensive)\textsuperscript{26}. The professionals
rated the answers independently and were unaware of which answers were
provided by ChatGPT or the clinical pharmacist. After reviewing all
answers, ChatGPT's capabilities and limitations in clinical pharmacy
practice were also qualitatively assessed according to the main content
of each aspect.

Statistical analysis

The score for each question was computed as the mean score evaluated by
five professional reviewers. Subsequently, the score for each aspect of
clinical pharmacy was determined by averaging the corresponding five
questions. An unpaired two-tailed Student's t-test was conducted to
compare the mean scores of ChatGPT and the clinical pharmacist, with a
p-value of \textless{}0.05 considered to be significant. GraphPad Prism
software (version 9.0) was used for these analyses. Additionally, the
interrater reliability was assessed through intraclass correlation
coefficients (ICCs) \textsuperscript{27,28}. ICC estimates and their
95\% confidence intervals were calculated using the ''irr'' package
(version 0.84.1) of R studio (powered by R version 4.2.0) based on a
single-rating, absolute-agreement, two-way random-effects model. An ICC
exceeding 0.70 is indicative of excellent
reliability\textsuperscript{29}.

Results

The quantitative evaluation of the performance of ChatGPT in clinical
pharmacy

Quantitative results indicated that ChatGPT's accuracy in answering
questions varied across different domains (Table 1). ChatGPT excelled in
drug counseling, achieving an equivalent score to the clinical
pharmacist (ChatGPT: 8.76 vs. pharmacist: 9.52, p-value=0.0596). In
other areas, ChatGPT's performance was significantly weaker, including
prescription review, patient medication education, ADR recognition, and
ADR causality assessment. As depicted in Figure 1, ChatGPT's performance
was average in patient medication education, scoring between 5 and 8,
and no response was rated as completely incorrect. In prescription
review, ADR recognition, and ADR causality assessment, scores of
ChatGPT's answers displayed high variability, with some questions
scoring 0. The evaluation results of all questions are recorded in Table
S1 and Table S2 of the \emph{Appendices} . The ICC value for the
interrater~reliability was 0.93 (95\% CI 0.90-0.96).

The qualitative analysis of the performance of ChatGPT in clinical
pharmacy

ChatGPT's capabilities and limitations in clinical pharmacy practice
were qualitatively assessed. Results are presented in Table 2. In
general, ChatGPT's responses exhibited coherent spelling and grammar.
The answers provided by the model followed a consistent pattern,
characterized by clarity, organization, and informativeness.

In terms of drug counseling, ChatGPT accurately provided comprehensive
drug-related information, answering all questions. It did not take into
consideration the patients' real-life circumstances. For instance, while
advising on reducing soy milk's impact on Eltroxin at breakfast, ChatGPT
suggested waiting at least four hours after taking Eltroxin before
consuming soy milk, which was an impractical solution in real life. It
was more feasible of the clinical pharmacist's recommendation to adjust
the medication timing to four hours after dinner. Furthermore, when the
patients expressed concerns about incorrect medication, the clinical
pharmacist paid attention to their emotions.

In prescription appropriateness review, ChatGPT successfully reviewed
prescriptions with fewer medication-related issues but struggled with
complex prescriptions. It performed poorly in identifying drug-related
problems concerning traditional Chinese medicine. For example, it was
unable to detect drug duplication by using Ganmao Qingre granule,
banlangen granule, and lanqin oral liquid, and mistook them as
antibiotics. It also failed to identify that glycyrrhizic acid
diammonium enteric-coated capsules lacked a corresponding diagnosis and
were contraindicated in patients with hypertension. ChatGPT was able to
identify inappropriate dosage and frequency, but it did not provide
specific adjustment advice.

For patient medication education, ChatGPT provided a well-organized and
detailed list of therapeutic indications, dosing regimens, and common
adverse reactions for each medication. Its answers were sometimes overly
verbose and specialized. In comparison, the clinical pharmacist used
layperson's terms to warn patients of common and life-threatening
adverse reactions. For example, when mentioned ticagrelor's side effects
of bleeding, the clinical pharmacist guided patients to monitor typical
signs including bruises, petechiae, hemoptysis, and black stools. In
addition, ChatGPT did not guide patients on necessary monitoring items
and lifestyle changes.

Regarding ADR recognition, ChatGPT accurately defined ADR but
misunderstood its connotation, as demonstrated by mistaking events
caused by non-drug and substandard drugs as ADRs. ChatGPT correctly
interpreted clinical indicators and identified simple ADRs but struggled
with complex cases. When assessing ADR causality, ChatGPT extracted key
information from cases and analyzed causality using WHO criteria. It
excelled in analyzing temporal relationships and previous knowledge but
often incorrectly assessed drug dechallenge, rechallenge, and
alternative causes, leading to misjudgments. Finally, ChatGPT tended to
classify ADR causality as possible. All the questions and answers were
reported in the \emph{Appendices} .

Discussion

Our study provides a comprehensive evaluation of ChatGPT's performance
in various aspects of clinical pharmacy practice compared to that of the
clinical pharmacist. Quantitative evaluation results revealed that
ChatGPT excelled in drug counseling, achieving a comparable score to the
clinical pharmacist. Its performance in prescription review, patient
medication education, ADR recognition, and ADR causality assessment was
significantly weaker. The qualitative analysis highlighted the key
strengths and limitations of ChatGPT in clinical settings.

Owing to training in extensive text datasets, ChatGPT can comprehend the
context and generate human-like responses\textsuperscript{1}. This
ability enables it to perform well in drug counseling and medication
education. Therefore, ChatGPT can be used as a pharmacy encyclopedia for
patients and aid clinical pharmacists in literature search and synthesis
to enhance work efficiency in the future. Several limitations of ChatGPT
should be noted. First, ChatGPT struggled with drug-related problems
involving traditional Chinese medicine, likely due to constraints in its
training data. It lacks a medical-specific database and is based on the
information before 2021. Seghier ML pointed out that ChatGPT faced
language barriers, with its non-English responses being notably
inferior\textsuperscript{30}. In particular, it is not open to the
public in China, potentially leading to informational gaps regarding
Chinese patent medicines. Second, ChatGPT often neglected patients'
real-life circumstances and lacks focus, indicating a deficiency in
situational awareness\textsuperscript{13}. This issue may be mitigated
by incorporating clinical pharmacy professional annotations and feedback
into ChatGPT's responses\textsuperscript{31}.

ChatGPT's performance in prescription review, ADR recognition, and ADR
causality assessment was suboptimal. The model tended to manage simple
cases effectively but struggled with complex ones, suggesting
difficulties in handling intricate instructions. While ChatGPT exhibited
some reasoning capabilities, they remained
limited\textsuperscript{6,14}. It misunderstood the implications of the
ADR concept and misjudged ADR causality. From these analyses, it could
be concluded that ChatGPT lacks human-like deep understanding and
adaptive application in complex real-world
situations\textsuperscript{32}.

In this study, ChatGPT's ability to provide emotional support was not
specifically assessed, but evidence suggested it may not be proactive in
offering such support. For instance, when patients expressed concerns
about incorrect medication, the clinical pharmacist addressed their
emotions first, whereas ChatGPT provided information only. In contrast,
another study investigating ChatGPT in hepatic disease management
discovered that it could supply empathetic advice to patients and
caregivers\textsuperscript{11}.

Prior to implementing ChatGPT in clinical pharmacy practice, ethical
issues must be carefully considered. Bias risks are present since AI
algorithms are often trained on biased datasets. OpenAI even
acknowledges that ChatGPT's output can perpetuate sexist
stereotypes\textsuperscript{33}. Additionally, the data used for ChatGPT
training lacks transparency, affecting reliability. ChatGPT also
occasionally generates plausible-sounding but incorrect or nonsensical
answers, called AI hallucinations, posing significant risks to patient
safety\textsuperscript{7,34}. As a result, legal frameworks and
accountability should be established for these
errors\textsuperscript{35}. Italy has banned ChatGPT due to privacy
concerns, necessitating attention towards data governance and
privacy\textsuperscript{7,36}.

This study has several limitations. Firstly, the evaluation focused on
specific aspects of clinical pharmacy practice, not encompassing all
potential applications. Secondly, the limited number of questions and
prompts may not fully capture ChatGPT's capabilities and limitations.
Lastly, the GPT-3.5 model instead of the more advanced GPT-4 model was
employed in this study. However, this is due to the GPT-3.5 model's
unrestricted access to the public. The GPT-4 model is only available for
paid ChatGPT Plus subscribers on a limited basis, thereby reducing its
accessibility to patients.

In conclusion, ChatGPT shows promise in specific domains such as drug
counseling, but its overall performance across various aspects of
clinical pharmacy practice is significantly weaker compared to the
clinical pharmacist. These findings indicate that ChatGPT has the
potential as a supplementary tool in clinical settings. Further
enhancements and refinements to the ChatGPT system, particularly in
expanding medicine-specific datasets and augmenting capabilities for
advanced reasoning and complex instructions, will be crucial for
optimizing its utility in clinical pharmacy practice.



\textbf{Acknowledgments}

The access to ChatGPT was obtained by Lishu Zhang in Dallas, TX, USA.

\textbf{Conflict of interest statement}

All authors declare no conflicts of interest.

\textbf{Funding information}

This study was supported by (1) National Natural Science Foundation of
China (NSFC Grant Nos. 81820108031, U21A20336, and 91939301); (2)
National Key Research and Development Program of China (Grant No.
2022YFC3602400); (3) Beijing Municipal Natural Science Foundation (Grant
No. 7222218, China); (4) Medical research management/basic and clinical
research unit of metabolic cardiovascular diseases, Chinese Academy of
Medical Sciences (Grant No. 2021RU003).

\textbf{Data availability statement}

All data generated and analyzed during this study are included in this
published article.

\textbf{Author contributions}

Zijian Li and Xiaoru Huang designed the research. Xiaoru Huang carried
out the investigation and performed data analysis. Dannya Estau, Xuening
Liu, Yang Yu, and Jiguang Qin participated in the evaluation part.
Xiaoru Huang wrote the manuscript. Zijian Li revised the manuscript. All
of the authors have read and approved the final manuscript.



\textbf{Reference}

1. OpenAI. OpenAI: Models GPT-3. Accessed
2023-4-12,\emph{https://beta.openai.com/docs/models}

2. Brown TB, Mann B, Ryder N, et al. Language Models are Few-Shot
Learners. \emph{ArXiv} . 2020;abs/2005.14165

3. Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the
Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical
and Research Scenarios. \emph{J Med Syst} . Mar 4 2023;47(1):33.
doi:10.1007/s10916-023-01925-4

4. DiGiorgio AM, Ehrenfeld JM. Artificial Intelligence in Medicine \&
ChatGPT: De-Tether the Physician. \emph{J Med Syst} . Mar 4
2023;47(1):32. doi:10.1007/s10916-023-01926-3

5. Sallam M. ChatGPT Utility in Healthcare Education, Research, and
Practice: Systematic Review on the Promising Perspectives and Valid
Concerns. \emph{Healthcare (Basel)} . Mar 19
2023;11(6)doi:10.3390/healthcare11060887

6. Gilson A, Safranek CW, Huang T, et al. How Does ChatGPT Perform on
the United States Medical Licensing Examination? The Implications of
Large Language Models for Medical Education and Knowledge
Assessment.\emph{JMIR Med Educ} . Feb 8 2023;9:e45312. doi:10.2196/45312

7. Patel SB, Lam K. ChatGPT: the future of discharge
summaries?\emph{Lancet Digit Health} . Mar 2023;5(3):e107-e108.
doi:10.1016/s2589-7500(23)00021-3

8. Holzinger A, Keiblinger K, Holub P, Zatloukal K, M\selectlanguage{ngerman}üller H. AI for
life: Trends in artificial intelligence for biotechnology. \emph{N
Biotechnol} . May 25 2023;74:16-24. doi:10.1016/j.nbt.2023.02.001

9. Rao A, Kim J, Kamineni M, Pang M, Lie W, Succi MD. Evaluating ChatGPT
as an Adjunct for Radiologic Decision-Making. \emph{medRxiv} . Feb 7
2023;doi:10.1101/2023.02.02.23285399

10. Sng GGR, Tung JYM, Lim DYZ, Bee YM. Potential and Pitfalls of
ChatGPT and Natural-Language Artificial Intelligence Models for Diabetes
Education. \emph{Diabetes Care} . Mar 15 2023;doi:10.2337/dc23-0197

11. Yeo YH, Samaan JS, Ng WH, et al. Assessing the performance of
ChatGPT in answering questions regarding cirrhosis and hepatocellular
carcinoma. \emph{Clin Mol Hepatol} . Mar 22
2023;doi:10.3350/cmh.2023.0089

12. Moons P, Van Bulck L. ChatGPT: Can artificial intelligence language
models be of value for cardiovascular nurses and allied health
professionals. \emph{Eur J Cardiovasc Nurs} . Feb 8
2023;doi:10.1093/eurjcn/zvad022

13. Howard A, Hope W, Gerada A. ChatGPT and antimicrobial advice: the
end of the consulting infection doctor? \emph{Lancet Infect Dis} . Apr
2023;23(4):405-406. doi:10.1016/s1473-3099(23)00113-5

14. Grünebaum A, Chervenak J, Pollet SL, Katz A, Chervenak FA. The
exciting potential for ChatGPT in obstetrics and gynecology. \emph{Am J
Obstet Gynecol} . Mar 15 2023;doi:10.1016/j.ajog.2023.03.009

15. D'Amico RS, White TG, Shah HA, Langer DJ. I Asked a ChatGPT to Write
an Editorial About How We Can Incorporate Chatbots Into Neurosurgical
Research and Patient Care\ldots{}. \emph{Neurosurgery} . Apr 1
2023;92(4):663-664. doi:10.1227/neu.0000000000002414

16. Hepler CD. Clinical pharmacy, pharmaceutical care, and the quality
of drug therapy. \emph{Pharmacotherapy} . Nov 2004;24(11):1491-8.
doi:10.1592/phco.24.16.1491.50950

17. Dunn SP, Birtcher KK, Beavers CJ, et al. The role of the clinical
pharmacist in the care of patients with cardiovascular disease. \emph{J
Am Coll Cardiol} . Nov 10 2015;66(19):2129-2139.
doi:10.1016/j.jacc.2015.09.025

18. Ranchon F, Chanoine S, Lambert-Lacroix S, Bosson JL, Moreau-Gaudry
A, Bedouch P. Development of artificial intelligence powered apps and
tools for clinical pharmacy services: A systematic review. \emph{Int J
Med Inform} . Apr 2023;172:104983. doi:10.1016/j.ijmedinf.2022.104983

19. Hogue SC, Chen F, Brassard G, et al. Pharmacists' perceptions of a
machine learning model for the identification of atypical medication
orders. \emph{J Am Med Inform Assoc} . Jul 30 2021;28(8):1712-1718.
doi:10.1093/jamia/ocab071

20. Balestra M, Chen J, Iturrate E, Aphinyanaphongs Y, Nov O. Predicting
inpatient pharmacy order interventions using provider action
data.\emph{JAMIA Open} . Jul 2021;4(3):ooab083.
doi:10.1093/jamiaopen/ooab083

21. Kessler S, Desai M, McConnell W, et al. Economic and utilization
outcomes of medication management at a large Medicaid plan with disease
management pharmacists using a novel artificial intelligence platform
from 2018 to 2019: a retrospective observational study using regression
methods. \emph{J Manag Care Spec Pharm} . Sep 2021;27(9):1186-1196.
doi:10.18553/jmcp.2021.21036

22. Zhao M, Hoti K, Wang H, Raghu A, Katabi D. Assessment of medication
self-administration using artificial intelligence. \emph{Nat Med} . Apr
2021;27(4):727-735. doi:10.1038/s41591-021-01273-1

23. Ahmad A, Patel I, Balkrishnan R, Mohanta GP, Manna PK. An evaluation
of knowledge, attitude and practice of Indian pharmacists towards
adverse drug reaction reporting: A pilot study. \emph{Perspect Clin Res}
. Oct 2013;4(4):204-10. doi:10.4103/2229-3485.120168

24. Natalie. ChatGPT --- Release Notes. Accessed
2023-4-15,\emph{https://help.openai.com/en/articles/6825453-chatgpt-release-notes}

25. Organization WH. The Use of the WHO-UMC System for Standardised Case
Causality Assessment. Accessed
2023-4-8.\emph{https://www.who.int/publications/m/item/WHO-causality-assessment}

26. Dowling M, Lucey B. ChatGPT for (Finance) research: The Bananarama
Conjecture. \emph{Finance Research Letters} . 2023/05/01/
2023;53:103662. doi:\emph{https://doi.org/10.1016/j.frl.2023.103662}

27. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass
Correlation Coefficients for Reliability Research. \emph{J Chiropr Med}
. Jun 2016;15(2):155-63. doi:10.1016/j.jcm.2016.02.012

28. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing
rater reliability. \emph{Psychol Bull} . Mar 1979;86(2):420-8.
doi:10.1037//0033-2909.86.2.420

29. Wang-Hsu E, Smith SS. Interrater and Test-Retest Reliability and
Minimal Detectable Change of the Balance Evaluation Systems Test
(BESTest) and Subsystems With Community-Dwelling Older Adults. \emph{J
Geriatr Phys Ther} . Jul/Sep 2018;41(3):173-179.
doi:10.1519/jpt.0000000000000117

30. Seghier ML. ChatGPT: not all languages are equal. \emph{Nature} .
Mar 2023;615(7951):216. doi:10.1038/d41586-023-00680-3

31. OpenAI. GPT-4 Technical Report. \emph{ArXiv} . 2023;abs/2303.08774

32. Homolak J. Opportunities and risks of ChatGPT in medicine, science,
and academic publishing: a modern Promethean dilemma. \emph{Croat Med J}
. Feb 28 2023;64(1):1-3. doi:10.3325/cmj.2023.64.1

33. The Lancet Digital H. ChatGPT: friend or foe? \emph{Lancet Digit
Health} . Mar 2023;5(3):e102. doi:10.1016/s2589-7500(23)00023-7

34. Ahn C. Exploring ChatGPT for information of cardiopulmonary
resuscitation. \emph{Resuscitation} . Apr 2023;185:109729.
doi:10.1016/j.resuscitation.2023.109729

35. Biswas S. ChatGPT and the Future of Medical Writing.\emph{Radiology}
. Apr 2023;307(2):e223312. doi:10.1148/radiol.223312

36. Mijwil MM, Aljanabi M, Ali A. ChatGPT: Exploring the Role of
Cybersecurity in the Protection of Medical
Information.\emph{Mesopotamian Journal of Cyber Security} . 2023;

\par\null

\textbf{Table 1} The comparison between ChatGPT and the clinical
pharmacist in clinical pharmacy practice.\selectlanguage{english}
\begin{longtable}[]{@{}lllll@{}}
\toprule
clinical pharmacy practice & Responder & Mean scores & SD &
p-value\tabularnewline
\midrule
\endhead
drug counseling & ChatGPT & 8.76 & 0.73 & 0.0596\tabularnewline
& Pharmacist & 9.52 & 0.27 &\tabularnewline
prescription appropriateness review & ChatGPT & 5.00 & 2.94 &
0.0052\tabularnewline
& Pharmacist & 10.00 & 0.00 &\tabularnewline
patient medication education & ChatGPT & 6.16 & 0.93 &
\textless{}0.0001\tabularnewline
& Pharmacist & 9.36 & 0.36 &\tabularnewline
ADR recognition & ChatGPT & 4.96 & 4.75 & 0.0495\tabularnewline
& Pharmacist & 9.88 & 0.18 &\tabularnewline
ADR causality assessment & ChatGPT & 4.16 & 4.80 & 0.0263\tabularnewline
& Pharmacist & 10.00 & 0.00 &\tabularnewline
\bottomrule
\end{longtable}

The mean score for each aspect was the average of the respective five
questions. 0 indicates thorough inaccuracy, whereas 10 represents
complete comprehensiveness. The p-value of \textless{}0.05 is considered
to be significant. SD=Standard Deviation, ADR=adverse drug reaction.

\par\null

\textbf{Table 2} ChatGPT's capabilities and limitations in clinical
pharmacy practice.\selectlanguage{english}
\begin{longtable}[]{@{}llll@{}}
\toprule
\begin{minipage}[b]{0.22\columnwidth}\raggedright\strut
Clinical pharmacy practice\strut
\end{minipage} & \begin{minipage}[b]{0.22\columnwidth}\raggedright\strut
Main content\strut
\end{minipage} & \begin{minipage}[b]{0.22\columnwidth}\raggedright\strut
ChatGPT's capabilities\strut
\end{minipage} & \begin{minipage}[b]{0.22\columnwidth}\raggedright\strut
ChatGPT's limitations\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.22\columnwidth}\raggedright\strut
drug counseling\strut
\end{minipage} & \begin{minipage}[t]{0.22\columnwidth}\raggedright\strut
Providing information, and advice regarding various aspects of
pharmaceuticals, including drug names, administration, dosage, adverse
effects, drug interactions, precautions, and medication use in special
populations.\strut
\end{minipage} & \begin{minipage}[t]{0.22\columnwidth}\raggedright\strut
Efficiently and precisely delivering comprehensive drug-related
information.\strut
\end{minipage} & \begin{minipage}[t]{0.22\columnwidth}\raggedright\strut
Insufficient integration of patients' real-life circumstances.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
prescription appropriateness review\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
Evaluating the appropriateness of indications, drug selection,
formulation, administration routes, dosage, frequency, and combination,
as well as assessing contraindications, drug duplication,
incompatibilities, and interactions.\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
1. Assessing prescriptions with minimal medication-related concerns; 2.
Identifying prescription issues, including absent indications,
unsuitable administration routes, improper dosage, incorrect frequency,
inappropriate drug combinations, and the presence of incompatibilities
and interactions.\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
1. Missing some issues in complex prescriptions; 2. Inability to detect
issues associated with traditional Chinese medicine; 3. Insufficient
guidance on proper dosage and frequency.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
patient medication education\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
Advising patients on rational drug use, including dosage, timing,
adverse reactions, required monitoring, and lifestyle adjustments for
specific conditions.\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
Offering a comprehensive overview of therapeutic indications, dosing
guidelines, and common adverse effects for each medication.\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
1. Excessively verbose and unfocused presentation; 2. Insufficient
integration of patients' real-life situations; 3. Inadequate monitoring
and lifestyle change guidance; 4. Overly specialized responses.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
ADR recognition\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
Comprehending the concept of ADRs and identifying ADRs.\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
Recognizing simple ADRs\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
1. Incorrect identification of ADRs; 2. Partial detection of complex
ADRs.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
ADR causality assessment\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
Categorizing ADRs based on causality magnitude: certain,
probable/likely, possible, unlikely, and unassessable\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
1. Systematic analysis of ADR causality: temporal relationship, prior
knowledge, drug dechallenge, rechallenge, and alternative causes; 2.
Accurate assessment of temporal relationship and prior knowledge.\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
1. Inaccurate evaluation of drug dechallenge, rechallenge, and
alternative causes; 2. Propensity to classify ADR causality as
possible.\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

ADR=adverse drug reaction.

\textbf{Figure legends}

\textbf{Figure 1.} Scores of ChatGPT and the clinical pharmacist in
clinical pharmacy practice.

The triangles and round dots represent the scores for answers provided
by ChatGPT and the clinical pharmacist, respectively. The height of the
red column represents the average score of the five questions answered
by ChatGPT, and the height of the blue column represents the average
score of the five questions answered by the clinical pharmacist. Scores
range from 0, signifying thorough inaccuracy, to 10, symbolizing
complete comprehensiveness. ADR=adverse drug reaction.\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/Figure-1/Figure-1}

\end{center}
\end{figure}

\selectlanguage{english}
\FloatBarrier
\end{document}

