What this study adds:

Introduction

Medications offer significant health benefits; however, their use can also result in patient harm [1-2]. Medication harm has negative clinical and economic outcomes, contributing to hospitalisations, increased length of stay (LOS), morbidity and mortality [3]. Annual costs are estimated at USD $42 billion internationally, and $1.2 billion in Australia [4-5]. The incidence of inpatient medication harm ranges from four to 14%, of which up to 50% of events are thought to be preventable [6-7]. Medication harm can cause physical, cognitive and emotional impairment to the patient [8]. The Australian healthcare system has introduced penalties for hospital-acquired complications (HACs) due to medications [9].
Two effective strategies to reduce medication harm include pharmacist-led medication reconciliation and a clinical evaluation of medications [10]. These services are resource intensive [10] and fiscal constraints, mean that pharmacists cannot provide extensive services to every inpatient. A suggested solution is to identify high-risk patients and prioritise them for early and targeted medication management [11-12]. However, there is currently no known standardised, evidence-based methods implemented in clinical practice.
An emerging approach are risk prediction models, which use statistical algorithms to quantify the probability that a patient will experience medication harm [13]. Predictive analytics plays a central role in medicine, by using influential factors to predict outcomes and facilitate timely, patient-specific interventions. Examples include the Framingham risk score for cardiovascular risk prediction [14], the CHA2DS2‐VASc score for predicting risk of stroke [15], and the HAS-BLED score for anticoagulant-related bleeding risk, in patients with atrial fibrillation [16].
Several medication risk prediction models have been developed and validated for use in the hospital setting, none have been implemented in practice, and all have methodological and/or reporting limitations [17]. Models exist for predicting risk of medication errors, medication-related problems, or actual harm, with studies predominantly including small cohorts of older adults, in European hospital settings [17]. Given that the majority of medication errors do not have clinically significant consequences, models that do not predict patient harm are limited in their potential to guide clinical decisions [18]. To date, no models that predict medication harm have been developed or evaluated for use in an Australian inpatient setting. As population characteristics have an important role in risk assessment, this study aimed to develop, validate, and report a robust model, predicting actual medication harm, from data obtained in an acute medical and rehabilitation setting.

Aim

To develop and internally validate a risk prediction model, to identify and prioritise hospitalised patients, at risk of medication harm.

Methods

Study participants and design

The Adverse Inpatient Medication Event model (AIME) study was a retrospective cohort study, including adult patients, sequentially admitted over six months, from the 1st of July 2017 to 31st of December 2017, to the general medical and/or the Geriatric Assessment and Rehabilitation Unit, of a quaternary teaching hospital in Australia. Patients admitted for less than 24 hours were excluded. Ethical approval was obtained from the Human Research Ethics Committee at Metro Health South (HREC/17/QPAH/353).

Outcome measure

The outcome of interest in this study was inpatient medication harm, as defined by Edwards and Aronson [19], and later updated by Aronson and Ferner [20]; “An appreciably harmful or unpleasant reaction, resulting from an intervention related to the use of a medicinal product; adverse effects usually predict hazard from future administration and warrant prevention, or specific treatment, or alteration of the dosage regimen, or withdrawal of the product.” This included any negative inpatient outcome or injury resulting in clinical signs, symptoms or physiological abnormalities, related to the use of a medication during hospitalisation.
Medication harm was identified from the medical records of patients coded as having had an adverse medication event (Y-codes), using the Tenth Edition of The International Classification of Disease Codes, Australian Modified (ICD-10 AM). Additionally, the hospital incident reporting database (RISKMAN) was reviewed to identify cases where a medication incident had resulted in patient harm.
All medical records of any patients flagged with a potential medication event were comprehensively reviewed by an investigator (NF), a senior clinical pharmacist, to evaluate the medication harm events and establish causality, severity and preventability. Causality was ascertained using the Hallas criteria [21]. Definite, probable or possible events were included. Events classified as unlikely were excluded. Severity of harm was determined using the definitions described by Morimoto et al. [22]. Preventability was assessed as per Hallas et al. [21], and using the Schumock and Thornton criteria [23]. Where there was uncertainty in rating an event, this was discussed with senior colleagues. A randomly selected 30 cases were also assessed for causality, severity and preventability by two independent experts; a senior hospital pharmacist and a clinical pharmacologist. Where event ratings differed between investigators, these were discussed to resolve discrepancies, and reach consensus.

Selection and definitions of variables for model development

Variables for model development were identified using two methods. First, a systematic review of the literature was undertaken to identify established risk factors for medication harm, and significant variables in existing risk models [17]. Second, hospital pharmacist focus groups and a national survey of Australian clinical pharmacists helped identify key criteria routinely used to prioritise patients at high-risk of medication harm [24]. To ensure clinical relevance and minimise noise, we selected clinically useful variables that could be quantified from digital sources, during a patient’s hospitalisation. Some continuous variables, such as International Normalised Ratio (INR), were categorised according to clinically meaningful risk thresholds, identified from hospital guidelines and pharmacist prioritisation criteria [24].
The 68 selected variables were categorised into patient demographics, social risk factors, hospital utilisation data, medications used and pathology results. Variables were extracted from the hospital’s electronic medical records (EMR) and presented in reports developed and validated for the purpose of this study, in collaboration with the hospital’s Informatics team. Medications were grouped guided by the Australian Medicines Handbook 2017 Edition [25]. Medications at admission were defined as the number of distinct medications, administered within the first 24 hours of hospitalisation.
Comorbidities were defined as per ICD-10 AM codes, which were used to group conditions. Laboratory tests were those measured within the first 24 hours of admission. These included full blood count, renal function, serum electrolyte levels, serum blood glucose levels and coagulation studies.
Renal function at admission was the estimated Glomerular Filtration Rate (eGFR) calculated using the Chronic Kidney Disease Epidemiology Collaborative (CKD-EPI) equation [26]. This was analysed as both a continuous variable and categorised at clinically informative thresholds [24]. Low serum sodium at admission was defined and categorised as sodium levels ≤ 125 mmol/L. INR and aPTT were also categorised into two groups, with supratherapeutic levels defined as greater than 3, and greater than 100 seconds, respectively [24]. Thresholds were informed from a previous study [24].

Sample size

Sample size was estimated using an event per variable (EPV) ratio of 10. This was based on a medication harm rate of 7% using local data [6], and the estimated inclusion of up to 14 variables from univariable pre-selection, as guided by prior prognostic model development studies [17].

Data handling and modelling methods

Before modelling, the distributions of variables were examined using graphs and descriptive statistics. Model development involved two stages. Univariable analysis was used to identify significant relationships with the outcome; chi-squared tests or regression analyses determined which variables should be included in a multivariable logistic regression analysis. Continuous variables were first analysed as continuous and later categorised using thresholds previously described [24]. Modelling was undertaken with both continuous and dichotomised variables, and where categorisation improved model fit, the dichotomised variable was used. Log transformation was applied where it optimised model fit.

Polynomials and interaction terms

Polynomials were examined for continuous variables with potential curvilinearity. Clinically plausible interaction terms were also examined and included potential interactions between gender with heart disease, and diabetes with glucose levels.

Missing values

The number of missing values was identified for each variable and patterns of missingness were examined. There were ten patients with missing data for inpatient medications and so complete case analysis was used. Missing laboratory test results (except INR and aPTT) were imputed using the mean value. Given the small number of missing values for the majority of variables, single imputation was considered sufficient to obtain reasonable predictions [27].
There were a larger proportion of missing values for INR and aPTT, and missingness was not deemed to be random. As INR and aPTT are not routinely measured for all patients, only patients who clinically required a test had values reported. Therefore, INR and aPTT were categorised into two groups and the most clinically plausible values were used to impute variables. Patients without an INR test during hospitalisation were categorised into the lower INR group (≤ 3). Patients without an aPTT test were categorised into the lower aPTT group (≤ 100 seconds). A Clinical Pharmacologist and Biostatistician independent of this study were consulted regarding the imputation of these variables.

Selection of variables in multivariable analysis

Binomial logistic regression analysis was undertaken to identify the optimal combination of the most influential predictors in the AIME model. The alpha level to determine appropriate variable inclusion was set at p ≤ 0.10 [28]. Using the above alpha level, significant variables from univariable analysis were included in the multivariable analysis.
The final model was selected using backward elimination as it is preferred to forward selection in predictive modelling [29]. Analysis was undertaken using R statistical software®version 5.3.1 [30], using both a manual method and also the automated stepAIC function from the R package ‘MASS’ [31]. The final model was determined by minimising the Akaike Information Criterion (AIC), and retaining variables with p-values < 0.10, whilst ensuring a clinically plausible risk model.

Model Performance

Model performance was evaluated using three measures; discrimination, calibration, and variance, as recommended in the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement [13, 32]. Model discrimination was measured using area under the receiver operative characteristic (AuROC) curve, where an AuROC of 1 is considered a perfect model and 0.5 unsatisfactory [27].
Model calibration was assessed using plots and formal statistical testing, using the Hosmer-Lemeshow goodness of fit (GOF) test. The Nagelkerke R² was calculated as a measure of variability explained by the model [27] and the Youden’s index was used to identify the optimal threshold.

Model validation

Internal validation of the final model was undertaken using 10-fold cross validation with 200 replications. The performance measures for the optimism corrected, internally validated model, were reported.

Clinical usefulness of the AIME model

The clinical usefulness of the AIME model was evaluated using decision curve analysis (DCA). DCA adds to performance measures to identify the potential net benefits and harms of using a model in practice, using values of true positives and false positives, at different thresholds. Decision curves were constructed for the AIME model, and a simpler ‘Polypharmacy’ model. The Polypharmacy model, based on a common approach of ‘number of medications’ for patient prioritisation [33], was informed by the average number of medications administered to study participants within the first 24 hours of hospitalisation. The standardised net benefit (equation 1.1) of applying the models were calculated at different threshold probabilities to create the decision curves. The models demonstrated clinical usefulness where they had positive net benefit values [34]. In this study, the term ‘clinical usefulness’ was used to show where a model had greater net benefit than a “treat-all” (pharmacist intervention for all patients), or “treat-none” (no patient receives pharmacist intervention) approach. Using guidance from the literature we identified a probability threshold of greater than 5% risk as a suitable threshold for intervention [33]. This was based on consensus by an expert panel, with the assumption here being that it would be unlikely for a pharmacist to prioritise a patient for urgent review who had a probability of risk below this threshold. This was then compared with our findings from the decision curve.
Equation 1.: Net Benefit Calculation for Decision Curve [34]
\begin{equation} Net\ Benefit=\frac{\text{TruePositives}}{n}-\frac{\text{FalsePositives}}{n}\ (\frac{\text{Pt}}{1-Pt})\nonumber \\ \end{equation}
Key: Pt is threshold probability

Results

Study participants

A total of 1982 patients were included in the study. The median (IQR) patient age was 74 (62-86) years, and 883 (45%) of patients were males. Key baseline characteristics of the participants are reported in Tables 1 and 2.
Insert Table 1
Insert Table 2

Study outcome (medication harm)

A total of 136 (7%) inpatients experienced one or more medication harm events. Some patients experienced multiple inpatient events, resulting in a total of 155 events. The causality assessment classified 20% of events as definite, 45% as probable, and 35% as possible. Events classified as unlikely to be medication related were excluded. Severity assessment showed that 12% of events were significant (defined as an adverse reaction that does not require a change in therapy, but may need supportive treatment), 70% were serious (requiring dose reduction or therapy cessation, some requiring additional therapeutic measures or specific treatment, and/or a minor increase in LOS). Eighteen percent of events were severe and potentially life-threatening (leading to severe or permanent harm, and causing a substantial increase in LOS [defined as greater than two days]). Examples of severe events included Heparin induced thrombocytopenia and thrombosis (HITT) resulting in pulmonary embolism, insulin related hypoglycaemia leading to seizure, anaphylaxis due to antibiotics, and gastrointestinal bleeding due to heparin. There were no fatal events.
Preventability assessment classified 28% of cases as definitely preventable, 31% as possibly preventable, and 41% as not preventable. Events classified as preventable predominantly comprised of incorrect medication choice, incorrect dose, inappropriate combinations of agents and/or inadequate patient review, deprescribing and surveillance, as well as ‘lack of knowledge’ errors. Greater focus on rationalisation of medications, for example older adults on multiple centrally acting or cardiovascular agents, education on best prescribing practices, and specialist involvement were identified by reviewers as strategies that may have mitigated events (for example, inappropriate conversion of opioids).
A total of 82% of events were classified as type A reactions (common, predictable and often dose-related), and 18% as type B reactions (uncommon, unpredictable and often immune mediated reactions)[19]. Adverse medication events, summarised by medication class and patient reactions are reported in Table 3. The top medication class implicated in medication harm were cardiovascular agents (in particular, beta-receptor blocking agents, digoxin, and diuretics).
Insert Table 3

Univariable preselection of variables

The twenty-one variables which had a statistically significant relationship with medication harm, at an alpha level of 10% (p ≤ 0.1), are shown in Table 4. Renal function was not statistically significant (p = 0.67), but was included, given its clinical relevance and presence in prior risk models. The inclusion of 22 variables with 136 harm events gave an EPV of approximately 6.
Insert Table 4

Multivariable Logistic Regression Analysis

Length of stay (LOS) was log transformed and INR, serum sodium and number of medications were dichotomised. The final model consisted of 10 variables (Table 5). The variables in the final prediction model (and reference levels for coding) are shown in Table 5:
Insert Table 5

Model Performance

The AuROC curve for the model prior to cross validation was 0.72, 95% CI 0.67 – 0.76. The cross-validated model performance reduced to 0.70, 95% CI 0.65 – 0.74 (Figure 1).
Insert Figure 1
The AIME model was well calibrated, with a Hosmer-Lemeshow statistic of p = 0.53. The regression equation to calculate the probability of medication harm for hospitalised patients is shown as Equation 1.2. Youden’s index was used to identify the optimal probability threshold for obtaining a balance between sensitivity and specificity. At a threshold probability of 0.05 for identifying high-risk patients, the model had a sensitivity of 77% and specificity of 58%.
\(p=\ \frac{1}{1+\ e^{-\ (-4.56+\beta_{1}X_{1}+\beta_{2}X_{2}+\ \ldots\ +{\beta_{10}X}_{10})}}\)Equation 1.: AIME model
where,