Weekly Paper Review: Can routine clinical data be used to predict which
patients with epilepsy will experience a psychiatric adverse effect
from levetiracetam?
It is another week of paper review. This week I read the paper titled: “Prediction Tools for Psychiatric Adverse Effects After Levetiracetam Prescription” by Colin B. Josephson, MD, MSc, FRCPC et al. (2019).
Also, I have reviewed another paper by these authors some weeks ago. I think you should read this review too. Read the review here.
If you are ready, I am ready. Let’s examine the beautiful work put together by these authors.
Preamble
Levetiracetam is a commonly used antiepileptic drug (AED). However, psychiatric adverse effects or reactions usually result from the use of this drug. This may lead to discontinuation of this treatment.
The idea here is that, if a physician is able to identify a patient that will develop this psychiatric adverse effects, then approaches can be put in place that will enhance personalized attention to such patient.
In addition, the goal of building prediction models is usually to build models that can generalize and be used to guide prescription in clinical practice.
Therefore, the main objective of the study was to derive prediction models using a data-driven and clinically-informed approach that can be used to estimate the risk of psychiatric adverse effects from levetiracetam use.
To achieve this objective, they built two prediction models; one for the overall population, and the other model for those without a history of psychiatric sign, symptoms, or disorders during the study period.
This looks to be a comprehensive and exhaustive approach to achieve this aim. I think this should be able to generalize for an independent population.
Let’s see the methods and some of the results derived from the analysis of this study.
Methods
The Health Improvement Network (THIN)
The Health Improvement Network (THIN) database is an electronic medical record (EMR) data platform based in the United Kingdom that consists of anonymized general practice (GP) patient records.
This data set contains GP records for a representative sample of approximately 5% of the national population. Medical events are coded using Read codes, and prescription data are classified according to the British National Formulary.
This study was performed using data from THIN version 1205, which included data between January 1, 2000 and May, 31, 2012.
Study Population
The authors used a retrospective open cohort design for the study.
Retrospective cohort studies are a type of observational research in which the investigator(s) looks back in time at archived or self-report data to examine whether the risk of disease was different between exposed and non-exposed patients.
In order to enhance high-quality data collection (that is to reduce the chances of wrongly classifying prevalent epilepsy cases as incident), the authors required that all patients were active after the Acceptable Mortality Reporting date (the year in which mortality reporting was deemed complete and accurate for each practice).
The authors used a 5-year washout period from first medical code after the Acceptable Mortality Reporting date to exclude patients with prevalent epilepsy. In addition, patients could not receive any codes for epilepsy during this period. All patients 18 years and above whose practice (first medical code entry) met the date when mortality reporting was considered complete were included in the analysis.
Exposure & Outcome
A prescription code for levetiracetam represented the exposure. To define the outcome of interest, a consensus-driven process by two of the authors who were experts in epilepsy was employed. The outcome of interest is a Read or Multilex therapeutic code for any psychiatric sign, symptom, or disorder.
Selection of Predictor Clinical Variables
In order to select variables that would be used to build the prediction models. The authors took a data-driven and clinically informed approach.
First, the authors developed a search strategy to interrogate MEDLINE and Embase for candidate predictors. In addition, senior authors (epileptologists) provided additional opinion.
MEDLINE & Embase
MEDLINE is a subset of PubMed (actually about 98%) made available by National Library of Medicine (NLM) to commercial suppliers, used to find literature in the life sciences with a concentration on biomedicine. Embase is an Elsevier database that covers the same subjects as PubMed/MEDLINE, with an additional focus on drugs and pharmacology, medical devices, clinical medicine, and basic science relevant to clinical medicine.
Source: https://library.fiu.edu/c.php?g=160191&p=1047492
You can read more about MEDLINE and Embase here.
What do you think happened here?
The authors searched these two database — I guess in order to get the best predictors for their models. That is not all. Next, they used a modified Delphi process for final variable selection. Let me highlight the process briefly:
- Each participant (panelist) completed a questionnaire rating all potential variables on a 5-point Likert type scale (where 1 indicates the criterion was not very important at all and 5 indicates the criterion was very important). Panelists were also given space for additional handwritten comments and were given 14 days to complete the first round of ranking the questions.
- Panelists filled a second questionnaire similar to the first one. Extra items identified by panelists during the first round were also included.
It is important to note that items with median ratings of 1 to 2 were considered irrelevant and were excluded, items scoring 3 were considered to be of uncertain relevance and were reevaluated in the second round, and items scoring 4 to 5 were used to build the final predictive model.
Finally, to facilitate variable inclusion, two of the authors used a consensus-building process to identify Read and Multilex codes to define each clinical feature of interest in THIN.
Statistical Analysis
This study used regression techniques to build two prediction models, one for the overall population and one for those without a history of a psychiatric sign, symptom, or disorder during the study period.
The index date (time zero) was that on which the patient received the first documented levetiracetam prescription (after meeting the epilepsy case definition). Patients were followed up for 2 years or until an event, loss to follow-up, or censoring.
The authors noted that data were fully available for all variables except levetiracetam daily dose(65%complete). In order to deal with the missing (null) values which was noted to be missing at random. The authors used Rubin and Schenker’s multiple imputation to replace null values.
Finally, the authors assessed model performance using the Brier score (a measure of the forecasting accuracy for probabilistic predictions), model discrimination using the mean area under the curve (AUC), calibration using the Hosmer-Lemeshow goodness-of-fit test, and generalizability using the mean AUC derived from stratified 5-fold cross validation to account for outcome imbalance between groups.
The predicted probability was calculated using 1/[1+exp(−risk score)], where the risk score is equal to the output of the multi-variable logistic regression model. In addition, they mapped sensitivity and specificity according to the probability cutoffs.
I hope the methods used by the authors was clear enough?
Let’s see some of the exciting results. Remember that the aim here is to build two prediction models to predict the risk of psychiatric adverse effects after the use of levetiracetam.
Results
Variable Selection
I hope you can still recall the approach the authors used to select the final variables. First the search from MEDLINE and Embase for candidate variables and the modified Delphi process where panelists filled two similar questionnaires for final variable selection.
Now, let see the results of this approach.
The authors’ search of MEDLINE and Embase yielded 136 articles, of which 103 remained after deduplication.
The variables gotten from this search are: histories of febrile seizures, status epilepticus,
longer duration of epilepsy, psychiatric comorbidities and behavioral issues, and cognitive impairment. These variables were found to be associated with psychiatric adverse effects from levetiracetam use. Also, coadministration of lamotrigine was associated with a protective effect. This means 7 variables were gotten from the search strategy.
An additional 29 variables were recommended for inclusion after consultation with senior authors. Therefore, the first round of the Delphi process consisted of a questionnaire comprising 36 items (7 from MEDLINE and Embase search, 29 from senior authors).
Consensus was ultimately achieved for 14 of 36 variables, of which 12 (85.7%) were considered relevant. This meant that there were 22 variables left out of the 36 initial variables. Also, 13 extra items were identified through the first round.
Therefore, the second round consisted of 35 (22+13) items, of which 11 (31.4%) met consensus for inclusion. Therefore, the authors evaluated 21 variables (12 from the first round and 9 from the second round) for inclusion in
the prediction models.
Prediction Modelling
Now, there are 21 variables. Let us see the results of the models. There are three segments here. General results, prediction model for overall population, prediction model for those without a history of psychiatric sign or disorder.
General Results
From the 11,194,182 patients registered in THIN. It was discovered that 7,400 presumed incident cases over a maximum of 12 years’ follow-up were identified, out of which approximately 16%(n=1173) received an incident prescription for levetiracetam during this period.
The overall median age was 39 years (IQR, 25–56 years), and 590 (50.3%) were female. Psychiatric disorders like depression, anxiety, and psychosis were encountered in 22.3% (262 of 1173), 13.6% (160 of 1173), and 3.2% (37 of 1173), respectively.
The table below shows that depression and behavioral issues were the most commonly reported adverse effects.
Also, as seen in the table below, out of those receiving levetiracetam, 165 (14.1%) experienced an outcome of any psychiatric symptom or therapeutic code over 2 years of follow-up.
Prediction model for overall population
When evaluating all significant variables in multivariable logistic regression, including somatic and psychiatric signs, symptoms, and disorders, the odds of reporting a psychiatric symptom or treatment code within 2 years of persistent levetiracetam use were elevated for some patients.
These included those with the following characteristics: female sex (odds ratio [OR], 1.41; 95% CI, 0.99–2.01; P = .05), increasing social deprivation (OR, 1.15; 95% CI, 1.01–1.31; P=.03), depression (OR, 2.20; 95%CI, 1.49–3.24; P < .001), or anxiety (OR, 1.74; 95% CI, 1.11–2.72; P = .02), and recreational drug use (OR, 2.02; 95% CI, 1.20–3.37; P = .008).
Even though low socioeconomic status was related to higher risk of psychiatric disorder, the authors made a decision to exclude Townsend Index of Social Deprivation from the final model because it would be challenging to assign this score in clinic.
Again, this suggest that the authors aimed at generalizability of the models.
Hence, the final model was as follows:
risk score = −2.34 + 0.27 × (female sex) + 0.82 × (history of depression) + 0.47 × (history of anxiety) + 0.74 × (history of recreational drug use).
There was no evidence of multicollinearity, with variance inflation factors ranging from 1.15 (history of recreational drug use) to 1.42 (history of depression).
Also, the model performed well, with a Brier score of 0.11. It had moderate discriminative capacity and appeared generalizable after stratified 5-fold cross-validation (AUC, 0.68; 95% CI, 0.58-0.79).
The sensitivity and specificity varied, but high specificity (83%) was achieved at a probability cutoff of 0.10 as shown in Figure 1 below. There was no evidence of poor calibration based on the Hosmer-Lemeshow Goodness-of-fit test (P=.29; 10 groups).
eTable 4 and figure 2 below has some interesting insights to show us.
Figure 2 showed a gradient that those with increasing numbers of risk factors were at incrementally greater risk of a psychiatric outcome.
I hope that makes sense? The table also corroborate what the bar plot showed. The table showed the risk score range for different number of risk factors.
For each group of risk factors, the presence of depression and recreational drug use, singularly or in combination, was consistently associated with higher risk
Prediction model for those with no history of Psychiatric Signs or disorder
For those with no documented pre–index date history of a mental health sign, symptom, or disorder (710 of 1173 [60.5%]), 69 of 710 (9.7%) received a code for a psychiatric event within 2 years of levetiracetam prescription.
The following model was generated: risk score = -3.83 + 0.013 × (age) + 0.89 × (female sex) + 1.16 × (recreational drug use) + 0.003 × (levetiracetam daily dose).
There was no evidence of multicollinearity (variance inflation factors range, 1.03–2.83) or poor calibration based on the Hosmer-Lemeshow goodness-of-fit test (P=.18; 10 groups). The model performed well overall (Brier score,0.09) and when using a threshold of 0.14 had a similar specificity of 83% as in Figure 3 below.
Discriminatory capacity and generalizability, as determined through stratified 5-fold cross-validation, was similar to the primary model (AUC, 0.72; 95% CI, 0.54–0.90).
Discussion & Limitations
The authors derived two prediction models for the risk of a psychiatric sign or symptom after a first-ever prescription for levetiracetam in a general population of patients with presumed incident epilepsy.
When using a threshold value of 0.10 (primary model) or 0.14 (for those lacking a premorbid psychiatric history) to guide treatment (a score below these thresholds indicates safety of prescription), the prediction models had a high specificity for predicting those with a psychiatric outcome, yielding few false-positive results.
The authors opined that if the models are validated in prospective cohorts, they could be useful at the point of care for a broad spectrum of patients with epilepsy seen in GP
and epilepsy clinics alike.
In addition, the authors discussed some limitations for the work. I have highlighted some here. They noted that:
- They were unable to extract informative indexes of seizure type, frequency, and severity and epilepsy type. Inclusion of these variables could further improve model performance.
- Caution should be taken in interpreting causation because machine learning regression techniques are designed for predictive rather than inferential purposes.
Link to the Paper
You can download the full paper here.
Conclusion
One of the benefits of the tools described in this study is its application in clinical settings.
Let me use an example from the paper.
A female patient with depression would have a risk score of −1.25 (ie, risk score = −2.34 + 0.27 + 0.82). This is calculated using the overall prediction model.
When incorporated into the algorithm, the patient’s risk of a psychiatric adverse event would be 22% over 2 years, that is: 1/[1+ exp(−1.25)]. This exceeds the threshold of 0.10 and suggests she would be at risk.
In addition, the authors showed the gradients of risk for populations of patients based on known and newly identified factors. The estimates of generalizability using a stratified 5-fold cross-validation are encouraging.
This is an important first step toward generating empirical, efficient, and practical prediction models with direct application in clinical settings.
I hope you loved the review. Please give this a clap if you learnt from this review.
See you next week!
