OUP user menu

Analysis of overridden alerts in a drug–drug interaction detection system

Frédéric Mille, Céline Schwartz, Francoise Brion, Jean-Eudes Fontan, Olivier Bourdon, Patrice Degoulet, Marie-Christine Jaulent
DOI: http://dx.doi.org/10.1093/intqhc/mzn038 400-405 First published online: 10 September 2008

Abstract

Objective The aim of this study was to evaluate the relevance of the signals generated by a computerized drug–drug interaction detection system and to design a classification of overridden drug–drug interaction alerts.

Study Design Prospective study over two months.

Setting Five hundred and ten-bed university paediatric hospital.

Main Outcome Measures In Robert Debré Hospital physicians generate drug orders online using a computerized physician order entry system that also detects drug–drug interactions in real time. We analysed the relevance of a sample of alerts overridden by physicians.

Results We analysed a sample of 613 overridden alerts. We defined three categories of overridden alerts: informational errors (35); system errors (244) and accurate alerts (334). Two reasons account for 40% of false-positive alerts: an inability of the system to recognize real conflicts between drug treatments and guidelines stating that the two drugs can be used together, because the benefit outweighs the risk of side effects due to the drug–drug interaction.

Conclusions We created a classification of overridden alerts, in the context of computerized physician order entry system coupled with a drug–drug interaction detection system. There is clearly room for improvement in the development of drug–drug interaction software. This classification should make it possible to break this work down into smaller tasks, making it possible to decrease the sensitivity to background noise of drug–drug interaction detection systems.

Keywords
  • drug–drug interaction
  • clinical decision support systems
  • alert relevance

Introduction

In 1999, an Institute of Medicine report entitled ‘To err is human: building a safer health system’ estimated that ∼80 000 people are hospitalized and 7000 die each year in the United States due to medication errors [1].

Technology-based intervention has been identified as a key means of reducing the likelihood of medication errors and improving safety [2]. Computerized physician order entry systems have been proposed as one of the most effective ways of avoiding medication errors [3, 4]. Particularly large improvements are expected to be achieved by coupling these systems to clinical decision support systems [5], which provide health professionals with direct assistance when deciding on the drugs to be prescribed. Drug–drug interaction detection systems were the first clinical decision support system to be developed.

In a previous study (conducted in AP-HP Robert Debré Hospital in Paris, France, with PCS® system presented in the Methods session), a drug–drug interaction detection system detected 12 326 drug–drug interactions. Physicians overrode a large proportion (70%) of these alerts. This limitation of the system (alert override) has been described in other studies, based on different methodologies, all of which have reported a significant rate of overridden alerts [610]. A significant proportion of physicians admit to overriding drug–drug interaction alerts [6]. Many clinicians complain of low signal-to-noise ratios and too many irrelevant warnings [7, 8]. They explain that major interactions might be missed through their becoming desensitized due to the excessive number of irrelevant alerts [9]. These studies indicate that the poor specificity and irrelevance of alerts may reduce the confidence of physicians in clinical decision support systems, a phenomenon known as the ‘Cry Wolf Syndrome’ [6].

Drug–drug interaction detection systems seem to lack specificity. The main objective of this work was therefore to determine which types of alerts were overridden. We evaluated the relevance of the alerts provided by a system, and designed a classification of the overridden alerts to facilitate the construction of effective drug–drug interaction detection systems. The long-term goal is to determine the specifications required to improve these systems, making it possible to endow systems with an ability to discard or filter out inappropriate alerts, thereby enhancing specificity.

Methods

Sites and subjects

The AP-HP Robert Debré Hospital in Paris, France, currently uses the hospital information system, PCS® (Patient Care System-IBM). PCS® manages the information relating to hospitalized patients: drug prescriptions, prescription of laboratory tests, and the results of laboratory tests. It includes a module for the real-time detection of drug–drug interactions. It supplies the physician with an online drug–drug interaction alert. The physician may then cancel the order or override the alert, thereby confirming the order. Each alert is recorded.

PCS® triggers alerts corresponding to four levels of constraint. The strongest constraint level is that of ‘contraindication’. Contraindications are absolute in nature and should not be disregarded (e.g. combination of cisaprid and erythromycin). The next level down is ‘not advised’. If a combination is identified as ‘not advised’ it should generally be avoided, unless rigorous examination of the risk/benefit ratio suggests otherwise, and close monitoring of the patient is necessary (e.g combination of nifedipine and cyclosporine). The next level down is ‘cautioned.’ If a combination is ‘cautioned’, it may be prescribed provided that simple recommendations are respected, particularly at the start of treatment, to prevent side effects (e.g combination of fluconazole and cyclosporine). The last constraint level is ‘drug duplication’. Drug duplication occurs if physicians prescribe different formulations containing the same active ingredient (e.g. two drug formulations containing acetaminophen).

Design of the study protocol

In this prospective quantitative and qualitative study, we aimed to describe the relevance of alerts and design a classification of overridden alerts. To describe relevance, a pharmacist analysed a sample of alerts overridden, between 1 November 2006 and 31 December 2006. He analysed each alert to determine whether the alert was relevant and whether the alert was justified (true positive alert) or unjustified (false positive alert).

An alert is relevant only if there is a real drug–drug interaction. There is a real drug–drug interaction only if the two drugs are concurrent and are taken in sufficiently large amounts via a route compatible with their interaction. Two drugs are concurrent if there is an overlap (whatever its duration) between the duration of treatment with one drug (determined from dose schedule prescribed) and the duration of treatment with the other drug.

An alert is justified if it is clinically relevant. An alert is clinically relevant if the two drugs are concurrent, taken in sufficiently large amounts, via routes compatible with interaction and if there are no clinical practice guidelines justifying the use of a given combination of drugs. Such guidelines often lead physicians to override alerts for good clinical reasons, when the benefits of a particular combination of drugs outweigh the disadvantages of possible drug–drug interaction. It may therefore not be worthwhile disrupting the clinical workflow to raise alerts concerning combinations that the physician is used to prescribing.

Finally, we also checked whether the physician had requested additional laboratory tests (e.g. serum blood creatinine concentration) for patient monitoring. Other forms of patient monitoring (e.g. electrocardiogram) were not included in this study.

We did not investigate the clinical impact of accurate interactions on the patients concerned in terms of adverse drug events.

Data collection

Analysed sample

The analysed sample was constructed by computerized randomization.

The pharmacist's analysis

Each alert was reviewed by a pharmacist, who considered the doses prescribed, the route of administration (e.g. intramuscular), periods of simultaneous drug administration and the clinical practice guidelines applied on the ward concerned. Analysis of these factors made it possible to determine whether the alert was relevant and justified.

Clinical practice guidelines collection

The clinical practice guidelines in use were determined by interviewing physicians and through literature review. They describe all known diseases or physiological conditions in which the benefit of a given combination outweighs the potential risk of an adverse event due to drug–drug interactions (e.g. the combination of lamotrigine and valproic acid may induce Lyell syndrome, but may also be effective against some cases of epilepsy).

Laboratory tests request

We searched the electronic health record contained in PCS® for evidence of the prescription of laboratory tests

Analysis

Data were recorded and analysed with Microsoft Excel. We thus propose a classification of overridden alerts. The number and percentage of each type of alert were noted. We also determined the proportion of alerts corresponding to each constraint level and compared the results obtained with the classification of overridden alerts.

Results

Between 1 November 2006 and 31 December 2006, physicians overrode 69% of the alerts (Table 1). We analysed a sample of 613 overridden alerts and defined three categories of overridden alert (Table 2).

View this table:
Table 1

Alerts generated by PCS® between 1 November 2006 and 31 December 2006

Level of constraintOverridden alertsAlerts not overriddenTotal (%)
Contraindication311647 (1.4)
Not advised648207855 (25.1)
Cautioned8002691069 (31.4)
Drug duplication8585751433 (42.1)
Total (%)2337 (68.7)1067 (31.3)3404 (100)
View this table:
Table 2

Classification of overridden drug–drug interaction alerts

TypeSubtypeNumber (%)
Informational errors35 (5.7)
System errorsConcerning date193 (31.5)
Concerning dose12 (1.9)
Concerning route of administration9 (1.5)
Others30 (4.9)
Accurate alerts334 (54.5)
Total613

The first of these categories, ‘informational error’ (35 alerts, 5.7%), corresponds to false-positive alerts. There is a real drug–drug interaction based on the doses prescribed, the route of prescription and the existence of an overlap in the periods of administration for the different drugs. However, there is evidence-based knowledge to override the alert (hence a false positive). In these cases, the benefit of the combination outweighs the risk.

‘Systems errors’ (244 alerts, 39.8%) are also false-positive alerts. For this category, the system is triggered in the absence of evidence of a drug–drug interaction based on the doses and routes of administration prescribed and the timing of administration (hence a false positive).

The third category of overridden alerts was the ‘accurate alert’ (334 alerts, 54.5%), corresponding to a true positive alert. In such cases the system is correctly triggered but the physician chose to override the alert and confirm the order.

Overall, 45.5% of the overridden alerts were false-positive alerts (244 and 35 for systems and informational errors, respectively). These alerts were not clinically relevant and were rightly overridden by the physicians.

The distribution of alerts as a function of level of constraint is shown in Table 3. No accurate alerts for ‘contraindication’ were overridden. This absolute level of constraint was overridden only for relevant reasons. Overall, 21.4% of alerts for the ‘not advised’ level of constraint and 26.1% of those for ‘cautioned’ alerts were not clinically relevant. We found that 88.9% of ‘drug duplication’ alerts corresponded to systems errors and were not clinically relevant.

View this table:
Table 3

Levels of constraint and classification of the overridden alerts

Level of constraintInformational errorsSystem errorsAccurate alertsTotal (%)
Contraindication4206 (1.0)
Not advised2028176224 (36.5)
Cautioned741136184 (30.0)
Drug duplication417322199 (32.5)
Total (%)35 (5.7)244 (39.8)334 (54.5)613 (100)

Five main clinical practice guidelines were identified (Table 4) as being used by physicians as a justification for overriding certain alerts. These clinical practice guidelines provide evidence-based knowledge going against certain alerts, and may therefore justify the overriding of false positive alerts. They account for informational errors.

View this table:
Table 4

Informational errors

ProtocolNumber of drug–drug interactions identified
Cyclosporine–nifedipine11
Lamotrigine–valproic acid6
Morphine–nalbuphine3
Hydrocortisone–dexamethasone2
Methotrexate–cyclosporine2
Others11

We found three subtypes of system errors (Table 2). The first was omission of the date of administration: the drug–drug interactions detection system does not take into account the dates on which drugs were administered. It includes no concept of time and is therefore unable to recognize truly concurrent treatments. For example, 10 mg of codeine twice daily between 1 December and 5 December and 20 mg nalbuphine twice daily between 20 December and 25 December would be recognized as ‘not advised’, but there is no overlap in the treatment periods for the two drugs and, therefore, no risk of drug–drug interaction.

The second subtype corresponded to omission of the dose prescribed: the drug–drug interactions detection system does not take into account the dose of the drug. This may result in the system announcing drug duplication without maximum daily dose being exceeded. For example, a patient might be prescribed one 36 mg CONCERTA® (methylphenidate) tablet daily, to be taken in the morning plus one 18 mg CONCERTA® tablet to be taken before lunch. In this case, there is no drug duplication because the total daily dose does not exceed 60 mg.

The final subtype concerned the omission of the prescribed route, as the drug–drug interactions detection system does not take into account the route of administration (e.g. intramuscular route, intravenous route). Thus, the drug–drug interactions detection system may issue an alert even if one of the drugs in the combination is not administered by systemic way. For example, a patient might be prescribed 5 mg/2.5 ml Ventolin® (albuterol sulphate) at a dose of 2.5 mg per inhalation over 5–15 min, three to four times daily plus 0.5 mg/1 ml Salbutamol® administered intravenously (albuterol sulphate) at a dose of 30 µg/min. This combination may be used to prevent preterm labour in a woman with asthma and might trigger a duplication alert. However, there is no real duplication as Ventolin® is not administered intravenously.

In total, 334 of the overridden alerts were accurate alerts. However, 159 of these alerts (48%) involved the combination of a formulation containing alcohol with a sedative drug (Table 5), such combinations generating a ‘not advised’ alert. In these cases, it is the ethanol present as an inert component in many liquid preparations rather than the active molecule itself that is involved in the interaction. Pharmaceutical companies are trying to replace alcohol with another agent when possible, so this type of drug–drug interaction should eventually disappear. However, clinicians are currently obliged to override some of these alerts as there is not always an alcohol-free alternative.

View this table:
Table 5

Accurate alerts

Number (%)
Alcohol interaction159 (47.6)
Laboratory-based monitoring65 (19.5)
No laboratory-based monitoring110 (32.9)

In 19.5% (65/334) of the accurate alerts, the physician may not have cancelled the order but did make use of the alert to prescribe laboratory tests for monitoring of the patient. No such laboratory testing was requested in 32.9% (110/334) of accurate alerts (Table 5), despite 89 of these drug–drug interactions being at the ‘cautioned’ level (15) or ‘not advised’ levels (74). Thus, 15% of overridden alerts correspond to accurate alerts for which the physician does not request laboratory tests. The physicians concerned may have requested another form of monitoring, but these requests were not recorded in the electronic health record. These other forms of precaution are therefore not measurable.

Discussion

In PCS®, 70% of alerts are overridden. However, this is not a problem in itself because the decision to override an alert is justified in many cases. As pointed out in the review by Van Der Sijs et al. [11], overriding alerts is unsafe only if not justified. Some alerts are inappropriate, because they are incorrect and do not apply to the patient concerned at the time considered. They can therefore be reasonably overridden. Some appropriate alerts may also be overridden with justification. However, overriding alerts is unacceptable if it results from the physician simply ignoring alerts, misinterpreting alerts or selecting the wrong option in the software used.

PCS® generates many alerts due to failings in its own system. These alerts are inappropriate and can be overridden with justification. The deficiency of the system is the lack of data concerning the date and time of administration. The system is based on the list of active ingredients prescribed rather than strict periods of overlap in the administration of different drugs.

Various drug–drug combinations were prescribed to patients despite alerts, with evidence-based justification. Particular drug–drug combinations are beneficial in certain diseases and alerts concerning them may therefore not be appropriate for the patient considered, justifying the overriding of the alert. Existing systems use static knowledge, mostly embedded in fixed mapping tables, and compare drugs based on a hard-coded interaction mapping table, without considering the clinical context of the patient. This constitutes another limitation of existing systems, many of which are highly inclusive, placing more emphasis on breadth of coverage than on clinical relevance.

Recommendations for the management of interactions exist for most drug–drug interactions: modification of the time sequence or manner of administration, compensation with a third agent or replacement of depleted endogenous substances, dose adjustment, discontinuation on the occurrence of symptoms and use of a non-interacting alternative. These risks are therefore manageable and recommendations for the monitoring of interactions are sometimes contained in the summary of product characteristics of the drugs concerned. This information is often shown as simple text on screen with the alert, but without explicit details concerning management of the drug–drug interaction. This is a third failing of these systems, which do not consider all the information contained in the summary of product characteristics and therefore cannot propose explicit monitoring. Physicians must read the summary of product characteristics in its entirety to identify appropriate management options, and this slows clinical workflow.

Finally, potentially hazardous drug–drug combinations without evidence-based justification were prescribed despite alerts. These alerts were appropriate. Two categories of such alerts exist: overridden alerts with and without the prescription of laboratory tests. In cases in which no laboratory tests were prescribed, it should not necessarily be assumed that the doctor was wrong not to prescribe testing. The doctor may have advised the patient to report any adverse reactions or requested another form of supervision (e.g. clinical surveillance). Moreover some drug–drug interactions may not have required laboratory monitoring.

Based on our findings, we can now propose several changes to improve clinical decision support systems. Attention to the interface with the user could improve the usefulness of clinical decision support systems [9], improving its integration into clinical workflow [1214]. Alerts with a lower level of clinical impact could be shown in a less intrusive manner [12]. Particular alerts for certain specialist groups could be turned off [9]. Another change might be allowing physicians to turn alerts off, based on their own practice, knowledge and comfort levels. Explicit instructions for patient monitoring or alternative drugs could be displayed with each alert. The triggering of lower-level alerts for which neither monitoring nor an alternative can be offered could be prevented [15]. Finally, overridden alerts should not be triggered repeatedly for the same patient, because the combination concerned may be considered to be tolerated by the patient [14].

Weingart et al. [10] showed that there were few adverse drug events, despite the maintenance of prescriptions by many physicians in response to interaction alerts. This finding suggests that the alert threshold was too low. We could not carry out the same analysis as adverse drug events are not recorded in PCS®. However, we feel that clinical decision support systems should suppress alerts for evidence-based drug combinations and for the renewal of drug combinations clearly tolerated by the patient. These alerts could be rendered non-intrusive, by simply presenting the information on the computer screen, without requiring specific action.

Automatic drug–drug interaction alerts could potentially increase the recognition of selected drug interactions by clinicians. However, the perceived low specificity of drug alerts may be a large obstacle for the efficient use of information, preventing any major gain in safety due to such alerts. In most studies [6, 7, 9, 10], practitioners have expressed the view that clinical decision support software should integrate context-relevant information, guidelines or evidence-based medicine. The conclusions of this study are consistent with these observations.

This study is subject to several limitations. Firstly it was carried out at a single academic medical institution. Secondly, the software did not record the sequence of actions for the physicians. It is therefore impossible to determine whether laboratory tests were prescribed and orders cancelled before or after the alerts were issued. Some interactions may not have required laboratory tests, but this was the only type of monitoring included in our study. However, other precautions are less easy to assess than the prescription of laboratory tests or order cancelling. Finally, PCS® software is an old system. It was installed at Robert Debré Hospital in 1988, but the method used to detect interactions is very similar to that of current systems. We therefore believe that the conclusions of this study are unlikely to be affected by the age of PCS®. Moreover, this tool can be developed further in-house at Robert Debré Hospital, as IBM has provided the basic software package. It therefore remains possible to develop PCS® further.

This study is the first study not based on interviews to show a potential risk of the ‘Cry Wolf Syndrome’. Computerized systems prevent most errors. However, studies similar to ours have suggested that computerized systems may also lead to additional medication errors [1620]. The positive effects of these systems may be compromised by new kinds of errors, specific to the inherent cognitive complexity of human–computer interactions [18]. Thus clinical decision support systems should have high specificity and present pertinent information, without disrupting the clinical workflow unnecessarily.

The results of this study probably identify specifications for the development of drug–drug interaction detection algorithms. These algorithms should take into account all the relevant information concerning an order, including, in particular, the days and hours at which the drugs are to be administered.

Conclusion

Drug–drug interactions detection software could be improved. Physicians require informative support and guidance concerning the management of drug interactions [11]. It is therefore essential to distinguish between clinically relevant and negligible drug interactions [21]. Refining order-check logic may reduce the frequency of overrides and physician acceptance of the system, thereby increasing the efficacy of order checks [22].

Our findings should help software developers to determine the needs and expectations of end users and to develop suitable drug–drug interactions detection systems. Specificity can be improved by providing access to information from additional sources. This information includes patient records, making it possible to take the clinical context of the patient.

References

View Abstract