OUP user menu

Selection of indicators for continuous monitoring of patient safety: recommendations of the project ‘safety improvement for patients in Europe’

Solvejg Kristensen, Jan Mainz, Paul Bartels
DOI: http://dx.doi.org/10.1093/intqhc/mzp015 169-175 First published online: 9 April 2009

Abstract

Background Initiatives to improve patient safety have high priority among health professionals and politicians in most developed countries. Currently, however, assessment of patient safety problems relies mainly on case-based methodologies. The evidence for their efficiency and reproducibility, proving that safety of care has improved with their usage, is questionable. The exact incidence and prevalence of patient safety quality problems are unknown. Therefore, there is a need for firm, evidence-based methods to survey and develop patient safety and derived activities.

Objectives The objective of this paper is to describe a method to select patient safety indicators and present the indicators derived through this process.

Methods The patient safety indicators were derived and recommended for use in a formalized consensus process based on literature review, targeted information gathering, expert consultation and rating procedures.

Results A total of 42 indicators, of which 28 originated from existing international indicator programmes, were selected. The processes and outcome indicators that were recommended for institutional-level use in Europe were 24, covering safety of care aspects such as culture, infections, surgical complications, medication errors, obstetrics, falls and specific diagnostic areas.

Conclusion The patient safety indicators recommended present a set of possible measures of patient safety. One of the future perspectives of implementing patient safety indicators for systematic monitoring is that it will be possible to continuously estimate the prevalence and incidence of patient safety quality problems. The lesson learnt from quality improvement is that it will pay off in terms of improving patient safety.

Keywords
  • safety indicators
  • patient safety
  • measurement of quality
  • systematic monitoring
  • improvement

Background

Currently, assessment of patient safety problems relies mainly on case-based methodologies, such as reporting systems and analysis of single or aggregated events covering the same theme, e.g. using root cause analysis, and resulting in action plans to reduce or eliminate risk. This is a highly resource demanding work, and although the methods are widely accepted and implemented, the evidence of their efficiency and reproducibility, proving directly reduced risk or harm—that safety of care has improved with their usage—is questionable [13]. To a large extent, safety of care development in Europe is still an area of quality improvement resting on qualitative processes based on expert assessment and recommendations. Across the European nations, a number of cross-sectional studies have been performed to estimate the frequency of adverse events [4]. These studies provide snapshots of safety of care, typically based on knowledge from review of patients' charts—a highly laborious task, which is generally only performed once. Continuous, systematic monitoring of the frequency and nature of safety of care incidents is hardly ever performed.

It has been documented that performance and outcome measures can improve the quality of care [5]. Such measures have supported accountability and transparency, helped to make judgements and set priorities, enabling comparison over time between providers and the effectiveness of interventions [6, 7]. Accordingly, specific indicators for systematic surveillance, monitoring and development of safety of care and patient safety activities are needed [8].

In 2004, the Committee of Ministers and The Council of Europe made a number of recommendations regarding patient safety [8]. One recommendation was to develop reliable and valid indicators of safety of care On the basis of these recommendations, the project ‘Safety Improvement for Patients in Europe’ set out to develop a toolbox for safety of care, presenting a variety of tools for development, surveillance and monitoring of safety of care and patient safety activities. One of the tools was a set of safety of care indicators [9].

Objectives

The objectives of this paper are to describe the methods used to select a set of evidence-based evaluated safety of care indicators applicable for use in clinical settings in European healthcare, and to present the clinical safety of care indicators selected and recommended. They are useful for:

  1. Identification of the institutional level of safety of care,

  2. Systematic surveillance and monitoring of the impact of safety of care activities.

Definitions and concepts

Safety of care indicators can be defined as measures assessing a particular healthcare process, structure or outcome, and as measuring tools, screens or flags used as guides to monitor, evaluate, and improve the quality of care, clinical support services and organisational functions affecting safety of care [10, 11].

Safety of care structure and process indicators aim at measuring healthcare organizational features, practices or interventions with evidence of effects on exposure to preventable risk factors (e.g. safety culture, hand washing practices, screening of schizophrenic patients for suicidal risk). Safety of care outcome measures aim at measuring harm which is or may be conceived as preventable events: death, permanent or serious temporary disability [12].

Most sentinel indicators have limited value as measures at the institutional level, primarily because they are rare and will be know off otherwise. However, these indicators have been found to be useful at the systems level, e.g. as part of the patient safety indicators of the Organisation for Economic Co-operation and Development (OECD) [13].

Institution-level indicators are intended to monitor potentially preventable problems and complications that arise at a specific institution. Whereas indicators targeted at the systems level are based on aggregated data from all of a particular type of institution, such as public hospitals or community health centres.

In terms of methodological demands for characterization, selection, evaluation and validation of safety of care indicators, they must be considered as ‘quality indicators’ with a specific framework for interpretation [10, 14], referring to preventable safety problems reflected in processes or outcomes [15]. Thus, the general basic principles described in the literature on defining, classifying and developing quality indicators also apply to patient safety indicators [11, 16]. The special characteristics and validity of patient safety indicators, therefore, depend strongly on interpretation in a safety of care context [17].

Methods

The step-wise methodological approach used to select the safety of care indicators is summarized in Table 1. Steps 1–6 were carried out in this project.

View this table:
Table 1

Steps in the process of selecting patient safety indicators

PhaseStep
Planning phase1. Choose the area of patient safety to develop
Establish importance (high volume, cost, variation, feasibility)
Identify opportunities for improvement of safety
2. Select and organize the developmental team and assign tasks
Developmental phase3. Provide an overview of existing evidence, methods and practices for potential patient safety indicators
4. Select process, structure and outcome indicators (and standards)
Identify confounding factors (risk adjustment)
Establish consensus and rating procedures
5. Characterize the indicators; describe/design measure specifications according to the characterisation form
Test phase6. Evaluate the selected patient safety indicators
7. Validate the psychometric properties
8. Adjust indicator characterisations—recommend validated indicators

An expert group of 12 European representatives of project partners, stakeholders and external experts was established. The expert group was selected according to the following criteria:

  1. Knowledge of establishing and/or working with quality indicators,

  2. Geographic representation of the variation of European healthcare systems,

  3. Practical knowledge of patient safety.

The group worked in a formalized consensus process by means of discussions and interactions via mail contact and telephone conferences.

An extensive literature search was initiated to identity: (i) nationwide and international safety of care indicators and/or indicator programmes, which had been clinically applied, and (ii) papers describing themes and suitable methods based on safety of care indicators. Furthermore, calls were made within the network of the expert group for information on nationwide and international safety of care indicators and/or indicator programmes. The literature identified from 10 indicator programmes was reviewed to determine suitability for selection of individual indicators. To assess whether an indicator qualified, it had to meet the definition: ‘Patient safety indicators are measures that directly or indirectly monitor preventable adverse events’ [10].

Taking into account the frequency and severity of safety of care problems, as well as the existence of evidence-based interventions towards problems [4, 18], three indicator areas for hospital-related safety of care indicators were chosen:

  1. institution-wide indicators,

  2. theme-specific indicators,

  3. diagnosis-specific as well as other specific safety of care indicators.

The institution-wide safety of care indicators were defined to address general safety of care characteristics of healthcare organizations. Theme-specific indicators monitor preventable processes or outcomes, related to specific clinical themes, e.g. infections, surgical complications, medication errors, obstetrics and falls.

All indicators chosen were characterized according to a common questionnaire, regardless of the original source and description. On the basis of the standardized characterization, the members of the expert group conducted a review of each indicator and discussed the indicators in telephone conferences to ensure common understanding of the definitions and phrasing, and to make decisions on alterations and refinements.

Indicators (both existing and new) were evaluated by individual members of the expert group using a scoring sheet. Evaluation of the indicators was carried out according to three dimensions (relevance and appropriateness, validity and reliability and feasibility) of the indicator on a scale ranging from 1 to 9, please see Table 2 below. Moreover, ‘validity and reliability’ covered both assessment of scientific properties of the indicators, and assessment of whether the indicator was measuring the right thing.

View this table:
Table 2

Questionnaire for evaluation of indicators

Evaluation questionnaire
DimensionDefinitionScore
Relevance and appropriatenessAre areas of significance covered (severity and frequency) in terms of patient safety within its specified domain (population and/or organisation)?1–3 Low degree of relevance; 4–6 medium degree of relevance; 7–9 high degree of relevance
Validity and reliabilityIs the instrument satisfactory in terms of: construct validity (evidence-based); internal consistency; exhaustiveness/exclusiveness; reliability1–3 low degree of validity; 4–6 medium degree of validity; 7–9 high degree of validity
FeasibilityHow is the: availability of data; clinical burden of data collection1–3 low degree of feasibility; 4–6 medium degree of feasibility; 7–9 high degree of feasibility
Scoring sheet
Title of the evaluated instrument:
ScoresAny additional comments
Relevance and appropriatenessValidity and reliabilityFeasibility
Score from 1 to 9Score from 1 to 9Score from 1 to 9Free text

Scores were divided into 1–3 low degree, 4–6 medium degree and 7–9 high degree. Ratings on each of the three dimensions of each indicator were added and percentiles, mode, minimum and maximum scores calculated and frequencies of scores displayed graphically. On the basis hereof, the expert group discussed recommendations for application in Europe in telephone conferences. When making these recommendations, the expert group also focused on aspects, such as resources available, organisation in individual EU countries, legal systems etc., in connections with the dimension feasibility. However, these aspects were not systematically uncovered Europe-wide.

Implementation was recommended according to four categories:

  1. Immediately workable ‘throughout’ the European healthcare systems,

  2. Immediately workable ‘in parts’ of the European healthcare systems,

  3. At present not workable for implementation in Europe—Recommendation for future decision on implementation or

  4. Not suitable as a PSI for recommendation in Europe.

Results

A total of 42 clinical institutional level safety of care indicators, of which 28 originated from existing international indicator programmes, were selected. A total of 24 indicators were recommended for use in Europe: 9 indicators were recommended for implementation across Europe, and 15 safety of care indicators were recommended for application in parts of Europe. The majority of indicators were rate-based, and most of them were process or outcome indicators. A list of the 24 recommended safety of care indicators is shown in Table 3, indicators not recommended are shown in Table 4. Application of a number of these indicators requires a general supplementary comment as they are regarded sensitive to bias caused by patient disease severity, comorbidities and/or lifestyle factors. Indicator results depend on exhaustive data definition, extensive collection of patient-related data and appropriate risk adjustment, for which the group did not find sufficient evidence. The indicators selected have not yet been systematically field-tested in a European setting.

View this table:
Table 3

List of the patient safety indicators ‘recommended for use in parts or throughout Europe’

Indicator category and nameSourceaApplicationb
Institution-wide patient safety indicators
Measuring hospital standardised mortality ratesSimPatIE2
Transition of care—patients' understanding of the purpose of their medicationSimPatIE2
Institution-wide use of cultural assessmentSimPatIE1
Surveying the development of the patient safety cultureSimPatIE1
Theme-related patient safety indicators: ‘infection control’
Hospital-acquired infection registration—post-operative wound infectionsSimPatIE2
Wound infectionOECD, CSP1
Ventilator pneumoniaOECD2
Hand hygiene—measured by alcohol consumptionSimPatIE1
Theme-related patient safety indicators: ‘surgical complications’
Complications of anaesthesiaAHRQ, CIHI, OECD2
Post-operative sepsisAHRQ, OECD1
Post-operative haemorrhage or haematomaAHRQ1
Post-operative physiologic metabolic derangementsAHRQ2
Post-operative respiratory failureAHRQ2
Theme-related patient safety indicators: ‘medication errors’
Transfusion reactionAHRQ, CIHI OECD2
Wrong blood typeOECD2
Electronic trigger tool—surveillance of adverse drug eventsSimPatIE2
Theme-related patient safety indicators: ‘obstetrics’
Obstetric trauma—vaginal delivery without instrumentAHRQ, JCAHO2
Obstetric trauma—vaginal delivery with instrumentAHRQ, CIHI, OECD2
Birth trauma—injury to neonateAHRQ, CIHI, OECD2
Theme-related patient safety indicators: ‘in-hospital fall’
Post-operative hip fractureAHRQ, CIHI, OECD1
In-hospital hip fracture or fallOECD,CSP1
Diagnosis-specific as well as other specific patient safety indicators
Decubitus ulcerAHRQ, CIHI, OECD1
Assessment of suicidal risk in schizophrenic patientsNIP2
Assessment of side effects of anti-psychotic treatmentNIP2
  • aSource: Agency for Healthcare Research and Quality (AHRQ), Canadian Institute for Health Information (CIHI), Complication Screening Programme (CSP), Joint Commission on accreditation in Health Care (JCAHO), Organisation for Economic Co-operation and Development (OECD), The Danish National Indicator Project (NIP)

  • bApplication: 1 = immediately workable throughout the European healthcare systems, 2 = workable in parts of Europe.

View this table:
Table 4

List of the safety of care indicators identified but ‘not recommended for use’

Indicator category and nameSourceaApplicationb
Institution-wide patient safety indicators
Death in low-mortality DRGsAHRQ3
Patients experiencing adverse eventsSimPatIE3
Patients informed about an adverse event by the staffSimPatIE3
Patient experiences of adverse events managementSimPatIE3
Theme-related patient safety indicators: ‘infection control’
Selected infections due to medical careAHRQ, OECD4
Hand hygiene—staff's compliance with guidelines for use of jewellerySimPatIE4
Theme-related patient safety indicators: ‘surgical complications’
Foreign body left during procedureAHRQ, CIHI, OECD4
Post-operative pulmonary embolism or deep vein thrombosisAHRQ, CIHI, OECD4
Accidental puncture or lacerationAHRQ, OECD3
Wrong site surgeryJCAHO, OECD3
Medical equipment-related adverse eventsJCAHO, OECD3
Patients experiencing harmful surgical adverse eventsSimPatIE3
Theme-related patient safety indicators: ‘medication errors’
Medication error (did not fulfil the criteria as an indicator, therefore deleted)JCAHO, OECD
Theme-related patient safety indicators: ‘obstetrics’
Obstetric trauma—caesarean deliveryAHRQ, OECD3
Problems with childbirthACSQ, OECD3
Theme-related patient safety indicators: ‘in-hospital fall’
Patient fallsJCAHO, OECD4
Diagnosis-specific as well as other specific patient safety indicators
Failure to rescueAHRQ4
Iatrogenic pneumothoraxAHRQ3
  • aSource: Agency for Healthcare Research and Quality (AHRQ), Australian Council for Safety and Quality (ACSQ), Canadian Institute for Health Information (CIHI), Joint Commission on accreditation in Health Care (JCAHO), Organisation for Economic Co-operation and Development (OECD); bApplication: 3 = at present not workable for implementation in Europe—Recommendation for future decision on implementation, 4 = Not suitable as a PSI for recommendation in Europe.

As the indicators were evaluated for feasibility, aspects such as data availability, quality of administrative data, resources available for indicator monitoring, organisation of data collection, legal systems concerning data collection of individual data etc. were identified as areas characterized by great variation in the European countries. As these aspects were not covered by the aim of the project, they remain to be investigated and systematically uncovered for Europe, if the recommendations on safety of care indicators are to be used for other purposes other than local monitoring of safety of care, e.g. national or European benchmarking.

Full descriptions of the safety of care terms used and the developmental process, and the measure specifications of the indicators are available on www.simpatie.org [10, 14, 19].

Discussion

In the current situation of safety of care, it seems reasonable to discuss whether the emphasis placed on patient safety initiatives is primarily a matter of convenience and political appeal. However, the most convenient action is not necessarily the correct action. Therefore, the evidence-based strategy argument is that scientifically established practices proven to improve safety of care should become the standard of high quality healthcare. This project was designed to meet this need and initiated to recommend an internal set of indicators to be used in efforts to improve safety of care at the institutional level (in hospitals) in Europe.

A set of 42 clinically multifacetted indicators covering safety of care aspects at the institutional level were established, and 24 indicators were recommended for implementation in parts or throughout Europe. The indicators divide into two groups: 17 indicators from known indicator programmes, mainly originating from the AHRQ and OECD, suitable for use at the systems level, and 7 SimPatIE indicators, which are recommend for clinical use at the institutional level. An exception was made with the sentinel event indicator of ‘wrong blood type’, and as the theme was considered highly important for surveillance and risk management, the indicator was included in the recommended indicators for use at the institutional level. The newly developed SimPatIE indicator set deviates from the indicators of the AHRQ and OECD by containing process indicators. Process indicators have been found to be the most suitable tool for performance management, directly focusing on the problem areas and encouraging improvement. Whereas clinical outcomes have been found to be affected by factors other than the quality of care, thus providing insufficient information about how to improve [20].

Prior to embarking on actual patient safety assessment activities using the recommended safety of care indicators, a systematic strategy should be established at the institutional level to measure, report and use the indicator information to ensure better safety of care and prevention of future events. Safety of care is multidimensional, and therefore attempts to understand that it requires supplementing measures [21, 22]. In this context, the continuous use of safety of care indicators supplemented by other measures to improve safety is highly recommended.

A number of the indicators are dependent on administrative data, requiring sophisticated resources in terms of informatics and reliable system-wide patient identification and data processing. It is strongly recommended that future projects follow up on and investigate these aspects. Also, general field-testing of the indicators focusing on indicator sensitivity and specificity remains necessary. Within the project European Network for Patient Safety (EUNetPaS), a smaller set of common indicators of the OECD and SimPatIE will be field-tested [13, 19].

Conclusion

The recommended safety of care indicators represent a broad selection of possible measures of safety of care. Systematically applied, they can supply clinicians, risk managers, policymakers, the public and researchers with ongoing and comprehensive data on aspects of patient safety for further investigation. The themes and areas covered by the suggested indicators are not intended to be exhaustive in the development of institutional safety of care. One of the perspectives of implementing safety of care indicators for systematic monitoring and surveillance is that it will be possible to continuously estimate the prevalence and incidence of safety of care quality problems. The lesson learnt from quality improvement is that it will pay off in terms of improvements in safety of care [23].

Funding

The SImPatIE project was made possible by a grant from the European Commission on ‘Public Health/Directorate General for Health and Consumer Affairs in 2004’ (OJ 2004/C52, 27 February 2004).

Acknowledgement

We would like to acknowledge all participating experts: MD, PhD A. Bourek, University Center for Healthcare Quality, Masaryk University, Czech Republic; MD I. Callanan, Irish Society for Quality and Safety in Healthcare, Ireland; MD K. Essinger, European Hospital and Healthcare Federation in cooperation with MD J. Ahlberg, Patientforsakring, Sweden; MSc R. Gijsen, in cooperation with MPH, J. de Koning appointed by Kwaliteitsinstituut voor de Gezondheidszorg CBO, Holland; Dr M. Kallewaard, Association of Medical Specialists, Holland; MD B. Lilja in cooperation with MD J. Anhoej, the Danish Society for Patient Safety, Denmark; MD G. Maguerez; appointed by Haute Autorité de Santé, France; Dr C. Thomeczek, German Agency for Quality in Medicine, Joint Institution of The German Medical Association and the National Association of the Statutory Health Insurance Physicians representing Prof. G. Ollenschläger, Germany; MD, PhD R. Sunõl, Avedis Donabedian Foundation, Spain.

References

View Abstract