Skip Navigation

International Journal for Quality in Health Care 2006 18(Supplement 1):14-20; doi:10.1093/intqhc/mzl030
This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by McLoughlin, V.
Right arrow Articles by Bates, D.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by McLoughlin, V.
Right arrow Articles by Bates, D.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?

International Journal for Quality in Health Care September 2006 © The Author 2006. Published by Oxford University Press on behalf of International Society for Quality in Health Care; all rights reserved

Selecting indicators for patient safety at the health system level in OECD countries

Vivienne McLoughlin1, John Millar2, Soeren Mattke3, Margarida Franca4, Pia Maria Jonsson5, David Somekh6 and David Bates7

1 The Health Foundation, London, UK, 2 (Chair), Canadian Institute for Health Information, Toronto, Canada, 3 OECD Health Policy Unit, Paris, France, 4 Instituto da Qualidade em Saude, Lisbon, Portugal, 5 National Board of Health and Welfare, Stockholm, Sweden, 6 European Society for Quality in Healthcare. London, UK, and 7 Harvard Medical School and Brigham and Women’s Hospital, Boston, MA, USA

Background. Concerns about patient safety have arisen with growing documentation of the extent and nature of harm. Yet there are no robust and meaningful data that can be used internationally to assess the extent of the problem and considerable methodological difficulties.

Purpose. This article describes a project undertaken as part of the Organization for Economic Cooperation and Development (OECD) Quality Indicator Project, which aimed at developing an initial set of patient safety indicators.

Methods. Patient safety indicators from OECD countries were identified and then rated against three principal criteria: importance to patient safety, scientific soundness, and potential feasibility. Although some countries are developing multi-source monitoring systems, these are not yet mature enough for international exchange. This project reviewed routine data collections as a starting point.

Results. Of an initial set of 59 candidate indicators identified, 21 were selected which cover known areas of harm to patients.

Conclusions. This project is an important initial step towards defining a usable set of patient safety indicators that will allow comparisons to be made internationally and will support mutual learning and quality improvement in health care. Measures of harm should be complemented over time with measures of effective improvement factors.

Keywords: patient safety, quality indicator

Address reprint requests to Dr Vin McLoughlin, Director of Programmes and Policy, The Health Foundation, 90 Long Acre, Covent Garden, London WC2E 9RA, UK. E-mail: vin.mcloughlin{at}health.org.uk

Accepted for publication July 14, 2006.

The objective of the Organization for Economic Cooperation and Development (OECD) Health Care Quality Indicator (HCQI) [1] Project is to develop a set of indicators to describe the state of the quality of health care that can be reliably reported across countries using comparable data. The project would be incomplete without the indicators of ‘patient safety’, as there is increasing recognition of error and lack of reliability in biomedical interventions [2].

This article describes the emergence of the supportive science in patient safety and the problems that beset measurement and presents a set of indicators. These were selected by an international panel of government representatives and experts from Australia, Canada, the EU, the OECD, Portugal, Spain and the United States.


   Is patient safety a distinct domain of quality?
Top
 Is patient safety a...
Background
 Materials and methods
 Results
 Discussion
 Acknowledgements
 References
 
Safety is a basic tenet of high health care quality, yet it is recognized as a distinctive domain in recent national reports on quality [3,4]. The patient safety domain assumes a reasonable consensus about the effectiveness of treatment and focuses on whether treatments have been delivered safely. Where ‘slips and lapses’ [5] occur across treatment modalities, their impact is not always readily detected using methods designed to compare relative effectiveness.

The term ‘patient safety’ encompasses harm to patients, incidents that may give rise to harm, the antecedents or processes that increase the likelihood of incidents, and the attributes of organizations that help guard against harm and enable rapid recovery when risk escalates.

The Institute of Medicine report [6] defined patient safety as ‘the freedom from accidental injury due to medical care or from medical errors’. Well-known examples include wrong-site surgery or lethal dose administration. Yet the public view patient safety much more broadly. High-profile failures [7] occur where service outcomes are believed to be significantly below an acceptable standard. Media coverage of health care acquired infections, errors of medication, and blood product use has increased. Some argue that criminal acts committed during medical care such as homicide, rape, and abduction should be excluded [8], as they involve deliberate violations of a rule or standard of behaviour.

The National Patient Safety Agency (NPSA) in the United Kingdom describes patient safety as ‘the process by which an organization makes patient care safer’ [9]. This involves risk assessment, the reporting and analysis of incidents, and the capacity to implement solutions to minimize the risk of recurrence. A patient safety incident is ‘any unintended or unexpected incident that could have or did lead to harm ...’ [10]. This excludes effects recognized as a risk of treatment. Some incidents include medical errors in planning or execution. The Institute of Medicine defines medical error as ‘the failure of a planned action to be completed as intended or the use of a wrong plan to achieve an aim’ [11].

Without a specific focus on patient safety, strategies to reduce and protect patients from accidental injury may not be developed.


   Background
Top
 Is patient safety a...
 Background
Materials and methods
 Results
 Discussion
 Acknowledgements
 References
 
As early as 1991, the Harvard Medical Practice Study documented an alarming incidence of medical errors [12]. Designed to investigate medical negligence, it identified 3–4% of hospital admissions as associated with an adverse event. But public awareness of the importance of patient safety first occurred in 1999 with the publication of To Err is Human [2]. This report estimated that adverse events arising from health care were the third leading cause of death in the United States [13]. Other countries replicating the Harvard study found similar error rates [12,14–20]. These findings brought patient safety to the top of the policy agenda internationally. In 2000, the Australian Council for Safety and Quality in Health Care (http://www.safetyandquality.org) was established in response to concerns about safety. In the United Kingdom, the Chief Medical Officer’s report in 2000, An Organisation with a Memory [21], documented the need to pay serious attention to patient safety. In 2004, the World Health Organization launched its World Alliance for Patient Safety (http://www.who.int/patientsafety/en) to ensure that the learning about safety is shared rapidly internationally.

Out of the policy interest, science for understanding and improving safety emerged with a massive rise in patient safety research over the last decade [22]. Much of this consists of research in psychology, sociology, ergonomics, and organizational risk management [23,24]. It reflects the learning from other industries that protocols, culture, equipment, and design are an integral part of making care safer.

Parallel efforts were started to introduce measurement systems although few countries have had extensive experience in developing patient safety indicators and fewer still have been able to generate national data on patient safety. Table 1 describes national measurement activities occurring in different countries.


View this table:
[in this window]
[in a new window]
  		Table 1 National measurement activities across countries

 
These activities in turn illuminate the considerable problems of definition and measurement in this field [25]. Routine hospital administrative data collection is designed for other purposes such as billing and operational management and may not paint a comprehensive picture of safety and quality problems. Adverse events studies rely on medical chart reviews but are expensive and provide only a snapshot of the extent and nature of incidents. The accuracy of these estimates has been widely debated [26]. Some countries have introduced national incident monitoring systems to capture ‘near miss’ incidents, but there is significant under-reporting [27–29]. Although there is still no internationally accepted taxonomy for patient safety incidents [30], some countries have introduced mandatory reporting of sentinel events or confidential national reporting schemes [31]. Information is used from many disparate sources such as coroners’ reports, confidential enquiries into mortality, routine scrutiny of surgical or all deaths, complaint processes, medical insurance data sets, patients, and staff surveys [31]. Progress in this area has been slow [6], but there are initiatives underway that show some promise [10]. This project attempts to further the design and implementation of patient safety monitoring systems by seeking international consensus of which measures to include.


   Materials and methods
Top
 Is patient safety a...
 Background
 Materials and methods
Results
 Discussion
 Acknowledgements
 References
 
The time and resource constraints on the project dictated that only existing indicators could be reviewed rather than new indicators be developed. The team of the OECD HCQI Project compiled lists of existing safety indicators from existing measurement and reporting systems in OECD countries. The sources for the indicators are summarized in Tables 1 and 2.


View this table:
[in this window]
[in a new window]
  		Table 2 Indicator sources

 

The set contained a total of 59 candidate indicators that were then reviewed for the following:

  1. their importance to patient safety—severity of impact, robustness of link to a failure of implementation or planning; the extent of public concern; link to attributes capable of influence by services; and sensitivity to policy changes;
  2. their scientific soundness—clinical face validity, content validity, and;
  3. their potential feasibility—likely data availability across countries; and estimated reporting burden.

After a structured review process developed by the RAND Corporation, each panel member ranked each of the 59 candidate indicators on a scale of 1–9 for the importance and scientific soundness criteria [32]. As panellists had only limited information about data systems in all OECD countries and thus feasibility, they were only asked to rate feasibility in the three broad categories—likely, possible, and unlikely—based on their previous knowledge. Ratings were summarized and measures of disagreement calculated. The panel discussed the ratings by teleconference and then assigned a final rating for each.

All indicators with a final median score of >7 for both importance and scientific soundness and at least ‘likely’ feasibility were accepted. All indicators with a median score ≤5 for importance or scientific soundness were rejected. The remaining indicators with median ratings of 6 or 7 were selected or rejected on a case-by-case basis through panel discussion.


   Results
Top
 Is patient safety a...
 Background
 Materials and methods
 Results
Discussion
 Acknowledgements
 References
 
The final list of 21 indicators selected is summarized in Table 3. Five domains emerged: hospital acquired infections, operative and post-operative complications, sentinel events, obstetrics, and other care-related events. Infection due to medical care, of wounds, pneumonia in patients on ventilators, and decubitus ulcer were selected, as were technical difficulty in operations, complications of anaesthesia, post-operative pulmonary embolism and deep vein thrombosis, sepsis, and hip fracture. Sentinel events included wrong-site surgery and retaining foreign bodies, transfusion, equipment, or medication errors. Injury to the neonate or to the mother was selected in obstetrics, and other care-related areas included patient falls.


View this table:
[in this window]
[in a new window]
  		Table 3 Final list of patient safety indicators

 

Detailed descriptions of these indicators with operational definitions can be found in the OECD publication: Selecting Indicators for Patient Safety at the Health Systems Level in OECD Countries [33].


   Discussion
Top
 Is patient safety a...
 Background
 Materials and methods
 Results
 Discussion
Acknowledgements
 References
 
The 21 indicators that the panellist selected assessed clinically important patient safety events that are widely perceived to be indicative of lapses in care such as procedural complications, child birth trauma, and medication error. The measures, selected by an international expert panel using a structured process, represent progress towards the goal of identifying consensus-based measures for international benchmarking of patient safety. However, this panel considers the set only as a preliminary step in a long-term process towards implementing a measurement system for patient safety on the health system level. There are four major challenges to this set: two of a conceptual nature and two related to the feasibility of the measures.

The first conceptual limitation is that the set is restricted to manifest adverse events and leaves out indicators for near misses and also for adherence to safe care processes. The second is that the indicators focus on hospital care, leaving out important areas such as primary care, mental health, nursing home care, and self-care. Both limitations are a result of the restriction of this review to existing indicator sets, which have traditionally focused on adverse outcomes of in-patient care.

The first feasibility problem will be data availability. There are strong disincentives for disclosing safety incidents, such as fear of litigation or shame, so that few dedicated reporting systems exist. Even where there are incident monitoring systems that capture adverse events and/or ‘near miss’ events, events may be under-reported. Alternative data sources also have limitations. Administrative data commonly lack the detail and completeness to capture such events, and the use of chart reviews is limited by cost concerns. The second feasibility problem is that the set includes events that are very rare and that may also be under-reported in administrative data, which are commonly used to identify those events. Thus, even at the health system level, the events might not occur with a sufficient frequency and variance to use them to draw inferences about differential safety of care. Key issues for each indicator are summarized in Table 4.


View this table:
[in this window]
[in a new window]
  		Table 4 Selected indicators, key data issues, and capacity to influence

 

Implications of this work
The issues of definition and measurement should benefit from the World Alliance for Patient Safety streams of work on classification, measurement, and reporting over the next few years (http://www.who.int/patientsafety/en). Where electronic data systems are not available, more labour-intensive techniques are needed such as critical incident reporting systems, the use of administrative databases, retrospective randomized chart audits, patient recall, incident reporting systems, and simulations. Even electronic health records do not always contain the fields to adequately capture patient safety data. Data collected internationally are unlikely to be useful for longitudinal analysis in the short term. Reporting levels are likely to vary significantly across countries and over time.

The adoption and use of these indicators may help raise the awareness of harm. Without them, the general public will not be adequately informed or empowered to play a role in improving patient safety.

Promoting continuous case note reviews and incident reporting at the local level will remain the bulwark against safety improvement. It could also lead to the improvement of routinely collected large data sets.

Beyond clinical indicators, indicators need to reflect other patient safety issues such as the national and local infrastructure that exists to promote learning and aspects of organizational performance. There are several typological and dimensional tools available to measure safety culture [9]. This could also include measurement of the implementation of effective safety practices—e.g. the Institute for Healthcare Improvement has designed a range of measures that enable hospitals to track their compliance with safety practices for The Health Foundation’s Safer Patient Initiative (personal communication).

In summary, patient safety is now recognized as a key dimension of health care quality. The OECD has selected an initial set of indicators to encourage further development. It is hoped that this will support international cooperation in improving health care quality around the world.


   Acknowledgements
Top
 Is patient safety a...
 Background
 Materials and methods
 Results
 Discussion
 Acknowledgements
References
 
This article describes the key recommendations made by the Patient Safety Panel of the OECD Health Care Quality Indicators Project. It reflects the opinion of the authors and not an official position of the OECD, its member countries, or institutions participating in the project. The full report of the panel proceedings can be found at www.oecd.org/dataoecd.


   References
Top
 Is patient safety a...
 Background
 Materials and methods
 Results
 Discussion
 Acknowledgements
 References
 

  1. Mattke S, Epstein AE, Leatherman S. The OECD Health Care Quality Indicators Project: History and Background. 2006

  2. Kohn LT, Corrigan JM, Donaldson MS, eds. To Err is Human. Washington, DC: Institute of Medicine, The National Academies Press, 1999.

  3. Leatherman S, Sutherland K. The Quest for Quality in the NHS. A Chartbook on Quality of Care in the UK. London: The Nuffield Trust, 2005.

  4. Australian Council for Safety and Quality in Healthcare (ACSQHC). Charting the Safety and Quality of Health Care in Australia. Canberra: ACSQHC, 2004.

  5. Reason J. Human error: models manage. BMJ 2000; 320: 768–770.[Free Full Text]

  6. Institute of Medicine. To Err is Human: Building a Safer Health System. Washington, DC: The National Academies Press, 2000.

  7. Bristol, King Edward (Australia) Winnipeg. Learning from Bristol: the report of the public inquiry into children’s heart surgery at the Bristol Royal Infirmary 1984–1995 Command Paper 5207 presented to the UK Parliament in July 2001; Australian Council for Safety and Quality in Healthcare 2002. Lessons from the Inquiry into Obstetrics and Gynaecology services at King Edward Memorial Hospital. 1990–2000; Provincial Court of Manitoba, Canada. The report of the Manitoba Pediatric Cardiac Surgery Inquest. An Inquiry into twelve deaths at the Winnipeg Health Sciences Centre in 1994.

  8. Fallowfield L, Fleissig A. Communication with Patients in the Context of Medical Error, 2003: http://www.pcpoh.bham.ac.uk/publichealth/psrp/pdf/fallowfield_comunication_final_report.pdf Accessed 16 August 2006.

  9. National Patient Safety Agency (NPSA). Seven Steps to Patient Safety: A Guide for NHS Staff. London: NPSA, 2003.

  10. National Patient Safety Agency (NPSA). Building a Memory: Preventing Harm, Reducing Risks and Improving Patient Safety. London: NPSA, 2005.

  11. Institute of Medicine. Patient Safety: Achieving a New Standard for Care. Washington, DC: The National Academies Press, 2004.

  12. Brennan T, Leape L, Laird N et al. Incidence of adverse events and negligence in hospitalized patients: results of the Harvard Medical Practice Study I. N Engl J Med 1991; 324: 370–376.[Abstract]

  13. Starfield B.Is US health really the best in the world? JAMA 2000; 284: 483–485.[Free Full Text]

  14. Leape LL, Brennan TA, Laird N et al. The nature of adverse events and negligence in hospitalised patients. Results of the Harvard Medical Practice Study II. N Engl J Med 1991; 324: 377–384.[Abstract]

  15. Wilson RM, Harrison BT, Gibberd RW, Hamilton JD. An Analysis of the causes of adverse events from the Quality in Australian Health Care Study. Med J Aust 1999; 170: 411–415.[ISI][Medline]

  16. Thomas EJ, Studdert DM, Burstin HR et al., Incidence and types of adverse events and negligent care in Utah and Colarado. Med Care 2000; 200 (38): 261–271.

  17. Vincent C, Neale G, Woloshynowych M. Adverse events in British hospitals: preliminary retrospective record review. BMJ 2001; 322 (7285): 517–519.[Abstract/Free Full Text]

  18. Davis P, Lay-Yee R, Briant R, Ali W, Scott A, Schug S. Adverse events in New Zealand public hospitals I: occurrence and impact N Z Med J 2002; 13 (1167): 115.

  19. Baker GR, Norton PG, Flintoff V. Canadian adverse events study. CMAJ 2004; 171: 834.

  20. Schioler T, Lipczak H, Pedersen BL et al. Incidence of adverse events in hospitals. A retrospective study of medial records. Ugeskr Laeger 2001; 163: 5370–5378.[Medline]

  21. Department of Health. An Organisation with a Memory: Report of an Expert Group on Learning from Adverse Events in the NHS Chaired by the Chief Medical Officer. London: Stationery Office, 2000.

  22. Maillard N, Stirling S, Lilford R, Johal A, Gilbert N. Patient Safety: An Unsystematic Review and Bibliography. 2005: http://www.pcpoh.bham.ac.uk/publichealth/psrp/Pdf/patientsafetybibliography_02SEPT.pdf Accessed 16 August 2006.

  23. National Academy of Engineering and Institute of Medicine. Building a Better Delivery System: a New Engineering/Health Care Partnership. Washington, DC: The National Academies Press, 2005.

  24. Leape LL. Error in medicine. JAMA 1994; 272: 1851–1857.[CrossRef][ISI][Medline]

  25. Bruce J, Russell EM, Mollinson J, Krukowski ZH. The measuring and monitoring of surgical adverse events. Health Technol Assess 2001; 5: 22.

  26. BMJ. Rapid Responses: http://bmj.bmjjournals.com/cgi/eletters/329/7462/369 Accessed 16 August 2006.

  27. Weingart SN, Wilson RM, Gibberd RW, Harrison B. Epidemiology of medical error. BMJ 2000; 320: 728–738.[Free Full Text]

  28. Vincent C, Stanhope N, Crowley-Murphy M. Reasons for note reporting adverse incidents: an empirical study. J Eval Clin Pract 1999; 5: 13–21.[CrossRef][ISI][Medline]

  29. Lawton R, Parker D. Barriers to incident reporting in a health care system. Qual Saf Health Care 2002; 11: 15–18.[Abstract/Free Full Text]

  30. Chang A, Schyve PM, Croteau RJ, O"Leary DS, Loeb JM. The JCAHO patient safety event taxonomy: a standardized terminology and classification schema for near misses and adverse events. Int J Qual Health Care 2005; 17 (2): 92–105.

  31. National Audit Office. A Safer Place for Patients: Learning to Improve Patient Safety. UK: National Audit Office, 2005.

  32. Kerr EA, Asch SM, Hamilton EG, McGlynn EA, eds. Quality of Care for Cardiopulmonary Conditions. A Review of the Literature and Quality Indicators. Santa Monica, CA: RAND Health, 2000.

  33. Millar J, Mattke S, the Members of the OECD Safety Panel. Selecting Indicators for Patient Safety at the Health Systems Level in OECD Countries: http://www.oecd.org/dataoecd/53/26/3387800.pdf Accessed 28 July 2006.

  34. Smallwood RA. The safety and quality of health care from council to commission. Med J Aust 2006; 184 (suppl. 10): S39–40.

  35. http://www.simpatie.org. Accessed 7 July 2006.

  36. Canadian Patient Safety Institute: http://www.cpsi-icsp.ca/docs/256,1

  37. Canadian Institute for Health Information (CIHI). Health Care in Canada. CIHI, Ottawa, Canada, 2004.

  38. Danish Society for Patient Safety. Patient Safety in Denmark. Past, Current and Future Activities. 2005: http://www.im.dk/publikationer/patient_safety/patient_safety.pdf Accessed 7 July 2006.


Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us    What's this?



This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by McLoughlin, V.
Right arrow Articles by Bates, D.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by McLoughlin, V.
Right arrow Articles by Bates, D.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?