International Journal for Quality in Health Care Advance Access originally published online on December 13, 2006
International Journal for Quality in Health Care 2007 19(1):1-3; doi:10.1093/intqhc/mzl066
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Editorial |
All or none measurement: why we know so little about the comprehensiveness of care
In a recent article published in JAMA, ‘All or None Measurement Raises the Bar on Performance’, Nolan and Berwick describe the methods used by the Institute for Healthcare Improvement (IHI), the US National Healthcare Reports, the Joint Commission on Accreditation of Healthcare Organizations, the National Committee for Quality Assurance, and other organizations to examine care across a set of measures [1]. This approach, to state it simply, looks at data at the patient level to examine whether a ‘bundle’ or set of evidence-based interventions that are supposed to be done for the patient are actually done for the patient. The all or none approach, as Berwick and Nolan point out, has the tremendous advantage of offering a more complete picture of care across a range of aspects of quality for specific conditions. However, there are a number of practical limitations to this method which have been inadequately explored in the literature. Based on experience at the Agency for Healthcare Research and Quality (AHRQ) in the US National Reports, this approach has tremendous potential. However, until further research is done on the impact on outcomes of receiving ‘all’ recommended interventions and until consensus is achieved on the interventions that ‘all’ patients with a given condition should receive, the potential of this method will remain unrealized.
When AHRQ applied this approach for work on the US National Healthcare Reports, first in the area of diabetes and then in the area of respiratory disease and cardiac care, the explicit goal was to try and move the discussion of health care quality away from the indicator-specific debates that too often dominate the health care debate in scientific circles in America. In devising the first US National Healthcare Reports, the AHRQ team received nominations for over 700 potential measures to be used in the initial measure set. A rigorous, two-year process of reviewing, rating, and reducing the measure set resulted in an initial measure set for the first National Healthcare Quality Report (NHQR), released in January 2003 of 
150 measures and of the National Healthcare Disparities Report of 250 measures. ‘Composite’ measures that summarize care across a set of recommended interventions such as the ‘all or none’ composites can be one tool to make sense of these measures mania.
Key limitations of ‘all or none’ measurement
In letters to the editors of JAMA and in an innovative new program by IHI and JAMA funded through the Robert Wood Johnson Foundation, ‘Author in the Room’, the limitations of the ‘all or none’ approach have been discussed. Specifically, these discussions focused on the mathematical limitations of simple counts across a set of measures. These discussions are helpful, as it is true that, in some ways, the ‘all or none’ approach is somewhat contrived in that one would expect performance to decrease as each successive indicator is added to set of ‘all’ measures. However, in all the otherwise thoughtful discussion, there are two primary problems with using ‘all or none’ measurement for national, state, or even local health system monitoring that have been ignored. The first is that the tide of new measures is inexorable. As with other measure efforts in health care, such as the National Quality Forum’s consensus measure sets and the Physician Consortium’s measure sets [2,3], it is rare that even large measure sets such as those for the US National Reports are reduced. Typically, they are only increased as new science give us new aspects of care to measure. In many cases, these new measures may be desirable, as they are developed to fill gaps in our knowledge. This is the case in under-studied areas such as mental health care (whose primary measure in the first US National Reports was the very inadequate ‘national suicide rates’) or coping with end of life. This has the added effect of changing the picture of performance when new measures are developed (or more rarely, when obsolete measures are ‘retired’ from measure sets). This is the case when you look at data presented in the US National Healthcare Reports. In the first US NHQR, AHRQ reported on five process measures of diabetes care quality as agreed to by the National Diabetes Quality Improvement Alliance. AHRQ reported that only 23.1% of diabetes patients received all of these recommended tests. In the 2005 NHQR, data limitations were such that only three, rather than five, of the recommended tests could be analyzed together. This meant that reporting on all five of these recommended tests was not feasible, despite the fact that the clinical guidelines have not changed regarding these measures. The resulting summary measure shows that 52.7% of diabetic patients reported receiving their HbA1c test, foot exam, and retinal eye exam. While still very useful, this gives a very different picture of overall performance as shown in Figure 1.
|
The second, possibly more important, problem with the ‘all or none’ method is that is completely reliant on broad national consensus across the clinical and scientific communities about what are the key aspects of care that all patients with a given disease should get. To date, there have been very few interdisciplinary efforts to generate such sets of measures. The AMA’s Physician Consortium and the National Diabetes Quality Improvement Alliance and the JCAHO’s measure sets in hospital-based cardiac care and pneumonia care are two such efforts. Other existing avenues such as the National Quality Forum move this agenda forward by creating consensus measure sets. However, many of the NQF’s measure sets are designed to assess interventions that all patients with a given condition should get. In most of the clinical condition areas considered high priority by the US Department of Health and Human Services or by the Institute of Medicine [4–7], there is no consensus on the essential measures that would make up the ‘all or none’ picture of health care delivery. In addition to the difficulty of gaining consensus on interventions that are appropriate for all patients with a given condition, there is the ongoing issue that no two patients are ever exactly alike—even if they present with the same condition. For example, while there is broad agreement in areas like in-patient care for pneumonia about the range of diagnostic and curative interventions that patients should receive (and even within what time period), three is a lack of agreement that every patient should receive every intervention. In particular, there has been disagreement on the need for taking blood cultures before ordering antibiotics [8]. In the US NHQR, data was presented with and without this indicator, and the ‘story’ that these ‘all or none’ set of measures tells is very different based on whether the measure is included or not. Figure 2 shows these findings.
|
It is clear that there are major advantages to the use of ‘all or none’ measurement as one of a range of tools for reporting on health care quality. The limitations noted above are discussed to try and improve the use of ‘all or none’ measurement in national and local reporting of health care quality. Some of these limitations are relevant to summary measures in general. Like all summary measures, the all or none approach provides a rapid ‘dashboard’ look at performance, but it is limited in its ability to provide specific guidance on improving care. That must be done in disaggregating the all or none indicator. However, despite these limitations, it is clear that more—not less—work is needed to improve our ability to tell a concise as well as accurate story of how the US health system performs in terms of quality of care and whether quality is improving or not. An approach such as the ‘all or none’ approach that broadens the focus beyond individual measures to better approximate the experience of patients with given conditions is a positive addition to set of measurement tools available to scientists and policy makers. It is also clear that more work is needed to continue to refine this approach to keep it consistent with evidence-based guidelines. More importantly, our examination shows that more consensus building is needed on what is important to measure for given conditions (and given comorbid conditions). We then need data systems that support that clinical consensus. By this, I mean we need data systems that allow us to track patients across care episodes and care settings—much as the Serveillance, Epidemiology and End Results (SEER) Cancer Registries run by the National Cancer Institute do. Until we generate that consensus, led by both the clinical and scientific community, we will never really know how comprehensive our health care is.
Head, Health Care Quality Indicators Project, Organization for Economic Cooperation and Development and Director, US National Healthcare Reports, US Agency for Healthcare Research and Quality, 540 Gaither Road, Rockville, MD 20850, USA E-mail: edward.kelley{at}ahrq.hhs.gov
References
- Nolan T, Berwick D. All-or-none measurement raises the bar on performance. JAMA 2006; 295 (10): 1168–1170.
[Free Full Text] - National Quality Forum. Consensus Measure Set publications. http://www.qualityforum.org/publications/reports/Accessed 17 November 2006.
- American Medical Association. Physician Consortium for Performance Improvement: Consortium Measure Sets. http://www.ama-assn.org/ama/pub/category/4837.html Accessed 17 November 2006.
- Joint Commission on Accreditation of Healthcare Organizations. Performance Measurement Initiatives: Core Measure Set Information. http://www.jointcommission.org/PerformanceMeasurement/PerformanceMeasurement/default.htm Accessed 17 November 2006.
- US Department of Health and Human Services. Healthy People 2010 Fact Sheet. http://www.healthypeople.gov/About/hpfact.htm Accessed 17 November 2006.
- US Department of Health and Human Services. Agency for Healthcare Research and Quality. The 2005 National Healthcare Quality Report. AHRQ: Rockville, MD, 2005.
- Institute of Medicine. Priority Areas for National Action: Transforming Health Care Quality. Washington, DC: The National Academies Press, 2003.
- Ramanujam P, Rathlev NK. Blood cultures do not change management in hospitalized patients with community-acquired pneumonia. Acad Emerg Med 2006; 13 (7): 740–745. [Epub 9 June 2006].[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||