Objective. To describe the background, history, and approach of the OECD Health Care Quality Indicators (HCQI) Project, an initiative to implement quality measures for international benchmarking of medical care at the health system level.
Method. The participating countries and international organizations selected five priority areas (cardiac care, diabetes, mental health, patient safety, and primary care/prevention) and developed a conceptual framework to guide the project. International expert panels were formed to identify clinically important, scientifically sound, and feasible measures based on a structured consensus process.
Results. The consensus process was successfully completed in all five priority areas leading to a recommendation of 86 indicators. Nine indicators were selected for diabetes, 12 for mental health, 17 for cardiac care, 21 for patient safety, and 27 for primary care and prevention.
Conclusions. The initial experience of the HCQI Project demonstrates that international consensus can be achieved in how to measure the quality of care in priority areas, suggesting substantial demand for and interest in comparative information at the health system level. However, much additional work remains necessary before the project can supply policymakers and researchers with ongoing, comprehensive, and reliable data on the quality of care in industrialized countries.
quality of care
Monitoring and improving the quality of care has become a priority issue for policymakers, along with ensuring appropriate access to care and controlling cost. Both egregious incidents of poor practice and systematic evidence across many countries that the quality of care is often much lower than optimal have raised questions about the underlying causes of deficient care and changed ideas about accountability . As a consequence, many countries have begun to introduce reforms in the area of quality monitoring and improvement with the goal of making health care predictably safer and more effective. These reforms have gained momentum because of concerns about the high cost of medical care for individuals and the impact on national economies. Although it is still controversial, many have argued that well-selected and effectively implemented quality improvement interventions can reduce costs by making the delivery of care more efficient .
The increased interest in measuring and reporting the quality of care has heightened efforts to develop quality indicators that can assess quality performance at multiple levels of the health care system, such as care provided by individual physicians or physician groups, hospitals, health plans, regions, and even countries. But, while much progress has been made in tracking and reporting quality at the level of institutional providers such as for hospitals and health plans, published international health data sets, such as Organization for Economic Co-operation and Development (OECD) Health Data, currently lack corresponding measures for national health systems. Comparative research on the international level has thus been confined to comparisons of health status indicators, such as mortality rates, which are regarded more as measures of overall societal achievement rather than the performance of the medical sector. Consequently, work on quality indicators that can be used for international comparisons constitutes an important step toward evidence-based health care assessment and quality improvement.
The OECD is an intergovernmental economic research institution headquartered in Paris, France, with a membership of 30 developed countries sharing a commitment to democratic government and the market economy. Its Health Care Quality Indicators (HCQI) Project is attempting to bridge this gap by bringing together 23 OECD countries, international organizations, such as the World Health Organization (WHO) and the European Commission (EC), expert organizations such as the International Society of Quality in Healthcare (ISQua) and the European Society for Quality in Healthcare (ESQH), and several universities, to work on the development and implementation of quality indicators at the international level. This article describes the history of the project, the approach taken, and the progress made with a particular focus on the work of the international expert panels in identifying clinically meaningful and comparable measures for the quality of care in five priority areas, which is the subject of this Special Issue of the Journal.
Before the inception of the HCQI Project, two smaller international collaborations, one sponsored by the Commonwealth Fund of New York, a charitable foundation (including representatives from Australia, Canada, New Zealand, the UK, and the US), and the other set up by the Nordic Council of Ministers (including Denmark, Finland, Iceland, Norway, and Sweden), established efforts to identify quality indicators that might be used successfully for international comparisons . Other measurement activities had been launched by the WHO , the WHO Regional Office for Europe , and the EC through its Eurocare  and European Community Health Indicators (ECHI) projects .
In an attempt to build on these efforts and on other national initiatives, the OECD Secretariat proposed to the two existing international collaborations and all other member countries that it launch the HCQI Project as a joint endeavor. The two collaborations, their participants, and nine additional countries (Austria, Germany, France, Ireland, Japan, the Netherlands, Portugal, Spain, and Switzerland) accepted this invitation, leading to the formal launch of the HCQI Project in January 2003. In addition, the EC, the WHO, ISQua, and ESQH were invited to participate. Later, four additional OECD countries (the Czech Republic, Italy, Mexico, and the Slovak Republic) joined the project, bringing the participation to 23 of the 30 OECD countries.
The project initiated its work with two major activities. The first was an effort to build on the set of indicators proposed by the two existing international collaborations. With additions from the HCQI participants, a pilot project was undertaken to demonstrate the feasibility of collecting internationally comparable data that could be released publicly. An initial list of indicators and corresponding data have recently been released . The second activity, and the subject of this Special Issue, was to specify priority areas for additional indicator development and to identify specific quality indicators for those priority areas that would be most appropriate for examination at an international level. Experts from each participating country were asked to nominate two topics that they regarded as having the greatest clinical importance and policy relevance. The six areas that received the most nominations were selected: cardiac care, diabetes mellitus, mental health, patient safety, primary care, and prevention/health promotion. Later on, primary care and prevention/health promotion were consolidated into a single topic, as it seemed too difficult to define a boundary between them.
The OECD Secretariat was tasked to convene international expert panels to identify, review, and evaluate indicators for the five selected areas. The proceedings of those expert panels, describing their decision process and recommendations, are the subject of five articles in this Special Issue. In addition to the work on the priority areas, countries had asked the OECD Secretariat to draft a conceptual framework to guide the work under this project. A final framework was developed in discussion with participating countries and is also described in this Special Issue. Having converged on a set of indicators for each of the priority areas, the OECD Secretariat is currently investigating the availability and validity of required data.
After selection of the priority areas, the OECD Secretariat asked the participating countries and institutions to nominate experts to conduct the indicator review and selection process in the five priority areas. Four expert panels with broad-based international representation were formed for the areas of cardiac care, mental health, patient safety, and prevention/primary care. For diabetes, it was decided that prior consensus formation processes had already led to convergence on a limited set of quality measures, and two researchers—one involved in the US consensus process and the other involved in the European consensus process—were asked to develop a recommendation without a formal panel vetting process.
Because of constraints on time and resources, the panels were charged with reviewing and evaluating existing quality indicators or sets of indicators rather than undertaking development of indicators de novo. The OECD Secretariat issued a call for measures to the participating countries and expert panel members and compiled lists of existing quality indicators from original work undertaken by many leading organizations in the relevant fields. Only measures that are applicable to developed countries were requested, as the OECD membership represents only developed market economies. The panelists were instructed to base their evaluation of specific indicators on the criteria proposed by the US Institute of Medicine: relevance, scientific soundness, and feasibility . Relevance reflects the impact of the construct captured by a quality measure on persons’ mortality, morbidity, and functioning and in some cases the cost of care. It also reflects the ability of the health system to influence performance as assessed by the quality measure. The scientific soundness of an indicator is judged by the evidence of its reliability and validity, including the appropriateness of risk adjustment. Feasibility depends on data availability and comparability as well as the likely administrative and financial burden for provider organizations and national bodies that collect and report data. Each expert panel evaluated the identified indicators with a structured consultation process using the modified Delphi method (originally developed by the RAND Corporation) .
Panelists were asked to base their assessment of relevance and scientific soundness on documentation provided by the OECD Secretariat, and their own sources and personal knowledge of the field, and to provide an explicit rating of the measures on a scale from 1 to 9 for each dimension. Ratings of 7–9 indicated support of a measure, ratings of 4–6 ambiguity, and ratings of 1–3 rejection. Panelists had only limited knowledge about data availability and comparability across the range of OECD countries, so they were directed to make a qualitative assessment of probable feasibility. The panels were allowed to define their own decision criteria for adoption or rejection of a measure based on the ratings. Typically, the panelists decided to retain measures with high ratings (7 and above) on both relevance and soundness, and also measures with intermediate scores on those dimensions if data collection was thought likely to be feasible. Details are described in each panel report.
Each panel developed a conceptual framework for their respective priority area to ensure that the final selection provided coverage of the key domains. After having completed their review, the panelists drafted recommendations that characterized the different measures [11–⇓⇓⇓15]. As customary in the OECD decision process, those recommendations were initially reviewed by technical experts from the countries participating in the HCQI Project and then by an advisory body with representatives from all OECD member countries. Both bodies agreed with the initial recommendations of the expert panels but emphasized that a final decision could only be made after investigating data availability and comparability specific to each participating country. It was also decided by the advisory body that the overall number of indicators that would finally be adopted should not exceed 50, and the OECD Secretariat was tasked to ensure that a balanced, complete, and parsimonious set guided by the project framework be selected within this ceiling.
The consensus process was successfully completed in all five priority areas leading to a recommendation of 86 indicators. Nine indicators were selected for diabetes, 12 for mental health, 17 for cardiac care, 21 for patient safety, and 27 for primary care and prevention.
Overall, the members of the panels concluded that the selected measures provided for adequate coverage of the key domains within the five different priority areas. Nevertheless, the selected indicator lists fell short of ideal for at least two reasons. First, the expert panels were confined to choosing among existing quality indicators, and consequently, important domains in which quality indicators are known to be deficient are not fully represented. This affected, for example, rapidly changing, high-end technologies (such as cardiac assist devices and cardiac transplantation in the cardiac care area), subpopulations of particular interest (such as mental health care quality in children), and new measurement concepts (such as overall cardiovascular risk in the primary care and prevention area). The second significant constraint was related to the feasibility of data collection in multiple countries. To illustrate, an indicator for annual foot examinations in diabetes was rejected because of concerns about data availability, even though it reflects an important care practice. Similarly, the Cardiac Care Panel decided not to select indicators for unstable angina, as the panelists were concerned about differences in ascertainment across countries. The Mental Health Panel restricted its selection to indicators with operational definitions that are based on internationally accepted classification systems, such as the International Classification of Diseases and the Diagnostic and Statistical Manual of Mental Disorders to avoid issues with differences in clinical definitions across countries. Even for the selected indicators, it is presently unknown whether comparable data exist in a sufficient number of OECD countries.
The experience of the expert panels of the HCQI Project described in this Special Issue demonstrates that consensus can be achieved internationally in how to measure the quality of care in various priority areas. All five panels were able to formulate a consensus recommendation for a conceptual framework and to identify a set of quality indicators that proved acceptable to the technical experts from the countries participating in the HCQI Project and the broader advisory body representing all OECD member countries. Perhaps this should not come as a surprise in the areas of cardiac care and diabetes where widely accepted standards for quality measurement exist. But consensus on a preliminary conceptual framework and initial indicator sets for patient safety, mental health, and primary care and prevention was achieved more easily and quicker than one might have expected, given the state of existing measurement research.
The success of the HCQI indicator development work lays the groundwork for the project’s future. The project started as a limited effort to demonstrate the feasibility of compiling comparative information on the quality of care across different health systems. The effort quickly drew in almost all OECD countries as active participants. In 2005 and 2006, interest and funding for the project increased and major international health organizations and additional countries joined the effort.
Although the project has been successful in its initial steps, future work on quality indicators will be much more challenging. Thus far, progress has been made mainly because of opportunism and the voluntary efforts undertaken by participating countries to provide already available data. Even with the newly committed funding, the project leadership still lacks the resources for either the development of new quality indicators or, perhaps more importantly, dedicated data collection. As the reports of the expert panels in the Special Issue show, substantial gaps in measurement remain in the five priority areas that would require developmental work. And the efforts undertaken so far to collect data for the initial list of indicators reveal substantial problems with data availability and comparability as well .
Sooner or later, OECD member countries will have to confront the question of whether the project should be expanded to encompass ongoing indicator development and dedicated data collection. Achieving the full potential of international quality performance reporting will surely require commitment of resources at a scale that is considerably higher than has heretofore been made available. A comparable endeavor in a different field that could serve as a model is currently maintained by the OECD Education Directorate that develops measures for educational attainment, collects data, and publishes benchmark reports as part of the Programme for International Student Assessment (PISA). Obviously, such an effort would have to be coordinated with national activities and activities of other international organizations, such as the EC and the WHO.
To conclude, the OECD HCQI Project has revealed substantial interest in information on the quality of care that can be used to compare the performance of different health systems. The conceptual framework, described in this Special Issue, provides a road map that can guide further progress. But much additional work remains necessary before this effort can supply policymakers and other stakeholders with ongoing, comprehensive, and reliable data on quality of health care in industrialized countries.
At the time this work was conducted, Dr Mattke was the head of the HCQI Project at the OECD Secretariat, Dr Epstein the chair of the International Expert Group, and Professor Leatherman an Adviser to the project and co-chair of the Primary Care and Prevention Panel. This article reflects the opinion of the authors and not an official position of the OECD, its member countries or institutions participating in the project.
MattkeS. Monitoring and improving the technical quality of medical care: a new challenge for policymakers in OECD countries. In Towards High Performing Health Systems. Paris: Policy Studies, OECD, 2004.
GreenfieldS, Nicolucci A, and Mattke S. Selecting Indicators for the Quality of Diabetes Care at the Health Systems Level in OECD Countries. OECD Health Technical Paper No. 15. Available at http://www.oecd.org/dataoecd/28/34/33865546.pdf Accessed 19 May2006.
HermannR, Mattke S, the members of the OECD Mental Health Care Panel. Selecting Indicators for the Quality of Mental Health Care at the Health Systems Level in OECD Countries. OECD Health Technical Paper No. 17. Available at http://www.oecd.org/dataoecd/53/26/33878001.pdf Accessed 19 May2006.
LambieL, Mattke S, the members of the OECD Cardiac Care Panel. Selecting Indicators for the Quality of Cardiac Care at the Health Systems Level in OECD Countries. OECD Health Technical Paper No. 14. Available at http://www.oecd.org/dataoecd/28/35/33865450.pdf Accessed 19 May2006.
MarshallM, Leatherman S, Mattke S, the members of the OECD Health Promotion, Prevention and Primary Care Panel. Selecting Indicators for the Quality of Health Promotion, Prevention and Primary Care at the Health Systems Level in OECD Countries. OECD Health Technical Paper No. 16. 2004. Available at http://www.oecd.org/dataoecd/27/52/33865865.pdf Accessed 19 May 2006.
MillarJ, Mattke S, the members of the OECD Patient Safety Panel. Selecting Indicators for Patient Safety at the Health Systems Level in OECD Countries. OECD Health Technical Paper No. 18. Available at http://www.oecd.org/dataoecd/53/26/33878001.pdf Accessed 19 May2006.