OUP user menu

OECD Health Care Quality Indicator Project. The expert panel on primary care prevention and health promotion

Martin Marshall, Niek Klazinga, Sheila Leatherman, Charlie Hardy, Eckhard Bergmann, Luis Pisco, Soeren Mattke, Jan Mainz
DOI: http://dx.doi.org/10.1093/intqhc/mzl021 21-25 First published online: 5 September 2006


Purpose. This article describes a project undertaken as part of the Organization for Economic Co-operation and Development (OECD)’s Healthcare Quality Indicator (HCQI) Project, which aimed to develop a set of quality indicators representing the domains of primary care, prevention and health promotion, and which could be used to assess the performance of primary care systems.

Methods. Existing quality indicators from around the world were mapped to an organizing framework which related primary care, prevention, and health promotion. The indicators were judged against the US Institute of Medicine’s assessment criteria of importance and scientific soundness, and only those which met these criteria and were likely to be feasible were included. An initial large set of indicators was reduced by the primary care expert panel using a modified Delphi process.

Results. A set of 27 indicators was produced. Six of them were related to health promotion, covering health-related behaviours that are typically targeted by health education and outreach campaigns, 13 to preventive care with a focus on prenatal care and immunizations and eight to primary clinical care mainly addressing activities related to risk reduction. The indicators selected placed a strong emphasis on the public health aspects of primary care.

Conclusions. This project represents an important but preliminary step towards a set of measures to evaluate and compare primary care quality. Further work is required to assess the operational feasibility of the indicators and the validity of any benchmarking data drawn from international comparisons. A conceptual framework needs to be developed that comprehensively captures the complex construct of primary care as a basis for the selection of additional indicators.

  • health promotion
  • prevention
  • primary care
  • quality indicator

This article describes the development of a set of indicators representing the domains of health promotion, prevention, and primary care. It summarizes work undertaken between 2003 and 2004 by the Primary Care Panel of the Organization for Economic Co-operation and Development (OECD)’s Healthcare Quality Indicator (HCQI) Project. The panel was composed of 11 quality-of-care researchers and policymakers from nine different OECD countries.

The importance of primary care

The term ‘primary care’ is a broad one and has different meanings in different countries [1]. It is perhaps best described in terms of function, rather than location [2]. The critical elements are a focus on the part of the providers on generalism rather than specialism; the provision of patient-centred rather than disease-centred, coordinated, and accessible services; and the integration of biomedical, psychological, and social dimensions of the presentation and management of presenting problems. Orientation to both individual and population health needs is important. So too is a focus on health promotion and disease prevention, alongside curing and caring for established health problems and the provision of palliative and end-of-life care. Continuity of care at the level of the personal and longitudinal relationship between patients and their health provider and between delivery settings is also a valued principle. In countries with a formal system of general practice, the term primary care is often confined to the functions of general practitioners and their teams, and the broader notion of public health is used to cover population-oriented preventive care and health promotion. In other countries, general practice and public health functions are more organizationally aligned.

In most countries, the provision of acute care within hospitals has tended to dominate the attention of policymakers and politicians. This reflects the historical development of health systems, the high costs associated with the provision of care in hospitals, and the profile of these services in the eyes of the public and the media. However, health care is chiefly provided outside hospitals. In the UK, for example, 95% of contacts with the National Health Service take place within primary care, 80% of these are managed without recourse to specialist services and the primary care sector consumes less than one-third of the total spending on health care.

Using international observational data, Barbara Starfield has demonstrated that countries with a strong system of primary care are more likely to have efficient health systems and better health outcomes than countries which have a strong focus on hospital services [3]. More detailed studies in individual countries have added weight to this finding, demonstrating the importance of key facets of primary care such as predictable and timely access [4], continuity of care [5], and patient-centredness [6].

Nevertheless, even countries with long-established and reputed systems of primary care have identified problems with the quality of these services. In particular, there is a growing body of evidence of wide variations in the quality of clinical care provided in the primary care sector [7,8]. Increasingly, there is sufficient clinical consensus, particularly on the technical processes of care and a limited set of patient outcomes, to make the quality of primary care amenable to measurement and improvement. For this reason, the OECD HCQI Project focused on the technical processes of clinical primary care as one of the five areas to develop quality indicators for international use and comparison.

Developing the indicator set

The conceptual framework

In line with the decision to focus on function rather than setting, the panel sought to identify indicators that capture the core components of ‘primary care’, irrespective of the settings within which these components are provided. For this reason, the panel opted for a broad definition of the term and one which goes beyond the restricted use in health systems focused on general practice. The panel considered that this would address the problem that, for example, vaccination programmes may be delivered by dedicated government agencies in one country, by private general practices in a second, and by health plans in a third. Following discussions within the panel, the core components of the broad category of ‘primary care’ were agreed as:

  1. Health promotion—the population-based strategies that target major risk factors of disease, mostly through efforts to change health-related behaviour.

  2. Preventive care—the systemic and systematically delivered population-directed services in areas such as vaccination, screening, and prenatal care.

  3. Primary clinical care—the subset of diagnostic and therapeutic activities underlying the delivery of clinical primary care.

This categorization was used as a broad organizational framework for the indicator set (Figure 1).

Figure 1

Conceptual model of the continuum of health promotion, prevention, and primary care.

Preliminary identification of potential indicators

The time and resource constraints on the project dictated that only existing indicators from around the world could be reviewed, rather than a set be developed de novo. The team of the OECD HCQI Project compiled lists of existing quality indicators relevant to the above framework from original work undertaken by a number of leading organizations in the field. These included RAND, the National Committee for Quality Assurance and the Agency for Healthcare Research and Quality in the USA, the National Primary Care Research and Development Centre in the UK, and the Canadian Institute for Health Information in Canada. Sources were identified by the OECD Secretariat on the basis of their own experience and suggestions from member countries and members of the Primary Care Panel. Many of these indicator sets, and those in use in other countries which were not used as a source for this project, have common roots in the scientific literature. Indicators for secondary and tertiary prevention of cardiovascular diseases and diabetes were evaluated by the HCQI panels on cardiac care [9] and diabetes [10]. In addition, two indicators for cancer screening (mammography rates and cervical cancer screening rates) had already been adopted by the HCQI Project at an earlier stage and comparative numbers had been released [11].

Only those indicators that satisfied the US Institute of Medicine’s criteria of importance, scientific soundness, and feasibility were considered. The indicators were judged on the basis of (i) their impact on health status in the specific clinical area, (ii) their policy relevance and susceptibility to being influenced by the health system, (iii) the scientific soundness of the indicators, and (iv) the feasibility as defined as probable data availability and reporting burden for the provider organizations and national bodies.

This preliminary scoping exercise identified 270 indicators, which met the initial inclusion criteria, out of over 1000 potential candidates. Removal of duplicate indicators and those that would require data drawn directly from individual patient records reduced this to 109 potential indicators.

Selection of final set of indicators

The Primary Care Panel then used a modification of the Delphi process developed originally by RAND [12] to reduce this long list of indicators to a manageable size. This process was carried out over a period of 6 months using a series of conference telephone calls and email discussions. Panel members individually rated each indicator on a scale of 1–9 for scientific soundness and importance. As the panels were not able to make a definite statement about data availability for an indicator in all OECD countries, feasibility was given less weight in the decision process. The participating experts were asked to express their opinion as to whether it was likely, possible, or unlikely to find comparable data on the international level for each indicator. If data availability was regarded as unlikely, an indicator was dropped, unless strong conceptual reasons existed to retain it.

Median scores and measures of disagreement for the whole panel and individual ratings were then fed back and the ratings were discussed. Panel members were given an opportunity to change their ratings after the discussions. Indicators receiving final scores of 7–9 were regarded as robust, 4–6 as equivocal, and 1–3 as weak. All indicators receiving scores of 7 or more for both importance and soundness were included in the final set. In addition, a small number of indicators which received scores of 4–6 for either or both dimensions were retained if the panellists considered the indicators essential to contribute to the overall balance and comprehensiveness of the final set.

This process resulted in a reduced set of 27 indicators, six of which were categorized as relating to health promotion, 13 to preventive care, and 8 to primary clinical care (Table 1).

View this table:
Table 1

List of 27 selected indicators

Health promotionObesity prevalence
Physical activity
Smoking rate
Diabetes prevalence
Gonorrhoea/chlamydia rates
Abortion rates
Preventive careBlood typing and antibody screening for prenatal patients
HIV screen for prenatal patients
Bacteriuria screen for prenatal patients
Immunizable conditions
Low birth weight rate
Adolescent immunization
Anaemia screening for pregnant women
Cervical gonorrhoea screening for pregnant women
Hepatitis B screen for pregnant women
Hepatitis B documentation in record at time of delivery
Hepatitis B immunization for high-risk groups
Influenza vaccination for high-risk groups
Pneumococcal vaccination for high-risk groups
Diagnosis and treatment: primary careCongestive heart failure readmission rate
First visit in first trimester
Smoking cessation counselling for asthmatics
Blood pressure measurement
Re-measurement of blood pressure for those with high blood pressure
Initial laboratory investigations for hypertension
Hospitalization for ambulatory care sensitive conditions

Description of indicator set

A description of the full set of indicators is available on the OECD website at www.oecd.org/document/31/0,2340,en_2649_201185_2484127_1_1_1_1,00.html.

Health promotion

The health promotion indicator subset covers important areas of health-related behaviour that are typically targeted by health education and patient awareness and outreach campaigns. These include smoking, obesity, physical exercise, sexually transmitted diseases, and abortion. In addition, diabetes prevalence was included to reflect the policy importance of the emerging epidemic of diabetes mellitus, and the fact that this indicator was not included within the diabetes indicator set.

Preventive care

The preventive care subset contains a high proportion of indicators relating to pre- and perinatal care, including blood typing, screening for antibodies, anaemia, cervical gonorrhoea, hepatitis B, HIV, and bacteriuria, and low birth weight rates. The remaining indicators all relate to vaccination and include hospital admissions with immunizable conditions, adolescent immunization rates, and immunization for hepatitis B, influenza, and pneumococcus in high-risk groups. Preventive indicators in disease areas such as diabetes and heart disease were considered by the respective expert panels in each condition-specific area.

Clinical primary care

The clinical primary care subset contains indicators describing the diagnosis and management of hypertension, the management of congestive heart failure, smoking cessation rates for patients with asthma, and an aggregated indicator describing hospitalization rates for conditions usually managed in primary care.


The expert panel considers the suggested list of indicators to be an important but preliminary step towards defining a usable set of measures to evaluate and compare the quality of health promotion, preventive care, and primary care services. The resource constraints under which the panel operated limited the work to a review of existing indicators in the public domains. It did not allow the development of a framework that would capture the complex and broad construct of primary care, prevention, and health promotion comprehensively nor for the development of new indicators to populate the domains of the framework. Inevitably, this then constrained the comprehensiveness of the final indicator set. Nevertheless, a preliminary set of 27 indicators has been selected with sufficient agreement on policy relevance and scientific soundness. In this final section of the article, we critique the development process and the resulting set of indicators created by the expert group. We then consider further work required to take this endeavour forward and to create a set of quality measurements that is widely credible and useful.

The development process

The scale of the task facing this panel was inevitably greater than that of the other panels, which were working in more tightly defined disease areas. For this reason, as well as the resource constraints, the scope of this exercise had to be limited in a number of ways. First, the primary search for potential indicators was not exhaustive, but focused on the most high-profile and most extensively used indicator sets. Second, we chose to use a simple categorization of ‘primary care’—health promotion, prevention, and primary clinical care. This helped define the task, but alternative categorizations that have been used by other indicator development teams could have been considered [13]. Third, the review of disease-specific indicators, such as for cardiac care and diabetes, was left to other expert panels.

Despite these practical limitations, the number of indicators that needed to be evaluated by the panel members was still large and required a pragmatic interpretation of the standard expert panel method. In particular, we used a single-stage rating process to make a judgement about the quality of the indicators, rather than the formal two-stage Delphi process which is conventionally used. However, it is not clear that this compromise will have made a significant difference to the final set of indicators [14].

The comprehensiveness of the indicator set

Whereas a set of 27 indicators can clearly not comprehensively represent a field as large as that of primary care, the size of the set represents the desire of the development team to produce a product which can serve as a starting point for future discussions on internationally comparable measures for primary care, prevention, and health promotion. The following critique should be seen in this light.

Overall, a reasonable balance has been achieved between the three categories of health promotion, prevention, and clinical primary care. The expert panel wanted to emphasize the importance of health promotion and prevention, and the number of indicators in these areas relative to that of clinical care demonstrates the high profile that these areas should have in health policy terms.

The health promotion set clearly focuses on the key public health issues of smoking, obesity, and exercise. The only area which is notably absent is that of substance abuse, and this should be addressed in future work. The preventive set is strongly dominated by indicators representing prenatal care. Whilst this is an important area, the imbalance may, in part, reflect the problems with prenatal care provision amongst a sizeable part of the population in the USA, where the indicators were developed and thus may not be of equal policy priority in other participating countries. The most obvious omission from the preventive set is the area of cancer screening. This area is being addressed elsewhere in the OECD HCQI Project [11].

The clinical primary care indicator subset is the least comprehensive part of the set. In part, this reflects the inclusion of key clinical indicators by other expert panels [9,10]. This reinforces a fundamental principle that the OECD HCQI Project must be seen as an integrated whole, not simply the individual subsets. It also reflects the requirement to specially collect data from medical records to satisfy most indicators of clinical process of care.

Considerable discussion took place around the indicator of hospitalization rates for primary care sensitive conditions, given some evidence that only a small proportion of these admissions can be directly attributed to deficiencies in primary care delivery [15]. Nevertheless, the panel agreed that it was supported by prior evidence review [16] and relevant to public policy because it pointed to avoidable hospitalizations as an intuitively important and tangible consequence of inadequate primary care.

Future work

It is clear to this panel that a highly constrained process cannot yield the final answer to measuring the quality of a construct that is as broad and complex as primary care, prevention, and health promotion. Further work remains necessary to ensure that the set becomes more comprehensive and therefore a more valid assessment of health system performance. This will require not only the addition of new disease and condition areas but also the development of indicators of important non-clinical areas of primary care practice, such as equity, coordination, and continuity of care. It will also require a more in-depth review of the evidence behind the recommended measures, as some had been developed many years ago.

A critical step will be to derive a conceptual framework that helps to organize the future development process and to define the boundaries of the construct. This will be a challenging task as countries draw the line between the roles and responsibilities of the health care system and public health as well as of the primary care system and other parts of health care quite differently [1]. But such a discussion ought to take place before final recommendations about measure selection can be made. Alignment and overlap with other international reporting programmes such as those of the World Health Organization and the European Commission need to be examined.

In parallel to the work on a solid foundation for this project, operational feasibility of the proposed indicators needs to be assessed by undertaking a survey of data availability in OECD countries. Further work is also required to examine whether the indicators, most of which have been developed to report on provider performance within countries, are also suitable for international benchmarking of health system performance.

There is a popular saying in the quality improvement world, namely that perfect must not be the enemy of good [17]. The indicator set, in current form, is far from perfect, but the philosophy underlying the project, the consensual development process among the participating experts, and the preliminary set of measures have all contributed to the policy debate about the worth and expectations of international health system comparisons. A big step has been taken and there is a long way to travel.


This article describes the key recommendations made by the primary care panel of the OECD HCQI Project . It reflects the opinions of the authors and not an official position of the OECD, its member countries, or institutions participating in the project. The full report of the panel proceedings can be found at www.oecd.org/dataoecd/27/52/33865865.pdf. The authors thank Peter Hussey for his contribution to the indicator selection process, Elizabeth Cote and Leighna Kim for providing research support, and Victoria Braithwaite and Orla Kilcullen for their help in preparing this manuscript. The comments of the members of OECD Expert Group on the Health Care Indicators Project and the Ad Hoc Group on the OECD Health Project on an earlier version of this manuscript are greatly appreciated. The authors would also like to thank John Martin, Martine Durand, Peter Scherer and Jeremy Hurst for review and comments.


View Abstract