International Journal for Quality in Health Care Advance Access originally published online on January 21, 2005
International Journal for Quality in Health Care 2005 17(2):107-114; doi:10.1093/intqhc/mzi007
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Development of a set of strategy-based system-level cancer care performance indicators in Ontario, Canada
1 Cancer Quality Council of Ontario, Toronto, Ontario, Canada, 2 Cancer Care Ontario, Toronto, Ontario, Canada, 3 Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
Objectives. To develop a set of scientifically sound and managerially useful system-level cancer care performance indicators for public reporting in Ontario, Canada.
Implementation. Using a modified Delphi panel method, comprising a systematic literature review and multiple rounds of structured feedback from 34 experts, the Cancer Quality Council of Ontario developed a set of quality indicators spanning cancer prevention through to end-of-life care. To be useful to decision-makers and providers, indicator selection criteria included a clear focus on the cancer system, relevance to a diversity of cancer providers, a strong link to the mission and strategic objectives of the cancer system, clear directionality of indicator results, presence of targets and/or benchmarks, feasibility of populating the indicator, and credibility of the measure as an indicator of quality. To ensure that the selected indicators would measure progress over time against specific and widely accepted goals, we created a strategy map based on the five strategic objectives of the Ontario cancer system: (i) to improve the measurement and reporting of cancer quality, (ii) to increase the use of evidence and innovation in decision-making, (iii) to improve access to cancer services and reduce waiting times, (iv) to increase efficiency across the system, (v) to reduce the burden of cancer. An analysis of the mean indicator ratings by experts, and the strategy mapping exercise resulted in the identification of 36 indicators deemed suitable for routine performance measurement of the Ontario cancer system.
Lessons learned. The resulting instrument incorporates a credible evidence basis for performance measurement aligned to the five strategic goals for the Ontario cancer system. It represents the integrating of a management culture, focused on the implementation of a new strategic direction for the cancer system, with the underlying evidence-based culture of clinicians.
Keywords: neoplasms, performance measurement, quality indicators of health care, quality of health care
Address reprint requests to Adalsteinn D. Brown, Assistant Professor, Associate SGS Member, Health Policy, Management and Evaluation, Faculty of Medicine, University of Toronto, McMurrich Building, 2nd Floor, 12 Queen’s Park Crescent, West Toronto, Ontario, M5S 1A8416-978-1484. E-mail: adalsteinn.brown{at}utoronto.ca
Accepted for publication October 8, 2004.
 |
Background |
|---|
 Top Background Reorganization and re...Previous methods for measuring...ImplementationLessons learnedNext steps for implementing...Next steps for researchReferences |
|---|
Pressures facing the cancer system in Ontario
Coordinating cancer care in Ontario presents a number of challenges, including the large size of the province and the need to provide equally high-quality care no matter where patients reside, despite regional variation in the types of cancer most commonly diagnosed and practice patterns for different cancers. Visible symptoms of these challenges include growing waiting times for cancer treatment over the past decade, and uneven access to both treatment and supportive services [1,2]. Capacity constraints are exacerbated by an ageing and growing population [3], and the need to continually integrate emerging advances in diagnostics and treatment into the delivery system. Moreover, until recently, Ontario has had a long history of no central oversight of the majority of cancer services delivered. Due to this legacy, the cancer ‘system’ in Ontario has been limited to coordination of the radiation therapy and about half of the systemic therapy delivered.
|
Reorganization and re-integration of the cancer system |
|---|
|
TopBackgroundReorganization and re...Previous methods for measuring...ImplementationLessons learnedNext steps for implementing...Next steps for researchReferences |
|---|
To respond to these pressures, the Ontario cancer system was recently restructured [4]. Cancer Care Ontario, the public agency previously responsible for providing radiation and systemic therapy to patients receiving treatment in regional cancer centres, has shifted away from direct service delivery to a new purchasing role with responsibility for ensuring quality for all cancer patients in Ontario. Cancer centres and affiliated hospitals—traditionally separate entities—were combined into Integrated Cancer Programs (ICPs) creating a single point of accountability for service delivery.
Role of performance reporting in guiding the system
In 2002, the Ontario Minister of Health and Long-term Care established the Cancer Quality Council of Ontario, an arm’s-length body of experts in the fields of cancer medicine, research, and policy, to monitor and report publicly on cancer system performance and guide improvements. Adopting a simple definition of quality—ensuring that all patients receive ‘medically appropriate and timely access to specialists and specialist treatment’ throughout the patient journey [5] the council set out to develop a valid, useful and comprehensive annual report of quality for the provincial cancer system. The report would measure progress at a high level against strategic objectives.
However, the culture of the cancer sector in Ontario in general has long been one in which evidence is highly valued. As one example, Cancer Care Ontario’s Program in Evidence Based Care is an internationally recognized clinical practice guidelines development initiative [6], which provides up-to-date expert recommendations based on the latest available scientific evidence. The recent change in Cancer Care Ontario’s role introduced a performance management philosophy on top of the existing evidence-based foundation. Vital to the success of this shift therefore is a sufficient evidence basis to ensure that indicators, targets, and benchmarks are acceptable to a broad range of stakeholders, many of whom are used to a clear articulation of the evidence for decision-making.
|
Previous methods for measuring cancer system performance |
|---|
|
TopBackgroundReorganization and re...Previous methods for measuring...ImplementationLessons learnedNext steps for implementing...Next steps for researchReferences |
|---|
The Ontario experience
In the past, performance measurement within Cancer Care Ontario was used largely for internal management purposes, and measures were collected and reported on an ad hoc basis. These measures spanned cancer surveillance, data quality, organized breast screening, radiotherapy, and systemic therapy. Apart from surveillance data from the Ontario Cancer Registry, these measures tended to focus only on those services provided directly by Cancer Care Ontario, accounting for roughly a quarter of all cancer spending in the province [7]. Moreover, significant measurement gaps have existed across much of the continuum of cancer prevention and treatment including risk factor surveillance, surgery, and palliative care (see Table 1).
|
The international experience
A small number of organizations around the world have developed comprehensive performance measurement systems specific to cancer care (see Table 2). Common to all of these systems is an attempt to assess quality across important and common cancer sites and across the continuum of care. For example, the National Health Service Executive in the UK developed a framework to measure quality in the prevention, early detection, treatment, and palliative care of breast, lung, and colorectal cancers [12].
|
Although there was significant value for Ontario in learning from these and other cancer performance measurement systems, it was not appropriate to adopt any one of the instruments wholesale. For example, the reliance on medical chart abstraction for many of these instruments is not suitable for routinely producing a high-level provincial picture of quality where more than 50 000 new cancer patients need care every year, in addition to a portion of the nearly 250 000 previously diagnosed patients [15]. Moreover, in no instance did the target audience, scope, or particular measurement objectives match that of the council: to monitor the delivery of medically appropriate and timely cancer services with a focus on measuring progress against the strategic objectives of the cancer system. A process of adapting indicators from other contexts for Ontario’s particular objectives, care delivery system, professional culture, and clinical practice was needed [16].
|
Implementation |
|---|
|
TopBackgroundReorganization and re...Previous methods for measuring...ImplementationLessons learnedNext steps for implementing...Next steps for researchReferences |
|---|
A four-person working group was established to set the project in motion. The group included a member of the Cancer Quality Council of Ontario, an expert in performance measurement, and two members of the council’s secretariat. The working group met 11 times over the course of 15 months.
The council emphasized the need for the report to have a system-level focus. This was to correct the legacy of ad hoc performance measurement focused in great detail on a small portion of cancer service delivery; to demonstrate to the public that the system as a whole was being monitored and was accountable; to be useful across all modalities of cancer care; and to complement and support a range of detailed program-level performance indicators already in use in Ontario.
Identifying potential measures of cancer system performance
Phase 1 of the project involved a systematic literature review, intended to generate a comprehensive list of published and unpublished cancer-specific quality indicators. The review comprised scholarly literature (1990–2003) and organizational documents on indicator initiatives in other jurisdictions; cancer clinical practice guidelines; and personal communication with health services researchers involved in indicator development.
The search strategy included MeSH terms for 12 cancer sites, and quality terms (quality indicators, quality control, quality assurance, audit, and benchmarking) in Medline, Cancerlit, HealthStar, CINAHL, and EMBASE. In addition, we performed web-based searches for gray literature using the keywords ‘cancer’ and ‘quality indicators’. Finally, we conducted a review of practice guidelines, referencing the National Guideline Clearinghouse and the Guidelines Advisory Committee of the Ontario Ministry of Health and Long Term Care and the Ontario Medical Association. We identified a total of 2437 potentially relevant articles and documents, the review of which generated a list of over 650 cancer-specific quality indicators.
Selecting indicators of cancer system performance
To reduce this list systematically, we initiated a series of modified Delphi panels [17] in which experts reviewed the measures and associated evidence, and rated each according to a defined set of criteria. To identify panellists, the working group nominated Ontario experts in each modality of cancer care and prevention. The nominated panellists in turn nominated other recognized experts. All nominated panellists were invited to participate in the review exercise. Thirty-four panellists representing expertise across seven modalities (prevention, surveillance, screening, surgery, systemic therapy, radiotherapy, and supportive and palliative care) were selected to review the indicators.
Experts were divided into two groups: those internal or external to Cancer Care Ontario. The first panel consisted of 10 Cancer Care Ontario clinical program leaders. The second panel consisted of 16 additional clinical experts. Eleven members of the council conducted the final review. In contrast to the previous two rounds, council members rated all indicators, regardless of area of expertise. In all, response to the surveys was high: 90% in the first round, 88% in the second, and 85% in the third.
In each case, panellists were asked to fill out a paper survey based on the following criteria:
- Captures quality at the level of the cancer system.
- Relevant to one or more types of cancer providers.
- Linked to the mission of Cancer Care Ontario.
- Linked to one or more of the strategic objectives of the Ontario cancer system.
- Has clear directionality.
- Has existing targets.
- Is feasible to measure and report.
- Has credible evidence basis.
The goal was to isolate indicators likely to facilitate valid, feasible, cancer performance information that would be useful to decision-makers and providers. In all three rounds of review, additional indicators were suggested and appended to the candidate list. We analysed mean scores for each indicator on a five-point scale for the above criteria. The original list was reduced to 45 indicators. Distinguishing indicators based on criterion 1—whether the indicator captured quality at the level of the provincial cancer system—largely drove the reduction.
Constructing a strategy-based scorecard
An interesting synergy occurred between the Council’s development of cancer system quality indicators and the strategic direction of Cancer Care Ontario. The process of surveying experts on what would be useful and important measures to monitor performance played a role in the refinement of CCO’s strategic objectives. Most importantly, this resulted in a strong and explicit emphasis on improving the measurement, collection, and reporting of cancer performance information. Through an iterative process that included structured feedback from Ontario cancer system managers and providers, the strategic goals of Cancer Care Ontario became:
- To improve measurement, collection and reporting of cancer system performance.
- To increase the use of evidence and innovation in decision-making.
- To increase the effective use of resources across the system.
- To improve access to cancer services and reduce waiting times.
- To reduce the burden of cancer.
With these general strategic directions in mind, we first attempted to map the core set of indicators to a typical Kaplan and Norton balanced scorecard model (financial performance, customer value, internal processes, and learning and growth) [18]. However, this approach resulted in three problems. Firstly, the selection process coupled with the particular emphases of the scorecard quadrants tended to underemphasize certain aspects of the continuum of cancer care delivery (prevention to palliation). For example, prevention was underrepresented. Secondly, the scorecard was unbalanced; the ‘financial performance’ quadrant had too few indicators. Finally, it was difficult to align the strategy implicit in the Kaplan and Norton framework with the cancer system strategic objectives.
Given general agreement, both within Cancer Care Ontario and among key stakeholders, on the validity of the five strategic goals for fulfilling the vision of improving the cancer system, we restructured the scorecard based on these. The resulting framework solved all three problems encountered with the Kaplan and Norton approach. It emphasized all services across the care continuum, it was more balanced, and it was by definition aligned with the strategic directions for the cancer system. As mentioned above, a key challenge of this initiative was to apply a credible evidence basis to the development of the performance measurement tool. The atypical scorecard approach we adopted was an attempt to mitigate the risks of implementing an untested model by aligning our measures with what is widely seen in the cancer system as a valid strategic direction.
To complete the scorecard, we first mapped the five strategic goals in relation to each other, based on a hypothesis of how each goal influences or is influenced by others (see Figure 1). As one example, we hypothesized that improving access to cancer services would require a combination of improved measurement, increased evidenced-based decision-making, and enhanced efficiency and capacity.
|
We then allocated each selected indicator to one of the five strategic goals. Finally, we analysed each indicator in turn to hypothesize lead and lag relationships. Taking one example, we hypothesized that waiting times for breast cancer assessment are likely to be sensitive to change in mammography rates and innovations in the delivery of diagnostic services for breast cancer. In turn, changes in waiting times for breast assessment are likely to influence patient satisfaction with access to care. The council reviewed and refined these relationships over the course of three in-person sessions. As shown in Table 3, the core set was ultimately reduced to 36 indicators.
|
The process helped the council to correct a slight imbalance under certain strategic goals between lead versus lag indicators. Specifically, the core set was somewhat weighted towards measuring performance on our strategic goals in the long term (e.g. smoking rates versus tobacco prevention activity). Beyond simply producing data on a series of measures, adapting and applying the Kaplan and Norton scorecard methodology will allow us to paint a comprehensive picture with the indicators. That is, based on our hypothesis of the relationships among the strategic goals, the scorecard is designed to describe, for example, whether overall performance is moving in the right direction.
Feasibility of the selected indicators
Only one of the selected indicators is currently infeasible, ‘radiation therapy quality assurance’, due to the lack of a centralized audit process to measure compliance with provincial guidelines. Measures are currently being taken in Ontario to address this issue. For a handful of other indicators (e.g. cancer research funding and clinical trial accrual), the corresponding information systems are in their infancy and so data are likely to be incomplete in the council’s first report. Apart from these issues, the main challenge to the feasibility of calculating the selected indicators is that many of the measures require complicated linkages between multiple population databases. Although do-able, this type of linkage is time consuming, making the routine measurement and reporting of cancer system performance more difficult.
|
Lessons learned |
|---|
|
TopBackgroundReorganization and re...Previous methods for measuring...ImplementationLessons learnedNext steps for implementing...Next steps for researchReferences |
|---|
Summary of the system and plans for implementation and public reporting
We set out to create a high-level tool to measure and monitor performance in Ontario’s cancer system. The resulting instrument incorporates a credible evidence basis for performance measurement, including a systematic literature review and modified Delphi panel method, as well as structured feedback from a broad base of cancer system managers and providers. Further, the development of a scorecard for the selected indicators based on the five strategic goals for the Ontario cancer system allowed us to integrate a management culture, focused on the implementation of new strategic direction for the cancer system, within the underlying evidence-based culture of clinicians. Bridging the gap between management and clinical practice through evidence-based performance measurement is increasingly critical to the field of health care generally [19].
The combination of measuring performance at a system level and across the spectrum of cancer prevention and treatment represents a single tool for managing the cancer system that was not previously available. The implementation of this tool should help us to fulfill three major goals of the Cancer Quality Council of Ontario:
- To provide timely, credible, public information on cancer system performance where little to none has existed previously.
- To assist cancer system managers, practice leaders, and policy-makers to make informed decisions about targeted quality improvement.
- To demonstrate to the public that the system is being monitored and is accountable.
This initiative is one of several recent efforts around the world to monitor performance in cancer care delivery systems. These initiatives reflect widespread acknowledgement that the complexity of the cancer patient journey—which encompasses multiple disease sites, and a host of providers, care settings, and specializations—warrants a performance measurement system specific to this group of diseases.
It is important to examine the limitations to the approach we have adopted. Firstly, while this initiative is based on an assessment of measures used in other jurisdictions to monitor quality in cancer care, there is the possibility that we could have missed important indicators. On the other hand, our systematic review was a 3-month exercise that included five medical databases, and concerted efforts to ensure that we were not simply relying on the published literature, but an assessment of cancer indicator development initiatives in all phases of completion through obtaining feedback on our review from experts in the field. Secondly, our use of the modified Delphi panel process to select indicators could have posed some issues of validity. Still, a separate study on surgical quality of care indicators carried out at Cancer Care Ontario at the same time showed that expert panels were more likely to select indicators with a stronger evidence base. This suggests that this type of process helps to improve the validity of the indicator set. Thirdly, the underlying strategy for the cancer system could be flawed. However, the strategy was carefully designed and vetted by all senior executives of the Ontario cancer system, with structured input from managers and providers throughout the province. We feel that we can rely on it being a sound approach to system improvement, and a reasonable basis upon which to measure progress.
|
Next steps for implementing the instrument |
|---|
|
TopBackgroundReorganization and re...Previous methods for measuring...ImplementationLessons learnedNext steps for implementing...Next steps for researchReferences |
|---|
There are a few target audiences for the results of this initiative including managers, providers, policy-makers, patients, and the general public. For managers, decision-makers, and providers, we plan to report confidential institution- or local-level performance relative to provincial performance. For our primary audience, patients, and the general public, we plan to create a graphical, easily interpretable report (paper and web-based versions) showing only provincial and regional data.
Reporting will occur in two phases. The release of the public report will occur at least 3 months after the release of the confidential data to allow system stakeholders an opportunity to comment on and/or challenge the results before they are aggregated and made public. The rationale for this approach is to validate the data, while ensuring privacy for each institution prior to publication.
|
Next steps for research |
|---|
|
TopBackgroundReorganization and re...Previous methods for measuring...ImplementationLessons learnedNext steps for implementing...Next steps for researchReferences |
|---|
A cancer-specific quality assessment tool has never before been implemented anywhere in Canada. As we roll out a program of public reporting of aggregate results and confidential reporting of local and institutional data, a number of questions will arise with respect to the utility of the results and their dissemination.
In the near term, there will be a need to evaluate the acceptability, interpretability and utility of the indicators among key stakeholders in the system. There will also be a need to test the validity of the underlying scorecard, in terms of how the strategic goals and indicators relate to each other. In the longer term, other important questions include whether or not the indicators are being used to promote quality improvement locally; and whether or not improvements are evident. Finally, given the multitude of cancer performance measurement initiatives internationally, it will be important to compare the impact of this initiative with that of others over time.
|
References |
|---|
|
TopBackgroundReorganization and re...Previous methods for measuring...ImplementationLessons learnedNext steps for implementing...Next steps for researchReferences |
|---|
- Simunovic M, Thériault ME, Paszat L et al. Waiting times among patients undergoing major breast, colorectal, lung and prostate cancer surgery in Ontario for years 1993 to 2000. Can J Surg 2004; still in press.
- Sullivan T, Evans W, Angus H, Hudson A, eds. Strengthening the Quality of Cancer Services in Ontario. Ottawa: Canadian Healthcare Association Press, 2003.
- National Cancer Institute of Canada. Canadian Cancer Statistics 2004, Toronto, Canada: NCI Canada, 2004.
- Sulllivan T, Dobrow M, Thompson L, Hudson A. Reconstructing cancer services in Ontario. Healthcare Papers. Healthcare Papers 2004; 5: 69–80.
- American Society of Clinical Oncology. Consensus Statement on Access to Quality Cancer Care (approved March 4, 1998). http://www.asco.org/ac/1,1003_12-002177–00_18-0011731–00_19-0011734–00_20-001,00.asp Accessed June 16, 2004.
- Browman GP. Development and aftercare of clinical guidelines: the balance between rigor and pragmatism. J Am Med Assoc 2001; 286: 1509–1511.
[Free Full Text] - Sullivan T, Evans W, Angus H, Hudson A, eds. Strengthening the Quality of Cancer Services in Ontario. Ottawa: Canadian Healthcare Association Press, 2003.
- Micheli A, Capocaccia R, Martinez C et al. Cancer control in Europe: a proposed set of European cancer health indicators. Eur J Public Health 2003; 13(3 Suppl): 116–118.
- Information on ASCO’s National Initiative on Cancer Care Quality (NICCQ). http://www.asco.org/ac/1,1003,_12-002177-00_18-0011725-00_19-0011728-00_20-001,00.asp Accessed June 16, 2004.
- National Cancer Institute. NIH, DHHS, Bethesda, MD Cancer Progress Report—2003 Update. National Cancer Institute, February 2004. http://progressreport.cancer.gov/ Accessed June 16, 2004.
- Asch SM, Kerr EA, Hamilton EG, Reifel JL, McGlynn EA, eds. Quality of care for oncologic conditions and HIV: a review of the literature and quality indicators. Report for RAND Health 2000. http://www.rand.org/publications/MR/MR1281/ Accessed June 16, 2004.
- National Health Service Executive. National Performance Assessment Framework (PAF): Consultation on Suggested National Cancer Performance Indicators. London: NHS Executive, 2000.
- Hillner BE, Smith TJ. The quality of cancer care: Does the literature support the rhetoric? Institute of Medicine Background Paper, U.S.A., 1999.
- Weeks J. Outcomes assessment in the NCCN: 1998 Update. National Comprehensive Cancer Network. Oncology (Huntingt) 1999; 13: 69–71.
- Cancer Care Ontario (Ontario Cancer Registry, 2004); Cancer Care Ontario. Ontario Cancer Facts: ‘What cancers are Ontarians living with?’ April 2004. http://www.cancercare.on.ca/pdf/CF-Apr2004-prevalence.pdf Accessed June 7, 2004.
- Marshall MN, Shekelle PG, McGlynn EA, Campbell S, Brook RH, Roland MO. Can health care quality indicators be transferred between countries? Qual Safety Health Care 2003; 12: 8–12.
- Helmer-Hirschberg O. Systematic Use of Expert Opinions. Santa Monica: The RAND Corporation, 1967.
- Kaplan RS, Norton DP. The Strategy-focused Organization: How Balanced Scorecard Companies Thrive in the New Business Environment. Boston, MA: Harvard Business School Press, 2001.
- McGlynn EA. An evidence-based national quality measurement and reporting system. Med Care 2003; 41 (Suppl. 1): I8–I5.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  A. Klassen, A. Miller, N. Anderson, J. Shen, V. Schiariti, and M. O'Donnell Performance measurement and improvement frameworks in health, education and social services systems: a systematic review Int. J. Qual. Health Care, February 1, 2010; 22(1): 44 - 69. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||