International Journal for Quality in Health Care 15:131-137 (2003)
© 2003 International Society for Quality in Health Care
Paper |
Impact of an acute myocardial infarction report card in Ontario, Canada
1Institute for Clinical Evaluative Sciences, Toronto, Ontario
2Division of General Internal Medicine and the Clinical Epidemiology and Health Care Research Program, Sunnybrook and Womens College Health Sciences Centre
3Departments of Medicine, Public Health Sciences, Health Policy, Management, and Evaluation, University of Toronto, Toronto, Ontario, Canada
Objectives. Acute myocardial infarction (AMI) ‘report cards’ are being developed using administrative databases in many jurisdictions, but little is known about their acceptance by and their usefulness to the medical community. The purpose of this study was to determine the impact of the publication of Cardiovascular Health and Services in Ontario: An ICES Atlas Naylor CD, Slaughter P. (eds), 1999, Toronto: ICES), the first report featuring hospital-specific AMI performance measures to be published in Canada.
Design. We conducted a mail survey of physicians at Ontario hospitals to determine their views on the usefulness of various atlas performance measures for assessing and improving quality of care, the types of quality initiatives launched at their hospital in response to the atlas, and their views on the concept and limitations of reporting hospital-specific AMI mortality data.
Results. Respondents to the survey indicated that information on process of care measures such as post-infarction beta-blocker and angiotensin-converting enzyme (ACE) inhibitor use, and cardiac procedure waiting times were the most useful, and outcomes data (e.g. 30-day and 1-year risk-adjusted AMI mortality rates) the least useful of the multiple performance measures published in the atlas (P = 0.0385). Fifty-four percent of respondents reported launching one or more quality of care initiatives at their hospital in response to the atlas. The majority of respondents (65%) indicated that they support the public release of hospital-specific AMI mortality data, although many had concerns about potential miscoding in administrative databases and the adequacy of risk-adjustment methods.
Conclusion. The publication of the first AMI report card in Canada stimulated quality of care initiatives at many Ontario hospitals. Inclusion of performance measures other than mortality in health care report cards may lead to greater acceptance and use by the medical community.
Keywords: myocardial infarction, Ontario, performance measures, quality improvement, quality of care, report cards
Health care ‘report cards’ are being published with increasing frequency, but there remains considerable uncertainty about the ability of these reports to stimulate meaningful improvements in the quality of health care delivery. Most report card initiatives have focussed solely on outcomes measures (e.g. mortality rates after myocardial infarction or bypass surgery), but the experience to date suggests that most of these reports have had limited or no impact on quality of care [1–4]. One study from California showed that the publication of hospital-specific acute myocardial infarction (AMI) mortality rates in California stimulated very few new quality of care initiatives [5]. A survey of cardiologists and cardiac surgeons in Pennsylvania revealed that the publication of hospital- and surgeon-specific cardiac surgery outcomes data had limited credibility among cardiovascular specialists, and had little impact on referral decisions [6]. Although several report cards have been evaluated in the United States, little is known about the impact of health care report cards in other countries.
In February 1999, the Institute for Clinical Evaluative Sciences (ICES) in Toronto, supported by the Heart and Stroke Foundation of Ontario, published Cardiovascular Health and Services in Ontario: an ICES Atlas (the ICES Cardiac Atlas), the first cardiac report card to be released in Canada. This comprehensive report contained detailed information on multiple aspects of cardiac care in Canada’s most populous province, Ontario, including data on multiple AMI performance indicators at a hospital-specific level [7,8]. Copies of this report were sent to all Ontario hospitals and were made available to interested members of the medical community (e.g. Ontario cardiologists) and the general public. Excerpts from the report were widely covered in the Ontario media (radio, television, and print media) in the weeks following the atlas’ release.
In this study, we conducted a survey of physicians working in Ontario hospitals to determine the impact of this report card on their hospital and their views on the report. We were interested in the respondents’ views on the usefulness of various indicators for improving quality, the types of quality initiatives launched in response to the report, and their views on the impact of public reporting of hospital-specific AMI performance data.
Materials and methods
ICES Cardiac Atlas
The methods and content of the ICES Cardiac Atlas are described in detail elsewhere [7,8]. This study focussed specifically on the AMI data included in the atlas. In brief, the atlas contained information on all 52 616 patients who were hospitalized with an AMI in Ontario between April 1, 1994 and March 31, 1997. Information on 12 AMI performance indicators was reported at the hospital-specific level as shown in Table 1. The median rate for Ontario hospitals for each of the indicators is shown along with the range (lowest, highest) of performance across hospitals. AMI patients were identified from the Canadian Institute for Health Information (CIHI) hospital discharge database [via a most responsible diagnosis with International Classification of Diseases (ICD)-9 code 410]; a series of exclusion criteria were then applied to maximize the likelihood that each patient had an AMI [9]. All hospitals in Ontario were given a list of the AMI patients that were going to be included in the report so that they could internally validate the accuracy of the diagnosis before the final analyses were conducted. The final atlas cohort excluded 283 patients who the hospitals identified as being miscoded as having an AMI. Results for hospitals treating <30 patients over the study period were not published because of small sample sizes.
|
Information in the CIHI hospital discharge database was linked to data on cardiac procedure use in the Ontario Health Insurance Plan physician claims database and to data on secondary preventive medication use (in elderly patients only) in the Ontario Drug Benefit database. In Ontario, universal drug coverage is provided to all residents aged 
65 years. The vital status (e.g. 30-day and 1-year mortality) of all patients was determined through linkage to the Ontario Registered Persons database. Record linkages across these databases were conducted using a unique Ontario health card number, which was encrypted to protect patient confidentiality. Variations in AMI patient case-mix across Ontario hospitals were adjusted for using the Ontario Acute Myocardial Infarction Prediction Rules [10]. These are logistic regression statistical models which were developed using age, sex and nine comorbidities obtained from the 15 secondary diagnosis fields in the CIHI database to predict 30-day and 1-year AMI mortality. The areas under the receiver operating characteristic (ROC) curve of these predictive models were 0.78 and 0.79, respectively [11].
Atlas survey
In January 2000, 
1 year after the initial publication of the ICES Cardiac Atlas, a follow-up survey was sent to all hospitals in Ontario to evaluate its impact. The survey contained 24 questions that covered multiple aspects of the atlas, including the respondents’ views on the utility of various performance measures included in the atlas, their assessment of the potential limitations of the atlas, their views on the media coverage of the atlas, and the specific changes made at their hospitals in response to the release of this information. The survey was mailed to the Chief Executive Officers (CEO) at all Ontario hospitals. The CEO was asked to pass the survey on for completion to the physician (or designate) most responsible for cardiac care at that hospital, although in some cases the CEO elected to fill it out directly. To maximize the response rate at each hospital, a copy of the survey was also sent directly to members of the Ontario Association of Cardiologists. Because we were primarily interested in changes at the hospital level undertaken by the medical staff, the overall response rate was calculated at the hospital level using only responses from physician responders. In the event that more than one physician responded from a hospital, only the response from the most senior physician in that organization was included in the analysis. Because of hospital mergers in Ontario between the time frame of the data used in the atlas and the time our survey was conducted, the total number of eligible hospitals for our survey was 121.
A pilot test of the survey was conducted among 10 cardiologists; the survey was then mailed out on January 12, 2000. A follow-up reminder card was mailed on February 2, 2000 and a second copy of the survey was mailed out to non-responders on February 23, 2000. Respondents were guaranteed that their responses would be treated and analyzed confidentially, and only minimal information was gathered on respondent characteristics. Overall, surveys were returned from physicians at 62 (51 complete) of the 121 eligible hospitals, representing an overall hospital response rate of 52% (41% for completed surveys). Responses were also received from non-physicians at 12 of the other hospitals, but these were considered non-respondent hospitals in the analysis of the results because we could not be certain whether their views and actions represented those of the medical staff at that hospital. The results presented in this manuscript are only from physician surveys that were largely complete. We did not impute any responses where the survey questions were left unanswered.
Statistical analysis
Analyses of the response to the questions were conducted using all of the completed responses received. The denominator for some questions varied depending on the number of completed responses. The SAS statistical package (Version 8) was used for all analyses.
To compare the perceived utility of the outcome measures (mortality, readmission) with process measures (procedure rates, waiting time, drug utilization) in the atlas, the response to each AMI-related item was assigned a numeric value based on its usefulness rating as follows: not useful = 0, somewhat = 1, moderately = 2, very = 3, and extremely = 4. A summary score was derived by summing the three values assigned to the outcome measures and dividing by three. If any of the items were missing, they were excluded from the numerator and denominator in producing the score. Similarly, a process score was derived as the arithmetic average of the non-missing values among the six process-related items. The null hypothesis of no difference in utility between outcome and process scores was tested using a paired t-test.
Results
Respondent characteristics
Table 2 shows the characteristics of the respondents and non-respondents to the survey. The vast majority of the respondents were either staff or senior (e.g. chief of cardiology, chief of medicine, CCU/ICU Directors) at their hospitals. A majority of respondents worked at large community or teaching hospitals and at hospitals without invasive cardiac procedure capability.
|
The median volume of AMI cases per year was much higher at hospitals of the respondents as shown in Table 2. However, the overall performance on key atlas performance measures (e.g. 30-day and 1-year mortality rates, beta-blocker rates, etc.) was very similar at hospitals of respondents and non-respondents, suggesting that our survey represented the full spectrum of hospital performance, with both high-performing and low-performing hospitals responding to the survey.
Value of performance measures for assessing and improving quality
Respondents’ views on the usefulness of various performance measures for assessing and improving the quality of AMI care in their hospitals are shown in Figure 1. Nearly all respondents indicated that they felt most atlas performance measures were at least somewhat useful in improving quality of care at their hospital. Respondents rated the three outcomes measures (e.g. 30-day and 1-year AMI mortality data, 1-year readmission rates) as less useful than the six process measures as opportunities for improving quality of care (P = 0.0385). Information on post-discharge use of beta-blockers, angiotensin-converting enzyme (ACE) inhibitors and cardiac procedure waiting times were rated as the most useful performance measures published in the atlas.
Quality improvement activities
The types of quality improvement activities launched by hospitals in response to the release of the atlas are shown in Table 3. Overall, 54% of the respondents indicated that one or more changes were made at their hospital. The most commonly reported changes were overviews of thrombolytic use and timeliness, reviews of the medical records of AMI patients at their hospital, and conducting continuing medical education.
|
Atlas limitations
Limitations of the atlas that were rated by respondents as very or extremely important are shown in Table 4. The most important limitations cited by the respondents were ‘CIHI hospital discharge data may be miscoded’, ‘inadequacy of the risk-adjustment methods’, ‘transferred patients assigned to admitting hospital’ and ‘lack of information on in-hospital drug use’, although less than half of the respondents indicated a major concern about the latter three issues.
|
|
Impact of the atlas
A majority of the respondents (62%) indicated that some of their hospitals’ atlas data were covered in their local media. By far the most commonly reported performance measure in the media was AMI mortality rates (81%). This was followed in order of frequency by AMI procedure rates, secondary prevention rates, and readmission rates, and AMI procedure waiting times (Table 5).
Many respondents (79%) felt that the publication of the atlas did not affect the reputation of their hospital and 15% thought that the atlas improved their hospital’s reputation. Only 6% thought that the coverage harmed their hospital’s reputation. Relatively few physicians reported that their patients discussed any findings from the atlas with them in the year after its publication, and very few physicians felt that the atlas influenced the volume of cardiac patients going to their hospital.
A majority of respondents (65%) indicated that they support the public release of hospital-specific AMI mortality data. For those individuals who did not support the public release of this data, the primary reasons cited were that the public does not understand the data, that the data are misleading or inaccurate, and that there could be potential harm to a hospital’s reputation.
Discussion
There has been increasing interest in Canada in the past few years on the part of policy makers, the media, hospital administrators, and the medical community in the development and publication of hospital report cards as a new method for improving health care quality and ensuring public accountability. The ICES Cardiac Atlas was the first report card to be released in any Canadian jurisdiction in which hospital-specific AMI mortality rates and other performance measures were compared and made publicly available. Our survey demonstrates that the publication of this information had an impact at many Ontario hospitals. A majority of respondents indicated that one or more quality improvement activities were launched at their hospitals in direct response to this information. These findings are particularly noteworthy, given the previous experience from several American studies where other cardiac report cards have generally been viewed quite negatively by the medical profession and have led to relatively few quality improvement initiatives [1,2,5,6].
A major difference between the ICES Cardiac Atlas and AMI report cards published in other jurisdictions [12–14] relates to the inclusion of multiple AMI performance measures other than mortality. The atlas contained information on multiple process of care measures including the post-discharge use of beta-blockers and ACE inhibitors in elderly AMI survivors, and the post-MI rates of invasive cardiac procedure use and waiting times. This information was made possible because of the availability of comprehensive, linkable population-based administrative databases in Ontario. The respondents to our survey indicated that these process of care performance measures were more useful to them in improving quality than the outcome measures such as risk-adjusted mortality rates. Although outcome measures are important, process of care measures are easier for physicians and hospitals to modify, even if they harbor reservations about the validity of risk-adjusted outcomes analyses. In contrast, the Ontario media’s coverage reported primarily on the mortality results and featured very little coverage of the other performance measures.
Although only a few hospitals in Ontario were classified as low or high AMI mortality outliers in the atlas, more than half of the respondents indicated that their hospitals made one or more changes in response to the release of the atlas. This may reflect the finding that there were wide variations in all the performance measures reported in the atlas, and no hospital in Ontario consistently performed ‘the best’ on all performance measures. Inclusion of multiple performance measures signaled opportunities for improvement at most Ontario hospitals. Many respondents indicated a particular interest in the secondary prevention data in the atlas, and launched quality improvement activities designed to improve their rates of secondary prevention use including the introduction or revision of existing critical care pathways/standing orders, and conducting continuing medical education events. Many respondents also indicated that their hospitals conducted overviews of thrombolytic use and timeliness, even though information on this specific quality measure was not available in Ontario administrative databases.
Although a majority of the respondents appeared to have a favorable view of the AMI report card, a significant number did express important concerns about some limitations of the atlas. The most commonly cited limitations of the atlas were the potential for the hospital discharge data to be miscoded and that risk-adjustment algorithms may incompletely adjust for case-mix differences. These concerns were expressed by many respondents even though all hospitals in Ontario were given a chance to validate the accuracy of AMI coding of their patients prior to the release of the information, and any patients who the hospitals said were miscoded as having an AMI were excluded from the final analysis. The risk-adjustment models developed for the atlas were based on administrative databases with relatively good statistical performance as measured by the area under the ROC curve [10,11]. However, clinicians may be reluctant to accept any model that does not include important prognostic factors such as the location of an infarct, admission heart rate, and blood pressure [15], which are not captured in most administrative databases.
Although the atlas was covered widely by the media throughout Ontario, only 12% of respondents indicated that one or more patients discussed any information from the atlas with them in the year following its release. Respondents also reported that there was very little change in the proportion of cardiac patients going to their hospitals. These findings are consistent with the experience in other jurisdictions [16] and suggest that the Ontario report card had less impact on patients than it did on hospitals. Overall, most respondents felt that the release of the report did not have a positive or negative effect on the reputation of their hospital.
This study has certain limitations. First, as with any survey, our responses are based on self-report and it is possible (although unlikely) that the respondents reported making quality of care changes when they actually may not have done so. Secondly, there is potential for respondent bias. Our analysis of a limited number of respondent and non-respondent characteristics showed that respondents were more likely to come from large community and teaching hospitals, which care for the vast majority of AMI patients in Ontario. We did not find any evidence of a greater response rate from either high- or low-performing hospitals in the atlas, although we recognize our moderate survey response rate could lead to concerns about unmeasured differences in the characteristics of respondents and non-respondents.
Conclusions
The publication of the first AMI report card in Canada appears to have had a significant impact in Ontario. A majority of respondents to our survey reported that their hospitals launched one or more quality improvement initiatives in response to the release of this information, and support the public release of hospital-specific AMI mortality data. Physicians in Ontario rated information on process of care measures such as beta-blocker use as being more useful, and rated information on outcomes such as mortality as being less useful for improving health care quality. These results have important implications for other health care report card initiatives, and suggest that to improve the acceptance and use of report cards in the medical community, future report card initiatives should consider incorporating more information on process of care measures, rather than focussing exclusively on patient outcomes.
The authors would like to thank all of the physicians in Ontario who responded to this survey, and all of their colleagues at ICES who contributed to the development and publication of the ICES Cardiac Atlas. They would also like to thank Francine Duquette for administrative help with the survey and the manuscripts, Alice Newman for assistance with survey analysis, and J. Ivan Williams, Curry Grant, and Pamela Slaughter for comments on earlier drafts of this manuscript. This study was funded by an operating grant from the Canadian Institutes for Health Research. Dr Tu is supported by a Canada Research Chair in Health Services Research. The results and conclusions are strictly those of the authors, and should not be attributed to any of the sponsoring agencies.
Address reprint requests to J. V. Tu, Institute for Clinical Evaluative Sciences, G-106, 2075 Bayview Avenue, Toronto, Ontario, Canada M4N 3M5. E-mail: tu{at}ices.on.ca 
Accepted for publication November 22, 2002.
References
- Marshall MN, Shekelle PG, Leatherman S, Brook RH. The public release of performance data: what do we expect to gain? A review of the evidence. JAMA 2000; 283: 1866–1874.
[Abstract/Free Full Text] - Green J, Wintfeld N. Report cards on cardiac surgeons: assessing New York State’s approach. N Engl J Med 1995; 332: 1229–1232.
[Free Full Text] - Hannan EL, Stone CC, Biddle TL, DeBuono BA. Public release of cardiac surgery outcomes data in New York: what do New York state cardiologists think of it? Am Heart J 1997; 134: 55–61.[CrossRef][Web of Science][Medline]
- Leyland AH, Boddy FA. League tables and acute myocardial infarction. Lancet 1998; 351: 555–558.[CrossRef][Web of Science][Medline]
- Rainwater JA, Romano PS, Antonius DM. The California Hospital Outcomes Project: how useful is California’s report card for quality improvement? Jt Comm J Qual Improv 1998; 24: 31–39.[Medline]
- Schneider EC, Epstein AM. Influence of cardiac-surgery performance reports on referral practices and access to care. A survey of cardiovascular specialists. N Engl J Med 1996; 335: 251–256.
[Abstract/Free Full Text] - Tu JV, Austin P, Naylor CD, Iron K, Zhang H. Acute myocardial infarction outcomes in Ontario. In: Naylor CD, Slaughter P, eds. Cardiovascular Health and Services in Ontario: An ICES Atlas, first edition. Toronto: ICES, 1999, pp. 83–110.
- Tu JV, Austin P, Rochon PA, Zhang H. Secondary prevention after acute myocardial infarction, congestive heart failure and coronary artery bypass graft surgery in Ontario. In: Naylor CD, Slaughter P, eds. Cardiovascular Health and Services in Ontario: An ICES Atlas, first edition. Toronto: ICES, 1999, pp. 199–238.
- Tu JV, Naylor CD, Austin P. Temporal changes in the outcomes of acute myocardial infarction in Ontario, Canada: 1992 to 1996. Can Med Assoc J 1999; 161: 1257–1261.
[Abstract/Free Full Text] - Tu JV, Austin PC, Walld R, Roos LL, Agras J, McDonald KM. Development and validation of the Ontario acute myocardial infarction mortality prediction rules. J Am Coll Cardiol 2001; 37: 992–997.
[Abstract/Free Full Text] - Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143: 29–36.
[Abstract/Free Full Text] - Dillner L. Scottish death rates published with health warning. BMJ 1994; 309: 1599–1600.
[Free Full Text] - Andersson J, Carling K, Mattson S. Random ranking of hospitals is unsound. Chance 1998; 11: 34–39.
- Jollis JG, Romano PS. Pennsylvania’s Focus on Heart Attack–grading the scorecard. N Engl J Med 1998; 338: 983–987.
[Free Full Text] - Lee KL, Woodlief LH, Topol EJ et al. Predictors of 30-day mortality in the era of reperfusion for acute myocardial infarction: results from an international trial of 41,021 patients. Circulation 1995; 91: 1659–1668.
[Abstract/Free Full Text] - Hannan EL, Kumar D, Racz M, Siu AL, Chassin MR. New York state’s cardiac surgery reporting system: four years later. Ann Thorac Surg 1994; 58: 1852–1857.[Abstract]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  C. H. Fung, Y.-W. Lim, S. Mattke, C. Damberg, and P. G. Shekelle Systematic Review: The Evidence That Publishing Patient Care Performance Data Improves Quality of Care Ann Intern Med, January 15, 2008; 148(2): 111 - 123. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. J. Epstein Do Cardiac Surgery Report Cards Reduce Mortality? Assessing the Evidence Med Care Res Rev, August 1, 2006; 63(4): 403 - 426. [Abstract] [PDF]
|
|
|
|
 |
|
 A. H. Rosenstein Hospital Report Cards: Intent, Impact, and Illusion American Journal of Medical Quality, September 1, 2004; 19(5): 183 - 192. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||