International Journal for Quality in Health Care Advance Access originally published online on January 23, 2006
International Journal for Quality in Health Care 2006 18(3):246-255; doi:10.1093/intqhc/mzi098
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Assessing the reliability of standardized performance indicators
Division of Research, Joint Commission on Accreditation of Healthcare Organizations, Oakbrook Terrace, IL, USA
Objectives. To investigate the reliability of self-reported standardized performance indicators introduced by the Joint Commission on Accreditation of Healthcare Organizations in July 2002 and implemented in approximately 3400 accredited US hospitals. The study sought to identify the most common data quality problems and determine causes and possible strategies for resolution.
Design. Data were independently reabstracted from a random sample of 30 hospitals. Reabstracted data were compared with data originally abstracted, and discrepancies were adjudicated with hospital staff. Structured interviews were used to probe possible reasons for abstraction discrepancies.
Results. The mean data element agreement rate for the 61 data elements evaluated was 91.9%, and the mean kappa statistic for binary data elements was 0.68. The rate of agreement for individual data elements ranged from 100 to 62.4%. The mean difference between calculated indicator rates was 4.88% (absolute value) and the range of differences was 0.0–13.3%. Symmetry of disagreement among original abstractors and reabstractors identified eight indicators whose differences in calculated rates were statistically unlikely to have occurred through random chance (P < 0.05).
Conclusion. Although improvement in the accuracy and completeness of the self-reported data is possible and desirable, the baseline level of data reliability appears to be acceptable for indicators used to assess and improve hospital performance on selected clinical topics.
Keywords: performance indicators, performance measures, reliability, hospital data quality
Address reprint requests to Scott C. Williams, Division of Research Joint Commission, One Renaissance Blvd, Oakbrook Terrace, IL 60181, USA. E-mail: swilliams{at}jcaho.org
Accepted for publication December 2, 2005.
In July 2002, the Joint Commission on Accreditation of Healthcare Organizations (evaluating over 4000 hospitals, representing over 90% of hospital beds in the United States) implemented four sets of indicators developed to permit a robust assessment of care provided to patients with acute myocardial infarction, heart failure, pneumonia, and pregnancy and related conditions (Table 1). A rigorous development and testing process led to indicator-specific data elements, algorithms, and technical specifications that were aligned in coordination with the United States Centers for Medicare and Medicaid Services (CMS) [1–4].
|
The development and testing process was intended to ensure that the indicators could fulfill three important functions: inform the accreditation process on an ongoing basis; provide objective feedback to accredited organizations about their own performance for use in quality improvement; and serve as the basis for comparing hospital performance across organizations. Subsequent to national implementation, the indicators have been used to demonstrate quality improvement in US hospitals [5], health care quality accountability to the public, purchasers, and payers, and as metrics for health care provider pay-for-performance pilot initiatives [6,7]. A complement of these indicators will also be included in an initial set of quality indicators that will be offered to Joint Commission International (JCI) accredited hospitals [8]. Once implemented, the indicators will permit international comparisons with US hospitals. Given the extensive use of these indicators, stakeholders must have confidence in consistent and accurate application in whatever setting they are used.
Data quality is vital to the use of data in any health care context [9]. The perception that data are reliable can significantly influence how organizations respond to data [10,11]. For example, hospitals identified as low performers in comparison to their peers are more likely to question the accuracy of indicator data [12]. Consequently, when the accuracy of feedback is questioned, hospitals with the most to gain through performance improvement efforts are often less likely to initiate improvement activities. If indicator feedback is to stimulate improvement efforts in hospitals, users must have confidence in the quality of the data.
Although initial pilot testing demonstrated that indicator data could be accurately collected and transmitted [2], it was necessary to assess the reliability of the indicators as they were implemented nationally across approximately 3400 hospitals. This is of particular importance in light of the fact that indicator data are generally self-reported. This study, therefore, was designed to answer two basic questions: how reliable are the data elements collected and reported by hospital abstractors, and what impact does individual data element reliability have on the reliability of indicators?
 |
Methods |
|---|
 Top Methods ResultsDiscussionLimitations and conclusionsReferences |
|---|
Participants
Thirty hospitals were identified through a stratified sample of accredited hospitals. Stratification categories included geographic location, size, setting (urban/rural), and ownership status (profit/not for profit). At the time of the study, accredited hospitals were required to collect data on two of the available indicator sets. Of the 30 sites, 19 collected acute myocardial infarction data, 17 heart failure data, 17 pneumonia data, and 7 pregnancy data (Table 2 provides a demographic breakdown). This distribution was proportional to indicator set selections for hospitals across the United States. Despite the comparatively fewer hospitals collecting data for pregnancy, no additional participants were sought for the study because the data elements for the pregnancy set, which are almost exclusively derived through administrative data, are also collected for the acute myocardial infarction, heart failure, and pneumonia sets. Consequently, sample sizes for these data elements were adequate for statistical analysis.
|
Institutional Review Board approval for the study was obtained, and recruitment letters were sent to hospital chief executive officers, followed by telephone contact from Joint Commission staff to secure participation. Hospitals were assured that participation would have no impact on accreditation status. Only four of the 30 hospitals initially recruited declined to participate. Four hospitals of comparable demographic characteristics were subsequently recruited to maintain the study sample size.
Approach
Before site visits, each participating hospital’s performance measurement system vendor (the entity through which a hospital’s indicator data are routed between the hospital and the Joint Commission) was requested to transmit de-identified patient-level data to the Joint Commission on a randomly selected set of previously abstracted medical records. These data were loaded onto data abstraction tools developed by the Joint Commission. Vendors then notified hospitals which medical records were selected for reabstraction. Hospital staff made these medical records available to Joint Commission staff.
In preparation for site visits, re-abstractors underwent training on the technical specifications of each performance indicator, including data definitions and abstraction guidelines. During the site visits, reabstraction was performed using data definitions and abstraction guidelines detailed in the Specification Manual for National Implementation of Hospital Core Measures [13].
While on site, Joint Commission staff reabstracted indicator data from the records originally abstracted by hospital staff. Results were compared, data element by data element, to identify any discrepancies. Discrepancies between original abstraction and reabstraction were discussed with hospital staff, who were given an opportunity to question the reabstractors’ findings. Based upon these discussions, reabstracted data were reviewed and adjudicated to ensure that they correctly reflected medical record documentation and were consistent with the data element definitions and abstractor guidelines. The adjudicated results were compared with original abstractions performed by hospital staff. At the conclusion of each site visit, structured on-site interviews were conducted with staff responsible for data abstraction to explore reasons for discrepancies and to identify potential solutions.
Statistical analysis
All analyses were conducted using Statistical Analysis Software (version 8.2, SAS, Cary, NC, USA) with 
set at 0.05. Reliability of individual data elements was assessed using percent agreement for continuous variable data elements and chance-corrected agreement using Cohen’s kappa for binary data elements. For the binary data elements (i.e. data elements with yes/no allowable values), McNemar’s test statistic was used to assess whether the disagreement was asymmetrical (i.e. whether there was bias present in classifying the data element). For indicator data elements common to all sets, results were combined across indicator sets to assess agreement. To determine whether variability in data element reliability influenced calculation of an indicator rate, rates for the original abstraction and for the reabstracted and adjudicated data were calculated. For the comparison of calculated indicator rates, the absolute value of the differences in the calculated rates for each indicator was calculated, and McNemar’s test statistic was partitioned to assess the symmetry of two kinds of disagreement: one for categorizing a record as being included or excluded from the indicator population and the other for positively categorizing a record as being included in the indicator numerator, given that both raters included the record in the indicator population. For continuous variable indicators, McNemar’s test statistic was calculated only for the agreement on inclusion or exclusion of the record from the indicator population.
|
Results |
|---|
|
TopMethodsResultsDiscussionLimitations and conclusionsReferences |
|---|
Reliability was first evaluated at the data element level. The 61 data elements represent individual pieces of information used to identify the indicator populations and calculate the indicator rates (e.g. discharge status, diagnosis, arrival time, vaccination status). Comparisons were expressed as a percentage (agreement rate) between the original hospital abstraction and the reabstracted and adjudicated results. The weighted average of data element agreement rates was 91.9%, and the rate of agreement for individual data elements ranged from 100 to 62.4%. Nine data elements had agreement rates below 85%, and six of those nine data elements addressed time (e.g. arrival time). Kappa statistics for binary data elements (i.e. yes/no variables) similarly had a weighted average of 0.68 and ranged from 1.00 to 0.25. Rates of agreement between originally abstracted data and reabstracted and adjudicated data, kappa statistics and symmetry statistic P-values for each of the individual data elements are summarized in Table 3.
|
To determine the impact of data element agreement, indicator rates were calculated using originally collected data and then compared with indicator rates calculated using reabstracted and adjudicated data (Table 4). The mean difference between calculated rates was 4.88% (absolute value), and the range of differences was 0.0–13.3%. Symmetry of disagreement among original abstractors and reabstractors identified eight indicators where differences in calculated rates were statistically unlikely to have occurred through random chance (P < 0.05). Disagreement on six of these indicators resulted in rates that were originally more favorable to the hospital, and two indicators had rates that were originally less favorable to the hospital.
|
To facilitate interpretation of the symmetry analysis, Table 5 illustrates disagreement between the original abstractor and the reabstractor using the aspirin prescribed at discharge indicator. Of the 227 records reabstracted for that indicator, the original abstractors excluded 107 records from the indicator population, 22 of which were based upon documented contraindications to aspirin. Eight of those 22 records were reabstracted and adjudicated as cases that should have been included as part of the indicator population (i.e. they did not have a contraindication to aspirin). Of those eight records, only three qualified for the numerator (i.e. actually received a prescription for aspirin at discharge). The same discrepancies were also examined from the perspective of the reabstractor. Of the 227 records that were reabstracted and adjudicated, 26 were excluded from the indicator population based upon documented contraindications to aspirin. Of those 26 records, 12 were originally abstracted by the hospital as part of the indicator population—and all 12 were assigned to the indicator numerator (i.e. the original abstractors failed to exclude the records from the population and then determined that all 12 had aspirin prescribed at discharge). In other words, where discrepancies were identified for this data element, original abstractors were more likely to incorrectly exclude records when the patient did not receive the medication, and they were also more likely to miss a documented contraindication when the patient had already received aspirin.
|
|
Discussion |
|---|
|
TopMethodsResultsDiscussionLimitations and conclusionsReferences |
|---|
Our results suggest that although agreement rates and kappa statistics offer valuable information about data element reliability, their interpretation is subjective, somewhat arbitrary, and can be misleading when used to evaluate indicator reliability. To illustrate, using standards established by previous studies of medical record abstraction reliability, the mean agreement rates and kappa statistics observed in this study would be considered to be acceptable, if not excellent [14–18]. CMS has recently initiated a quarterly validation process to evaluate the accuracy of patient-level data used in their Hospital Quality Initiative [19]. Hospitals achieving an overall agreement rate of 80% are considered ‘validated’ [20]. From a purely statistical perspective, kappa statistics of 0.80 or better are frequently considered ‘excellent’, 0.60–0.80 are considered ‘good’, 0.40–0.60 ‘fair’ and less than 0.40 are considered ‘poor’ [21]. Although these benchmarks can be useful tools for interpreting data accuracy, they can also be misleading. For example, the data element contraindication to aspirin at discharge, a yes/no variable, had an agreement rate of 87.2%, a kappa of 0.52, which is considered ‘fair’, and no symmetry bias was identified. Yet, as described previously, discrepancies on this data element did significantly impact the calculation of the aspirin prescribed at discharge indicator.
Despite variation of data element agreement rates and kappa statistics, indicator reliability was good, producing an average difference in the calculated rates of 4.88% points. Where the comparison of self-reported data and independently reabstracted and adjudicated data did reveal differences in indicator rate calculations that were unlikely due to chance, the differences did not universally favor the hospital. Results of this study illustrate the importance of analysing reliability from a combination of perspectives, as simply examining data element agreement rates and kappa statistics at the data element level did not predict the impact of discrepancies in the calculation of indicator rates.
Comparisons of indicator rate calculations should be interpreted with caution, as small sample sizes for some indicators may have produced differences in calculated rates that appear artificially large. For example, 227 records were reabstracted for the acute myocardial infarction indicator set. From those records, original hospital abstractors identified 165 patients that were included in the indicator population for aspirin at arrival. Only 17 of the 227 records, however, were identified as eligible for inclusion in the angiotensin converting enzyme inhibitor for patients with left ventricular systolic dysfunction indicator population. Although this may be appropriate and consistent with the inclusion and exclusion criteria—an acute myocardial infarction patient with a left ventricular ejection fraction of greater than 40% may have been included in the aspirin at arrival indicator population but would not be eligible for inclusion in the angiotensin converting enzyme inhibitor for patients with left ventricular systolic dysfunction indicator population—it also reduces the ability to interpret differences between indicator rate calculations based upon small sample sizes.
For that reason, in addition to assessing reliability from a quantitative perspective, structured interviews with hospital staff were conducted during the reabstraction and adjudication process to help to explain data element agreement discrepancies. Based upon feedback from hospital staff, incomplete or inconsistent documentation in the medical record most likely contributed to the discrepancies found for the aspirin prescribed at discharge indicator. Abstractors, unable to consistently find documentation related to contraindications, often assumed that a contraindication must have existed, given their clinical knowledge and the fact that aspirin was ultimately not prescribed.
Analysis of structured on-site interview responses also led to identification of other potential problem areas. These issues included the following:
a. Accuracy of pre-populated administrative data
In most hospitals, a number of data elements are derived from administrative/billing data. These data elements are often automatically populated into indicator data collection software. In some cases, interviews indicated that because data abstractors did not abstract these data elements, they did not feel responsible for their accuracy and/or did not realize that data in these fields impact indicator rates. This was particularly true for admission source, discharge status, and diagnosis and procedure codes. Hospitals which ensured that data abstractors understood the definitions and potential impact of these general data elements, and took responsibility for ensuring the accuracy of all pre-populated data, anecdotally appeared to have better reliability than those that did not.
b. Abstraction inconsistent with abstraction guidelines
A number of data elements were abstracted in a manner inconsistent with the definitions and abstractor guidelines provided in the specifications manual. In general, this was due to abstractor ‘interpretation’ of the specifications.
Data elements impacted by this issue include the following:
- Discharge instructions: This indicator comprises six individual data elements, all of which must have been provided to the patient or caregiver in writing for this indicator to be met. In some cases, interviews revealed that abstractors disregarded the requirement for all six elements or correctly identified the presence of several data elements, but incorrectly ‘assumed’ that a missing instruction was also provided. Others overlooked the absence of documentation that written instructions had been given was not present.
- Contraindication to medication data elements: Contraindications that were not part of the data element definition or abstraction guidelines were frequently abstracted as contraindications based upon the abstractor’s clinical knowledge. The specifications clearly require that, in the absence of a contraindication included as part of the data element definition, a contraindication may also be considered if a physician, physician’s assistant, or nurse practitioner has specifically documented that the medication was not given for some other reason. At times, abstraction staff ‘assumed’ that a contraindication existed and overlooked the lack of required documentation.
- Time-based data elements: Abstractors often failed to adhere to specifications for time-based data elements. For example, arrival time is defined as the ‘earliest documented time the patient arrived at the hospital’. In some cases, this field was automatically filled with patient registration time, which is not necessarily the earliest documented time. In other cases, abstractors were reluctant to use the earliest time the patient was seen in the hospital, particularly if treatment was delayed, because the physician had not yet arrived to take care of the patient. The abstractors essentially changed the definition to abstract times more closely associated with physician arrival. A third issue was the fact that times were frequently scattered throughout the medical record, making them difficult to find. Records also contained different times for the same event (e.g. an emergency department nurse documents that blood was drawn at a certain time, but the lab indicated the draw time to be different). Interview findings indicated that those hospitals which dealt most successfully with this issue had standardized clocks or had designated a specific clock as the ‘official’ timepiece for a particular case.
c. Weak abstraction guidelines and/or data element definitions
In one instance, the definition of the initial ECG interpretation data element was reported by abstractors to be unclear and/or incomplete. As a result, abstractors did not have a clear understanding of how to abstract this element. This definition has since been augmented and clarified by the Joint Commission and CMS and changes were implemented.
d. Technical issues
Structured interviews identified technical issues related to data collection software resulting in inconsistent or inaccurate indicator data. In several cases, birth weights were recorded in the medical record in pounds/ounces. The data collection software inaccurately calculated the conversion of pound/ounces into grams, creating discrepancies. Hospitals were encouraged to contact their vendor’s technical support office to have the issue corrected.
On-site interviews revealed that only eight of the 30 participating hospitals (27%) conducted any form of internal reviews or audits (formal or informal) to assess accuracy of data abstraction. Hospital-based data quality auditing could improve data quality and reduce (but not eliminate) the need for independent reabstraction. Future research should evaluate the use and potential benefits of self-reabstraction within hospitals.
|
Limitations and conclusions |
|---|
|
TopMethodsResultsDiscussionLimitations and conclusionsReferences |
|---|
The study has a number of limitations. First, data used were abstracted from the first available period of standardized indicator data collection. Results, therefore, may under-represent the current state of indicator reliability, if abstractor experience is considered a factor. The data presented in this study do not account for the impact of subsequent efforts to improve the accuracy and consistency of data collection. In 2004, the Joint Commission began requiring vendors to independently assess reliability within client hospitals. Similarly, the CMS quarterly validation study of hospital data may influence abstractor practices. Additional research is needed to evaluate the impact on abstractor performance. Another limitation was the limited record sample size at each participating hospital, which does not permit a statistically valid analysis at the hospital level or by specific hospital characteristic. The limited sample size also made it difficult to detect statistically significant differences on calculated indicator rates and precluded an assessment of between-hospital variability in coding practices. Similarly, statistical analysis was limited by the random sampling of records where rare events were the focus of the indicator (i.e. inpatient neonatal mortality). Because few numerator cases were observed among the randomly selected records, it was difficult to evaluate a comparison of calculated indicator rates.
If indicator data are to be effectively used to stimulate and track improvement efforts, users must have confidence in the reliability of the data, because concerns over data quality can influence the responses of organizations to indicator feedback. Importantly, our results did not reveal any evidence to suggest ‘gaming’, which is often a source of anxiety about self-reported data. As indicator data continue to be used by stakeholders for public reporting or pay-for-performance initiatives, ongoing monitoring of data quality is critical to maintain confidence in the data. It will be important to examine issues related to variation and the randomness of discrepancies to establish evidence-based thresholds of acceptable reliability.
|
Acknowledgments |
|---|
|
TopMethodsResultsDiscussionLimitations and conclusionsReferences |
|---|
Support for this study was provided through grant 1U18HS013728-01 from the US Department of Health and Human Service’s Agency for Healthcare Research and Quality. Assistance in the conduct of this study was provided by Linda Hanold, MHSA, Michele Bozikis, MPH, Gail Bielanski, RHIA and Carolyn Gillespie, RN. In addition, the authors would like to thank the considerable number of people who contributed to the development and testing of these standardized performance indicators, including, but not limited to, Sharon Sprenger, RHIA, CPHQ, MPA, and the Joint Commission core measure development team, Harlan Krumholz MD, MSC and the cardiovascular expert panel, Peter A Gross, MD and the pneumonia expert panel, Michael G Ross, MD, MPH and the pregnancy and related conditions expert panel.
|
References |
|---|
|
TopMethodsResultsDiscussionLimitations and conclusionsReferences |
|---|
- Joint Commission on Accreditation of Healthcare Organizations. Specification manual for national hospital quality measures: http://www.jcaho.org/pms/core+measures/aligned_manual.htm Accessed 13 October 2005.
- Joint Commission on Accreditation of Healthcare Organizations. A comprehensive review of development and testing for national implementation of hospital core measures: http://www.jcaho.org/pms/core +measures/cr_hos_cm.htm Accessed 13 October 2005.
- Braun BI, Koss RG, Loeb JM. Integrating performance measure data into the Joint Commission accreditation process. Eval Health Prof 1999; 22: 283–297.
[Abstract/Free Full Text] - Turpin RS, Darcy LA, Koss RG et al. A model to assess the usefulness of performance indicators. Int J Qual Health Care 1996; 8: 321–329.
[Abstract/Free Full Text] - Williams SC, Schmaltz SP, Morton DM, Koss RG, Loeb JM. Quality of care in U.S. hospitals as reflected by standardized measures, 2002-2004. N Engl J Med 2005; 353: 255–264.
- Centers for Medicare and Medicaid Services (CMS) Office of Public Affairs. Medicare "pay for performance (P4P)" initiatives. January 31, 2005: http://www.cms.hhs.gov/media/press/release.asp?Counter=1343 Accessed 13 October 2005.
- Kuhn H. Statement on pay for performance initiatives. Testimony to the U. S. Congress Subcommittee on Health of the Committee on Ways and Means, delivered March 15, 2005: http://www.hhs.gov/asl/testify/t050315a.html Accessed 13 October 2005.
- Joint Commission Resources. Joint Commission International Update. International hospital quality measures approved: http://www.jcrinc.com/subscribers/intlnewsletter.asp?durki=10118#story4 Accessed 13 October 2005.
- Loeb JM.The current state of performance measurement in health care. Int J Qual Health Care 2004; 16 (suppl. 1): 15–19.
- Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM. Data feedback efforts in quality improvement: lessons learned from US hospitals. Qual Saf Health Care 2004; 13: 26–31.
[Abstract/Free Full Text] - Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM. A qualitative study of increasing beta-blocker use after myocardial infarction: why do some hospitals succeed? JAMA 2001; 285: 2604–2611.
[Abstract/Free Full Text] - Hibbard JH, Stockard J, Tusler M. Does publicizing hospital performance stimulate quality improvement efforts? Health Aff (Millwood) 2003; 22: 84–94.
[Abstract/Free Full Text] - Joint Commission on Accreditation of Healthcare Organizations. Specification Manual for National Implementation of Hospital Core Measures, Version 1.06. Oakbrook Terrace, IL: Joint Commission on Accreditation of Healthcare Organizations, 2002.
- Reisch LM, Fosse JS, Beverly K et al. Training, quality assurance, and assessment of medical record abstraction in a multisite study. Am J Epidemiol 2003; 157: 546–551.
[Abstract/Free Full Text] - Thomas EJ, Studdert DM, Brennan TA. The reliability of medical record review for estimating adverse event rates. Ann Intern Med 2002; 136: 812–816.
[Abstract/Free Full Text] - Cassidy LD, Marsh GM, Holleran MK, Ruhl LS. Methodology to improve data quality from chart review in the managed care setting. Am J Manag Care 2002; 8: 787–793.[Web of Science][Medline]
- Labelle J, Swaine BR. Reliability associated with the abstraction of data from medical records for inclusion in an information system for persons with a traumatic brain injury. Brain Inj 2002; 16: 713–727.[CrossRef][Web of Science][Medline]
- Watt A, Williams S, Lee K, Robertson J, Koss RG, Loeb JM. Keen eye on core measures. Joint Commission data quality study offers insights into data collection, abstracting processes. J AHIMA 2003; 74: 21–25.
- Centers for Medicare and Medicaid Services. Hospital Quality Initiative (HQI): http://www.cms.hhs.gov/quality/hospital/ Accessed 13 October 2005.
- Centers for Medicare and Medicaid Services. Specifications for calculating hospital validation results. September 29, 2004: http://qnetexchange.org/public/docs/hdc/datavldtn/calc_specs.pdf Accessed 13 October 2005.
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–174.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  A. R. Campbell, D. Satran, D. M. Larson, I. J. Chavez, B. T. Unger, B. P. Chacko, C. Kapsner, and T. D. Henry ST-Elevation Myocardial Infarction: Which Patients Do Quality Assurance Programs Include? Circ Cardiovasc Qual Outcomes, November 1, 2009; 2(6): 648 - 655. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. Pulcini, S. Defres, I. Aggarwal, D. Nathwani, and P. Davey Design of a 'day 3 bundle' to improve the reassessment of inpatient empirical antibiotic prescriptions J. Antimicrob. Chemother., June 1, 2008; 61(6): 1384 - 1388. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. Greenfield and J. Braithwaite Health sector accreditation research: a systematic review Int. J. Qual. Health Care, June 1, 2008; 20(3): 172 - 183. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. C. Williams, R. G. Koss, D. J. Morton, S. P. Schmaltz, and J. M. Loeb Case volume and hospital compliance with evidence-based processes of care Int. J. Qual. Health Care, April 1, 2008; 20(2): 79 - 87. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. Marwick, E. Watts, J. Evans, and P. Davey Quality of care in sepsis management: development and testing of measures for improvement J. Antimicrob. Chemother., September 1, 2007; 60(3): 694 - 697. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 B. E. Landon, S.-L. T. Normand, A. Lessler, A. J. O'Malley, S. Schmaltz, J. M. Loeb, and B. J. McNeil Quality of Care for the Treatment of Acute Medical Conditions in US Hospitals Arch Intern Med, December 11, 2006; 166(22): 2511 - 2517. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. C. Williams, R. G. Koss, D. J. Morton, and J. M. Loeb Performance of Top-Ranked Heart Care Hospitals on Evidence-Based Process Measures Circulation, August 8, 2006; 114(6): 558 - 564. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||