OUP user menu

Limitations of using same-hospital readmission metrics

Sheryl M. Davies, Olga Saynina, Kathryn M. McDonald, Laurence C. Baker
DOI: http://dx.doi.org/10.1093/intqhc/mzt068 633-639 First published online: 28 October 2013

Abstract

Objective To quantify the limitations associated with restricting readmission metrics to same-hospital only readmission.

Design Using 2000–2009 California Office of Statewide Health Planning and Development Patient Discharge Data Nonpublic file, we identified the proportion of 7-, 15- and 30-day readmissions occurring to the same hospital as the initial admission using All-cause Readmission (ACR) and 3M Corporation Potentially Preventable Readmissions (PPR) Metric. We examined the correlation between performance using same and different hospital readmission, the percent of hospitals remaining in the extreme deciles when utilizing different metrics, agreement in identifying outliers and differences in longitudinal performance. Using logistic regression, we examined the factors associated with admission to the same hospital.

Results 68% of 30-day ACR and 70% of 30-day PPR occurred to the same hospital. Abdominopelvic procedures had higher proportions of same-hospital readmissions (87.4–88.9%), cardiac surgery had lower (72.5–74.9%) and medical DRGs were lower than surgical DRGs (67.1 vs. 71.1%). Correlation and agreement in identifying high- and low-performing hospitals was weak to moderate, except for 7-day metrics where agreement was stronger (r = 0.23–0.80, Kappa = 0.38–0.76). Agreement for within-hospital significant (P < 0.05) longitudinal change was weak (Kappa = 0.05–0.11). Beyond all patient refined-diagnostic related groups, payer was the most predictive factor with Medicare and MediCal patients having a higher likelihood of same-hospital readmission (OR 1.62, 1.73).

Conclusions Same-hospital readmission metrics are limited for all tested applications. Caution should be used when conducting research, quality improvement or comparative applications that do not account for readmissions to other hospitals.

  • readmissions
  • quality indicators
  • hospital quality

Introduction

With nearly 20 % of Medicare hospital discharges followed by readmission to an acute care facility within 30 days, and the resulting $15 billion in spending by the Centers for Medicare and Medicaid (CMS), readmissions have prominence as a quality metric, endorsed by the National Quality Forum (NQF), and used for payment adjustment by CMS [13].

In addition, readmissions metrics are used as quality improvement tools within hospitals. Readmissions have the advantage of being more frequent than mortality and potentially reflect a wide variety of quality processes occurring in hospitals and during care transitions [47].

Readmissions may be to the same hospital as the index admission or to a different hospital. Studies have previously found that readmissions to different hospitals occur in ∼30% of all readmissions [8]. Although NQF endorsed metrics and those used in most payment adjustment programs require the inclusion of readmissions to any hospital, not only the index hospitals, all-payer data that allow for the identification of readmissions at all hospitals are often unavailable to hospitals wishing to conduct quality improvement studies, or at least delayed several years, limiting its utility. As a result, timely available data usually contain a high rate of missing information, namely readmissions that occur to another hospital. These incomplete data are then used to calculate readmissions for some research and local quality improvement initiatives.

This incomplete information is particularly problematic when the missing information due to examining only same-hospital readmissions differs systematically between hospitals. Nasir et al. found poor correspondence between same-hospital readmission rates and all-hospital readmission rates, suggesting that bias due to missing data is systematic [8].

This study aims to more closely examine the implications of using only same-hospital readmissions compared with two typical all-hospital readmission metrics. To inform choices and interpretation of current applications of readmissions metrics, we focus on the relationship between easily identifiable hospital and patient characteristics and the proportion of readmissions that occur to the same hospital.

Methods

Data used

We used 2000–2009 California Office of Statewide Health Planning and Development Patient Discharge Data. The dataset includes information of ∼9 million unique discharges from acute care hospitals in California. Each record has an encrypted patient identifier and contains a range of information about the hospitalization including diagnoses, treatments, length of stay (LOS) and patient characteristics.

We analyzed medical and surgical admissions (excluding obstetric and psychiatric admissions) for patients aged 18 and older as index admissions. We did not allow admissions in December 2009 to serve as index admissions because 30-day readmissions would not be captured in the dataset. We also excluded admissions that resulted in transfer to another acute care hospital, discharge against medical advice, or death because readmission is not relevant. In identifying readmissions, linkages between index and subsequent admissions were made using patient identifier, date of birth and gender.

Readmissions metrics

We calculated 7-, 14- and 30-day readmission measures using two metrics: All-cause Readmission (ACR), developed for this research project, and the 3M Corporation Potentially Preventable Readmissions (PPR) [9] for each hospital and year. These metrics represent two broad categories of readmission metrics—those that include all readmissions and those that restrict to potentially related readmissions. The unit of analysis was hospital-year. We did not calculate a readmission rate for years in which hospitals had <20 index admissions, overall, surgical, medical or for each DRG.

ACRs

The ACR metric, used in a prior study [10], includes nearly all readmissions to patients admitted to a hospital for an index hospitalization but excludes cases meeting certain definitions that may reflect planned or entirely unrelated subsequent admissions, specifically trauma, malignancies, obstetric, transplants and cardiac procedures following acute myocardial infarction (AMI). Readmissions on the same day as the index admission are excluded, as these are likely to reflect transfers rather than true readmissions.

PPR

PPR is a proprietary algorithm developed by 3M Corporation aimed at reducing the number of unrelated readmissions included. Using International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM), the algorithm identifies readmissions with diagnoses or procedures that are clinically related to the index admission diagnoses and procedures and labels those readmissions as ‘potentially preventable.’ The algorithm also employs a chain logic, which combines multiple readmissions in the same patient into one readmission event, essentially measuring whether a patient has any qualifying readmissions, not the number of readmissions in cases where patients have more than one re-hospitalization. The development of the PPR algorithm has been previously described [9].

Case mix adjustment

We used the all patient refined-diagnostic related group (APR-DRG) Risk of Mortality subclass for case mix adjustment. We also applied the Agency for Healthcare Research and Quality (AHRQ) Comorbidity Index [11]. All metrics are reported as risk-adjusted, or the observed to expected, rates.

Analyses

We identified 10 medical and surgical APR-DRGs with the highest absolute number of readmissions in our data. We then calculated the proportion of all readmissions that occurred to a different hospital than the index admission for all hospitalizations, for each of the identified APR-DRGs, and for each hospital for each year of data. These included other pneumonia (APR-DRG 139), chronic obstructive pulmonary disorder (COPD) (140), heart failure (HF) (194), angina and coronary atherosclerosis (198), cerebral vascular accident (45), renal failure (460), kidney and urinary tract infection (463), cardiac arrhythmia and conduction disorder (201), AMI, initial episode of care (190), pacemaker without AMI, HF or shock (171), other vascular (173), percutaneous cardiac procedures with AMI (174), percutaneous cardiac procedures without AMI (175), major small and large bowel disease (221), laparoscopic cholecystectomy (263), Hip joint replacement (301), knee joint replacement (302), hip and femur procedures for trauma, except joint replacement (308) and uterine and adnexa procedures for non-malignancy except leiomyoma (513).

We quantified the impact of using a metric based on only same-hospital readmissions compared with using readmissions to any hospital (all-hospital) in three ways. All tests were run separately for index admissions with (i) medical DRGs, (ii) surgical DRGs and (iii) for each of the APR-DRGs with the highest absolute number of readmissions (as listed in the previous paragraph). First, we calculated the correlation between hospital-level same-hospital and all-hospital risk-adjusted readmission rates. Second, we calculated the change in relative performance when using the two metrics. We examined the percent of hospitals within the top and bottom deciles using the all-hospital readmission metric that remain in the decile after applying the same-hospital metric and the percent that move two or more deciles. We also used a Bland–Altman plot to visualize the level of agreement between the two metrics [12]. Third, we examined the impact of restricting analyses to same-hospital readmissions when examining longitudinal trends within the same hospital using three statistics. To create comparable early and late cohort with temporal separation, we divided the data into three time periods, 2000–2002, 2003–2006 and 2007–2009, and calculated the change in readmission rates from combined years 2000–2002 to combined years 2007–2009. We calculated the Pearson correlation coefficient for change in performance using same-hospital and all-hospital readmission rates. We also calculated the kappa statistic for detecting a significant change (P < 0.05) in readmission rates between the first and last time periods. Because kappa statistics are sensitive to rare outcomes [13], we also calculated the percent agreement for detecting a significant change (P < 0.05) between the first and last time periods.

To examine the factors associated with readmission location, we performed a mixed-model logistic regression, clustered by hospital to account for hospital effects. The dependent variable was 30-day readmission (both ACR and PPR separately to the same hospital and independent variables included patient age (18–44, 45–64, 65–74, 75–84 and 85+), gender (male and female), discharge location (home, skilled nursing facility or residential care and other), distance traveled (6–10, 11–25, 26–50 and 51+ miles), LOS (0–1, 2–3, 4–6, 7–10, 11–20, 21–50 and 51+ days), primary payer (Medicare, MediCal, Self-pay and other), hospital ownership (Not-for-profit, state, University of California hospital and private), bed size (three terciles) and teaching status (teaching and non-teaching). We added APR-DRG and AHRQ Comorbidity Index to the model to control for case mix.

Results

Overall, 68.0% of 30-day ACR and 70.1% 30-day PPR occurred to the same hospital as the index admission (Table 1). There were no statistically significant differences between metrics (ACR or PPR), or 7-, 15- or 30-day readmissions (range 67.1–74.0%). Among surgical APR-DRGs with high absolute numbers of readmissions, abdominopelvic APR-DRGs (laparoscopic cholecystectomy, large and small bowel procedures and uterine procedures) had the highest number of readmissions occurring to the same hospital (87.4–88.9%), whereas cardiac APR-DRGs (percutaneous cardiac intervention and ICD) had the lowest number of readmissions occurring to the same hospital (72.5–74.9%). In general, medical APR-DRGs had a lower proportion of same-hospital 30-day ACR than surgical APR-DRGs (67.1 vs. 71.1%).

View this table:
Table 1

Readmission rates and percent readmitted to same hospital as index admission

PopulationNPPRa rate (%)ACRb rate (%)Percent readmitted to same hospital
7 days30 days7 days30 days30-day PPR30-day ACR
All11 564 7042.45.43.410.170.167.5
Surgicalc4 280 1121.63.62.46.672.071.1
Medicald7 284 5922.86.64.012.170.067.1
Lower extremity joint procedurese430 7751.34.01.96.778.076.9
Abdominopelvic proceduresf766 9231.11.91.83.885.685.0
Cardiac and vascular proceduresg313 7312.76.53.811.076.476.7
Heart failure421 3214.812.75.819.977.073.7
Pneumonia425 2722.97.23.711.676.975.5
COPD259 8614.211.44.716.276.874.9
  • aPotentially preventable readmissions. bAll-cause Readmission. cSurgical APR-DRGs for index admission. dMedical APR-DRGs for index admission. eHip and femur procedures, knee joint replacement and hip joint replacement. fMajor small and large bowel disease, laparoscopic cholecystectomy and uterine procedures. gPacemaker placement, other vascular, percutaneous procedures with and without AMI.

Table 2 shows the impact on relative hospital performance of restricting analyses to same-hospital readmissions. The largest change in performance was seen for ACR, where about half of hospitals remained in the worst-performing decile following application of the all-hospital metric, and just shy of 1/3 of hospitals moved two or more deciles. The Pearson correlation coefficients for same-hospital readmission rates and all-hospital rates ranged from weak (r = 0.23) for 15-day PPR for surgical patients to strong for 7-day ACR for all patients (r = 0.80). The agreement between same-hospital and all-hospital readmission metrics in identifying high- and low-performing outliers was moderate (Table 2). In general, agreement for medical patients was lower than for surgical patients, and agreement using the PPR metric was higher than using the ACR metric.

View this table:
Table 2

Relationships between same-hospital and all-hospital readmission metrics: correlation, relative performance and Kappa scores

ACRPPR
MetricPearson correlationaDecile changeKappabPearson correlationaDecile changeKappab
% Stablec% changing 2+ deciledLoweHighf% Stablec% changing 2+ deciledLoweHighf
LoweHighfLoweHighfLoweHighfLoweHighf
7-day, all patients0.8050.557.730.423.00.500.600.5152.558.323.216.50.520.65
15-day, all patients0.5851.159.529.021.60.470.590.4557.860.319.318.10.510.64
30-day, all patients0.6050.957.128.624.00.460.560.4454.863.018.417.20.500.65
7-day, surgicalg0.46N/AN/AN/AN/A0.600.750.24N/AN/AN/AN/A0.540.72
15-day, surgical0.43N/AN/AN/AN/A0.560.760.23N/AN/AN/AN/A0.490.76
30-day, surgical0.37N/AN/AN/AN/A0.510.740.24N/AN/AN/AN/A0.500.75
7-day, medicalh0.80N/AN/AN/AN/A0.480.560.48N/AN/AN/AN/A0.480.59
15-day, medical0.58N/AN/AN/AN/A0.390.500.43N/AN/AN/AN/A0.470.59
30-day, medical0.58N/AN/AN/AN/A0.380.480.41N/AN/AN/AN/A0.460.58
  • aAll correlation coefficients significant at the P < 0.0001 level. bKappa score for identifying hospitals as low- or high-performing outliers (outside the 95% confidence interval).  cAverage percent of hospitals remaining in extreme deciles when using same-hospital vs. all-hospital readmission metrics. dAverage percent of hospitals that move two or more deciles when using same-hospital vs. all-hospital readmission metrics. eLowest-performing decile. fHighest-performing decile. gSurgical APR-DRGs for index admission. hMedical APR-DRGs for index admission. ACR, all-cause Readmissions; PPR, potentially preventable readmissions.

Figure 1 shows selected Bland–Altman plots for all-hospital readmission rates compared with same-hospital readmission rates. These again demonstrate the tighter agreement between the metrics for surgical patients as compared with medical patients.

Figure 1

Bland–Altman Plots for the average between same-hospital readmission rate and all-hospital readmission rate against the differences between the rates, shown for four different 30-day metrics (medical PPR, surgical PPR, medical ACR and surgical ACR). Larger dispersion shows poorer agreement between the metrics.

Figure 2 shows the difference in measured change in longitudinal performance on ACR and PPR when using same- vs. all-hospital readmissions. The agreement for detecting a significant (P < 0.05) longitudinal change, measured by percent agreement and kappa statistic, was poor (Kappa range 0.05–0.11). Correlations for surgical cases were lower than those for medical cases. The scatterplots show a tight distribution for surgical cases where no longitudinal change was observed but less tightly distributed where observed change was more substantial.

Figure 2

Scatterplots and kappa coefficients of longitudinal change as measured by all-hospital readmission rate vs. same-hospital readmission rate.

Controlling for APR-DRG and comorbidity, distance traveled was the most predictive factor of readmission location (same vs. other hospital), with patients estimated to have traveled longer distances having a lower risk of 30-day ACR readmission to the same hospital (Table 3). Payer was also significantly predictive, with MediCal and Self-Pay patients having a lower likelihood of same-hospital readmission. Length of stay, i.e. 1 day or less or over 21 days, was associated with lower odds of being readmitted to the same hospital. Results were similar using readmission location for 30-day PPR as the dependent variable.

View this table:
Table 3

Logistic regression of 30-day ACR to the same hospital as index admission

FactorOdds ratio (95% Wald confidence limits)
Age 18–441.00
Age 45–641.00 (0.97–1.03)
Age 65–741.08 (1.03–1.13)a
Age 75–841.16 (1.11–1.22)a
Age 85 +1.28 (1.22–1.35)a
Female gender1.07 (1.06–1.09)
LOS 0–1 days0.87 (0.85–0.89)a
LOS 4–6 days1.06 (1.04–1.07)a
LOS 7–10 days1.00 (1.02–1.07)
LOS 11–20 days0.97 (0.94–1.00)a
LOS 21–50 days0.75 (0.67–0.84)a
LOS 51 days +0.71 (0.57–0.89)
Payer—Private1.00
Payer—Medicare0.91 (0.87–0.95)a
Payer—MediCal0.77 (0.72–0.81)a
Payer—Self Pay0.53 (0.46–0.61)a
Payer—Other0.88 (0.78–0.98)
For-profit hospital1.00
Not-for-profit hospital1.61 (1.46–1.78)a
State hospital1.70 (1.48–1.96)a
UC hospital1.65 (1.39–1.96)a
Small bed size1.00
Medium bed size1.30 (1.02–1.65)
Large bed size1.55 (1.27–1.90)
Distance traveled <6 milesb1.00
Distance traveled 6–10 milesb0.81 (0.77–0.86)a
Distance traveled 11–25 milesb0.59 (0.55–0.63)a
Distance traveled 26–50 milesb0.42 (0.39–0.45)a
Distance traveled 51+ milesb0.32 (0.29–0.34)a
Discharge to home1.00
Discharge to SNF or residential care0.90 (0.87–0.94)a
Discharge to other0.79 (0.72–0.88)a
Drug usec0.76 (0.73–0.80)a
Psychiatricc0.80 (0.78–0.84)a
  • aSignificant at P < 0.05 level. bCalculated using centroid of patient and hospital zip codes. cAgency for Healthcare Research and Quality Comorbidity Software. Only significant CCS listed. APR-DRG, all patient refined-diagnostic related groups; LOS, length of stay; SNF, skilled nursing facility; UC, University of California.

Discussion

Among California patients hospitalized during the last decade, we found modest differences in readmission rates when using only same-hospital readmissions compared with readmissions to all hospitals that persist for all uses of the metrics, suggesting that same-hospital readmissions metrics should be used with caution. Differences were most apparent when measuring relative hospital performance for medical patients. In addition, the agreement was only modest for longitudinal performance and when identifying significant longitudinal change, the agreement was poor, suggesting that same-hospital readmissions may bias results focusing on changes over time (e.g. evaluating quality improvement programs in a hospital).

In their examination of same-hospital readmission rates in Medicare fee-for-service patients, Nasir et al. found a slightly higher rate of 30-day heart failure readmissions to the same hospital (80.9%) than that observed in this study (67.5%) but also found only moderate relationships between same-hospital and all-hospital metrics. The authors argued that same-hospital readmissions cannot stand as a proxy for all-hospital readmission rates [8]. However, these authors did not look specifically at agreement in evaluating hospital performance, an application for which readmission metrics are often used.

This study extends the examination of same-hospital readmission rates by focusing on the typical uses of quality metrics: identification of outliers and high- or low-performing hospitals and longitudinal change over time, finding consistent differences in the conclusion drawn when using only same-hospital readmissions vs. all-hospital readmission across each potential use. First, we found that the agreement between metrics in identifying outliers was modest (Kappa range 0.38–0.76). Agreement was higher between metrics for surgical populations than for medical populations, likely due to the higher proportion of readmissions occurring to the same hospital in surgical populations. The implications of using same-hospital readmission persisted regardless of the exact permutation of readmission metric used (ACRs vs. PPR) or the readmission period measured (7, 15 or 30 days). For example, 57–59% of hospitals categorized in the highest decile for all-hospital readmissions remained in this part of the performance distribution for same-hospital readmissions regardless of time interval examined. The bias inherent in restricting metrics to same-hospital readmissions data suggests that same-hospital data may result in unfair characterization when readmission metrics are used in payment and public reporting schemes. This point is understood by policy-makers. As of the end year for this study (2009), six states had published public reports on readmissions [14]. All reports include all-hospital readmissions—the standard for national or state-level pay for performance initiatives and public reporting. All-hospital data are collected by organizations such as CMS [15] and AHRQ or specific states [16], but these data are either only available on a limited basis or delayed several years. Hospitals, researchers and other users may not have the ability to utilize data that link patients across hospitals and as a result local or hospital-based quality initiatives often rely only on same-hospital readmission data, with potential discrepancies in comparisons to ratings used for payment, as highlighted in this study.

Although the same-hospital readmission rate is limited in its ability to serve as a proxy for all-hospital readmissions, this limitation is primarily a missing data problem. In this study, we demonstrate that longer distance traveled, Medicaid or self-pay patients and longer LOS are associated with readmission to a different hospital than the index admission. This implies that same-hospital readmission rates will be most misleading for hospitals treating a large proportion of patients with these characteristics. Although this study provides only an initial investigation, such knowledge could allow for adjustment to account for potentially missing data, such as weighting readmission rates based on some of these factors. In addition, focused analyses on surgical conditions with higher proportions of same-hospital readmissions, such as abdominopelvic surgeries, may be appropriate.

This study is limited by its reliance on only one state, California. We were unable to identify readmissions to another state, which is particularly important in areas close to state borders. In addition, we estimated distance traveled based on county centroids, which may be inaccurate where urban centers span county lines. Finally, the use of administrative data limited the variables that could be included in a predictive model.

In this study, we found that identification of high- and low-performing hospitals and analyses of trends in readmissions over time are likely to differ when using same-hospital vs. all-hospital readmission metrics. Rates of medical readmissions, in particular, are likely to be biased when based on same-hospital metrics. Caution should be used when basing hospital evaluation—whether for payment, public reporting—on metrics that rely only on readmission to the same hospital as the index admission. For longitudinal evaluations within a single hospital, users should consider whether the hospital is at high risk for bias based on patient case mix or hospital characteristics. When linked data are unavailable, efforts should be made to account for the missing data from readmissions to other hospitals.

Funding

This work was supported by a grant from the Gordon and Betty Moore Foundation Grant No. 1983 and NIA Grants No. AG017253 and AG024957. We take sole responsibility for the design and conduct of the study; collection, management, analysis and interpretation of the data and preparation, review and approval of the manuscript.

Acknowledgements

We acknowledge Ellen Schultz for her assistance in preparing the manuscript.

References

View Abstract