International Journal for Quality in Health Care Advance Access originally published online on June 28, 2007
International Journal for Quality in Health Care 2007 19(4):225-231; doi:10.1093/intqhc/mzm017
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Administrative data outperformed single-day chart review for comorbidity measure
1 Health Observatory, Canton of Valais, Switzerland
2 Health Care Evaluation Unit, Institute of Social and Preventive Medicine, University of Lausanne, Switzerland
3 Centre for Infections Diseases and Epidemiology, Central Institute of the Valais Hospitals, Switzerland
4 Infection Control Programme, University of Geneva Hospitals, Switzerland
5 Centre for Health and Policy Studies, University of Calgary, Calgary, Alberta, Canada
6 Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada
7 Department of Medicine, University of Calgary, Calgary, Alberta, Canada
Address reprint requests to: Jean-Christophe Luthi, Health Observatory, Canton of Valais, Av. Grand-Champsec 86, CH-1950 Sion, Switzerland. Tel: +41 27 603 49 60; Fax: +41 27 603 49 51; E-mail: jean-christophe.luthi{at}ichv.ch
 |
Abstract |
|---|
 Top Abstract MethodsResultsDiscussionAcknowledgementsReferences |
|---|
Objective. The purpose of this article is to compare the Charlson comorbidity index derived from a rapid single-day chart review with the same index derived from administrative data to determine how well each predicted inpatient mortality and nosocomial infection.
Design. Cross-sectional study.
Setting. The study was conducted in the context of the Swiss Nosocomial Infection Prevalence (SNIP) study in six hospitals, canton of Valais, Switzerland, in 2002 and 2003.
Participants. We included 890 adult patients hospitalized from acute care wards.
Main outcome measures. The Charlson comorbidity index was recorded during one single-day for the SNIP study, and from administrative data (International Classification of Disease, 10th revision codes). Outcomes of interest were hospital mortality and nosocomial infection.
Results. Out of 17 comorbidities from the Charlson index, 11 had higher prevalence in administrative data, 4 a lower and two a similar compared with the single-day chart review. Kappa values between both databases ranged from – 0.001 to 0.56. Using logistic regression to predict hospital outcomes, Charlson index derived from administrative data provided a higher C statistic compared with single-day chart review for hospital mortality (C = 0.863 and C = 0.795, respectively) and for nosocomial infection (C = 0.645 and C = 0.614, respectively).
Conclusions. The Charlson index derived from administrative data was superior to the index derived from rapid single-day chart review. We suggest therefore using administrative data, instead of single-day chart review, when assessing comorbidities in the context of the evaluation of nosocomial infections.
Keywords: administrative data, comorbidities, death in hospital, nosocomial infection
Many health services researchers are measuring outcomes such as mortality or nosocomial infections. Among the different indexes available to control for comorbid conditions, the most frequently used is the Charlson index [1], which is a weighted score of 17 comorbidities that was initially used to predict in-hospital and 1-year mortality. It was then adapted for use with administrative data and in particular with the International Classification of Diseases, 9th revision, Clinical Modifications (ICD-9-CM) by different groups [2–5]. In the last decade, the International Classification of Disease, 10th revision (ICD-10), was introduced by the World Health Organization in many European countries, in Australia and in Canada [6]. Responding to this coding transition, Halfon et al. [7] in Switzerland and Sundararajan et al. [8] in Australia developed ICD-10 coding algorithms for the Charlson index. Subsequently, a new and more comprehensive ICD-10 coding algorithm for the Charlson index was developed by collaboration between three international research groups in Switzerland, Australia and Canada [9]. We assess this new and comprehensive algorithm in the present study.
Different approaches exist for measuring the severity of illness and predicting outcomes. Administrative data have been widely used by health service researchers in studies of outcomes, effectiveness, appropriateness and utilization of health care services [10–13], often with the use of the coding algorithms described above for defining comorbidities [2–5, 9]. A second traditional approach is to conduct a detailed chart review after discharge. This methodology is often used to assess outcomes and processes of care after hospital stay and, in some instances, has been used by researchers to evaluate the validity of administrative data [14–17]. A third approach occasionally used is to conduct a rapid single-day chart review during hospital stay. Such an approach has appeal because it can be done more quickly and at lower cost compared with a complete chart review. A second advantage is that it permits real time implementation and produces data while the patient is still in the hospital. These potential advantages lead to the use of the single-day chart review method for the measurement of comorbidity in the context of the Swiss Nosocomial Infection Prevalence (SNIP) study [18]. A single-day review conducted during patient hospitalization produces information on comorbidities that then permit appropriate risk-adjustment for comparing nosocomial infection rates across hospitals. In this study, we used the Charlson index derived from the SNIP study, since this was already available, and compared these results with the Charlson index derived from administrative data.
Of some concern, the performance of such single-day comorbidity assessments from chart reviews has, to our knowledge, never been formally assessed. Recognizing this, we conducted a formal study to assess the performance of this rapid assessment method for deriving the Charlson comorbidity index in comparison with administrative data. Our second objective is to compare the prognostic value of Charlson comorbidity index measures derived using the single-day chart review vs. administrative data for predicting the outcomes of in-hospital mortality and nosocomial infections.
|
Methods |
|---|
|
TopAbstractMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Setting
The SNIP study [18], a cross-sectional 1-week period prevalence study was conducted simultaneously in six acute care hospitals of the canton of Valais, Switzerland in 2002 and 2003, respectively, from May 27 to June 6 and from May 19 to 28. The canton of Valais is located in the middle of the Alps in the western part of Switzerland. These hospitals have between 100 and 240 beds and were participating voluntarily in the Swiss-Noso Network.
Data collection (SNIP study)
All adult patients hospitalized on internal medicine, surgery, orthopaedics and gynaecology wards and those in intensive care units were included in the SNIP study. Data collection took place during any single day while patients were hospitalized. Data abstracters were infection control practitioners who attended at least three specific 1-day training sessions. A standardized documentation was provided for each data abstractor by the Swiss-Noso team. This included the study protocol, standardized case report form, written definitions of all variables and a code list.
Data collection included demographic characteristics, information on the type of nosocomial infection, type of pathogens, severity of infections, risk factors, management and treatment. Details have been reported in a previous publication [19]. The modified Center for Disease Control and Prevention criteria were used to define nosocomial infections [20–21], which were recorded if present during the 7 days before the day of investigation. Data collected during the SNIP study included also the presence of all comorbidities from the Charlson index [17], which is a weighted average of selected comorbidities. While recording individual comorbidities that constitute the Charlson index, research nurses were instructed to report comorbidities retrospectively at the time of admission and specifically not to include the main cause of admission as comorbidities.
Administrative data
In the canton of Valais, Switzerland, routinely collected administrative data since the year 2000 are transmitted to the Health Observatory, by the six acute care community hospitals. The medical part of administrative data includes patients' demographic characteristics, discharge status including death, up to 10 diagnostics coded (ICD-10) and up to 10 procedures coded (International Classification of Disease-9-CM).
The Charlson comorbidity index was also calculated using administrative data and derived using the ICD-10 coding algorithm recently published by Quan et al. [9]. Information on hospital mortality was determined from administrative data as well.
Since administrative data are completely anonymous in Switzerland, we needed to merge both data sets (SNIP study and administrative data) using three variables: date of birth, date of admission and sex. We identified 1054 eligible patients (481 in 2002 and 573 in 2003) with the SNIP study. Then, after matching, 890 patients were available for the analysis (430 in 2002 and 460 in 2003). Thus, we had data from two sources for the same hospital admission.
Statistical analysis
First, descriptive statistics were employed to calculate the prevalence of individual comorbidities for data collected by infection control practitioners and administrative data. Agreement between the two datasets was measured using kappa values with corresponding 95% confidence intervals (95% CI) [22].
We then developed two logistic regression models for in-hospital mortality and two for nosocomial infection as outcome variables to assess model performance. For each outcome, one model used the Charlson comorbidity index computed from the SNIP study's single-day chart review and the second model used the Charlson index obtained from administrative data. We used the SAS procedure ‘proc logistic’ with the class option and effect coding for the Charlson index in each model. Age and sex were used as control variables. Odds ratios [23], 95% CIs, C statistics and Brier scores [24] were computed for each model in order to assess which model performed best in predicting the in-hospital mortality and nosocomial infection. The C statistic is the area under the receiver operating characteristic (ROC) curve, measuring the ability of the predictive model to discriminate among those who do or do not die at the hospital or those who do have a nosocomial infection and those who do not [25–26]. All analyses were conducted using SAS, version 8.02 (SAS institute inc., Cary, NC, USA).
|
Results |
|---|
|
TopAbstractMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Prevalence of comorbidities
Among 890 patients available for the analysis, the mean age (standard deviation) was 62.2 years (19.0 years), and 53.3% were female. Table 1 presents the prevalence of the 17 comorbidities from the Charlson index in administrative data and from the SNIP study. Eleven comorbidities were more frequently observed in administrative data than chart review data. Four were more frequently seen in the single-day chart review (i.e. diabetes with chronic complication, myocardial infarction, peptic ulcer disease and rheumatologic disease). The frequency of AIDS and of moderate or severe liver disease, meanwhile, was similar in both databases.
|
Agreement between data sources
The kappa value assessing the agreement between the 17 comorbidities of the Charlson index in the SNIP study compared with administrative data is represented in Table 1. For the purposes of interpretation, kappa values can be categorized into five groups: < 0.2 (poor agreement), 0.21–0.40 (fair agreement), 0.41–0.60 (moderate agreement), 0.61–0.80 (substantial agreement) and 0.81–1.00 (near perfect agreement) according to Landis and Koch [27]. Using this categorization, we found four comorbidities with poor agreement, seven with fair agreement and six with moderate agreement.
The agreement of the Charlson index score between both databases is shown in Table 2. The numbers in bold represent perfect agreements between the two data sources. The overall kappa value was 0.30 (95% CI: 0.26–0.34), and the weighted kappa value 0.44 (95% CI: 0.39–0.49).
|
Predicting outcomes
Of the sample of 890 patients, 26 died in the hospital and 51 had a nosocomial infection. Table 3 presents the detailed number of persons who died or who had a nosocomial infection for each Charlson score calculated based on administrative data or assessed by the SNIP study single-day chart review. When the Charlson index score increases, the proportion of patients dying in hospital or getting nosocomial infections increases, in both databases, but more notably with administrative data.
|
Table 4 presents the results of the four logistic regression models with adjusted odds ratios and measure of discrimination, using hospital mortality and nosocomial infections as outcome variables. The adjusted odds ratio for in-hospital mortality for administrative data was 2.07 (95% CI: 0.88–4.87) for a Charlson score of 1, 0.39 (95% CI: 0.11–1.33) for a Charlson score of 2, 3.02 (95% CI: 1.03–8.82) for a Charlson score of 3 and 33.85 (95% CI: 16.37–69.98) for a Charlson score
4, compared with Charlson score 0. These odds ratios were higher than that for a chart review data. Further, for in-hospital mortality, the C statistic was also higher for administrative data (C = 0.863) compared with single-day chart review data (C = 0.795). The Brier score was smaller for administrative data compared with single-day chart review data; a finding that also indicates better performance for the administrative data model.
|
For nosocomial infections, odds ratio ranged from 0.22 to 6.46 and from 0.70 to 3.51, respectively, for administrative data and single-day chart review from the SNIP study when the Charlson score increased from 1 to
4, compared with Charlson score 0. Odds ratios were therefore close for both databases for this outcome. The areas under the ROC curve were 0.645 for administrative data and 0.614 for the single-day chart review data. |
Discussion |
|---|
|
TopAbstractMethodsResultsDiscussionAcknowledgementsReferences |
|---|
We compared administrative data and a rapid single-day chart review data in identifying Charlson comorbidities for the purposes of risk adjustment. Our results demonstrate only poor to fair agreement for the majority of 17 Charlson comorbidities. The administrative data reported more cases and predicted hospital mortality and nosocomial infections better, compared with rapid single-day chart review. In addition, odds ratios and C statistics calculated using both databases were lower for nosocomial infections compared with hospital mortality, suggesting that the Charlson comorbidity index predicted the in-hospital mortality better than it did in nosocomial infections.
There are three possible explanations for these results. The first relates to completeness of clinical information in the chart. Single-day review data are generated based on information documented at the initial stage of hospitalization, but administrative data are generated after discharge. Conditions present at admission might be incompletely documented in the chart at the beginning of hospitalization and the chart gradually becomes ‘richer’ in clinical content, as documents such as the discharge summary, consultation notes, pathology reports, surgical reports or discharge letters are fully compiled. We performed secondary analyses to explore this hypothesis and to asses if our results were robust with respect to the timing of the single-day chart review. Kappa values between both databases were effectively higher for the majority of comorbidities if the review was performed close to discharge compared with early in the hospital stay. A second possible explanation is that in the SNIP study, abstractors were instructed to exclude the main cause of admission in the Charlson score. However, administrative data included conditions present at both stages without such distinction. Most previous publications demonstrate that administrative data generally under report Charlson comorbidities compared with entire chart review and agreement between two databases varies by condition.
A study conducted in Calgary, Canada, reported kappa values across comorbidities ranging from a high of 0.87 to a low of 0.34 [14]. In another study conducted in six general medicine wards at Yale (in New Haven, CT, USA), the kappa value for the Charlson index score was 0.35 and ranged between 0.00 and 0.83 among the different comorbidities [17]. Results of our study are slightly lower in terms of kappa values in comparison with these two North-American studies [14, 17], presumably because those studies assessed agreement between administrative data and full chart review done after discharge, whereas ours assessed agreement with the single-day chart review. In another study conducted among Medicare beneficiaries in the State of Georgia (USA), kappa values ranged from 0.12 to 0.68, with most agreement being fair to moderate, as in our study [16]. In this study, the authors found that the odds ratio for in-hospital mortality was 10.15 for the chart index and 2.06 for the ICD-9 index. Further, the area under the ROC curve was 0.697 for a multivariate model incorporating the chart index and 0.639 for the model using the administrative index; these results are much lower compared with the C statistics obtained in our study for hospital mortality as outcome [16]. Further, our results are also comparable with another published study, using complete chart review compared with administrative data [28].
The rapid single-day chart review has its value for collection of clinical information for certain conditions. We found two comorbidities (i.e. AIDS and moderate or severe liver diseases) that have been detected at the similar level by both methods and four conditions with a higher level of detection by the single-day chart review (i.e. diabetes with chronic complication, myocardial infarction, peptic ulcer diseases and rheumatologic diseases). However, administrative data predicted the hospital mortality and nosocomial infection better than rapid single-day chart review. The main advantage of this rapid single-day chart review might be that it can be implemented in less time with much less resources, when compared with a complete chart review after discharge. Therefore, results might be available more quickly. However, despite the fact that such single-day review is cheaper than complete chart review, we recommend not using this method when administrative data are available, because administrative data represent an even less expensive way to derive comorbidity information, with relatively strong performance in comorbidity measurement and outcome prediction.
One limitation is that the study was implemented in one Swiss canton, and thus might not apply to other cantons or even countries. Another potential caveat in the interpretation of the study results is the well-described limitations of administrative data. Inadequate assessment of comorbidity using administrative data has been documented previously [29]. However, most hospitals are actively working to improve the accuracy of their administrative coding systems. In the canton of Valais, Switzerland, the coding has been professionalized, first in one hospital since 2000 and then in all hospitals since 2003 with the creation of a cantonal Coding Unit, which has standardized and professionalized coding rules and methods. One study assessed the quality of coding in the canton of Valais, showing high quality coding in 2003 and major improvements since the year 2000 [30].
In conclusion, for adult patients from six community hospitals, comorbidities from the Charlson index were more often detected using administrative data compared with a rapid single-day chart review. Further, the Charlson index derived from administrative data was shown to be slightly superior to the index derived from the rapid single-day chart review as a risk predictor and, therefore, as adjustment measure. Therefore, we recommend using administrative data in order to get information on comorbidities rather than using the single-day chart review method that was in use in the SNIP study, assessing nosocomial infections rates.
|
Acknowledgements |
|---|
|
TopAbstractMethodsResultsDiscussionAcknowledgementsReferences |
|---|
The methodological development of the study was performed in the context of a Swiss National Sciences Foundation supported study (grant 3200B0-103852) that used a different dataset.
|
References |
|---|
|
TopAbstractMethodsResultsDiscussionAcknowledgementsReferences |
|---|
- Charlson ME, Pompei P, Ales KL, et al. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis (1987) 40:373–83.[CrossRef][Web of Science][Medline]
- Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol (1992) 45:613–9.[CrossRef][Web of Science][Medline]
- Romano PS, Roos LL, Jollis JG. Adapting a clinical comorbidity index for use with ICD-9-CM administrative data: differing perspectives. J Clin Epidemiol (1993) 46:1075–9.[CrossRef][Web of Science][Medline]
- D'Hoore WD, Bouckaert A, Tilquin C. Practical considerations on the use of the Charlson comorbidity index with administrative data bases. J Clin Epidemiol (1996) 49:1429–33.[CrossRef][Web of Science][Medline]
- Ghali WA, Hall RE, Rosen AK, et al. Searching for an improved clinical comorbidity index for use with ICD-9-CM administrative data. J Clin Epidemiol (1996) 49:273–8.[CrossRef][Web of Science][Medline]
- Anonymous. Implementation of the International statistical classification of diseases and related health problems, 10th revision (ICD-10). Epidemiol Bull (1997) 18:1–4.[Medline]
- Halfon P, Eggli Y, van Melle G, et al. Measuring potentially avoidable hospital readmissions. J Clin Epidemiol (2002) 55:573–87.[CrossRef][Web of Science][Medline]
- Sundararajan V, Henderson T, Perry C, et al. New ICD-10 version of the Charlson Comorbidity Index predicted in-hospital mortality. J Clin Epidemiol (2004) 57:1288–94.[CrossRef][Web of Science][Medline]
- Quan H, Sundararajan V, Halfon P, et al. Coding algorithm for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care (2005) 43:1130–9.[CrossRef][Web of Science][Medline]
- Roos NP, Black C, Roos LL, et al. Managing health services: how administrative data and population-based analyses can focus the agenda. Health Serv Manage Res (1998) 11:49–67.[Medline]
- Roos NP, Black C, Roos LL, et al. Managing health services: how administrative data and population-based analyses can focus the agenda. Health Serv Manage Res (1998) 11:49–67.[Medline]
- Magid DJ, Calonge BN, Rumsfeld JS, et al. Relation between hospital primary angioplasty volume and mortality for patients with acute MI treated with primary angioplasty vs. thrombolytic therapy. JAMA (2000) 284:3131–8.
[Abstract/Free Full Text] - Schrag D, Cramer LD, Bach PB, et al. Influence of hospital procedure volume on outcomes following surgery for colon cancer. JAMA (2000) 284:3028–35.
[Abstract/Free Full Text] - Quan H, Parsons GA, Ghali WA. Validity on information on comorbidity derived from ICD-9-CM administrative data. Med Care (2002) 40:675–85.[CrossRef][Web of Science][Medline]
- Preen DB, Holman CDAJ, Lawrence DN, et al. Hospital chart review provided more accurate comorbidity information than data from general practitioner survey or administrative data base. J Clin Epidemiol (2004) 57:1295–304.[CrossRef][Web of Science][Medline]
- Kieszak SM, Flanders WD, Kosinski AS, et al. A comparison of the Charlson comorbidity index derived from medical record data and administrative billing data. J Clin Epidemiol (1999) 52:137–42.[CrossRef][Web of Science][Medline]
- Van Doorn C, Bogardus ST, Williams CS, et al. Risk adjustment for older hospitalized persons:a comparison of two methods of data collection for the Charlson index. J Clin Epidemiol (2001) 54:694–701.[CrossRef][Web of Science][Medline]
- Pittet D, Francioli P, Von Overbeck J, et al. Infection control in Switzerland. Infect Control Hosp Epidemiol (1995) 16:49–56.[Web of Science][Medline]
- Sax H, Pittet D. Inter-hospital differences in nosocomial infection rates: the importance of case mix adjustment. Arch Intern Med (2002) 162:2437–42. the Swiss-Noso-Network.
[Abstract/Free Full Text] - Garner JS, Jarvis WR, Emory TG, et al. Definitions for nosocomial infections. Am J Infect Contr (1988) 16:128–40.[CrossRef]
- Pittet D, Harbarth S, Ruef C, et al. Prevalence and risk factors for nosocomial infections in four university hospitals in Switzerland. Infect Control Epidemiol (1999) 20:37–42.[CrossRef]
- Cicchetti DV, Feinstein VAR. High agreement II: resolving the paradoxes. J Clin Epidemiol (1990) 43:551–8.[CrossRef][Web of Science][Medline]
- Kleinbaum DG, Klein M. Logistic Regression: A Self Learning Text (2000) 2nd edn. New York: Springer-Verlag.
- Harrell FE Jr. Regression Modeling Strategies: With Application to Linear Models, Logistic Regression, and Survival Analysis (2001) New York: Springer-Verlag, Springer. Series in Statistics.
- Hanley JA, McNiff KJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Diagn Radiol (1982) 143:29–36.
- Melfi C. Selecting a patient characteristics index for the prediction of medical outcomes using administrative claims data. J Clin Epidemiol (1995) 48:917–26.[CrossRef][Web of Science][Medline]
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics (1977) 33:159–74.[CrossRef][Web of Science][Medline]
- Malenka DJ, McLerran D, Roos N, et al. Using administrative data to describe case-mix: a comparison with medical record. J Clin Epidemiol (1994) 47:1027–32.[CrossRef][Web of Science][Medline]
- Concato J, Horwitz RI, Freinstein AR, et al. Problems of comorbidity in mortality after prostactomy. JAMA (1992) 267:1077–82.
[Abstract/Free Full Text] - Institut universitaire de médecine sociale et préventive. 27 March 2006. Lausanne, Suisse: http:\\www.ovs.ch/ Validation du codage, des diagnostics et des interventions dans les établissements hospitalier valaisans. Données 2003.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  K. E. Chan, J. M. Lazarus, R. Thadhani, and R. M. Hakim Warfarin Use Associates with Increased Risk for Stroke in Hemodialysis Patients with Atrial Fibrillation J. Am. Soc. Nephrol., October 1, 2009; 20(10): 2223 - 2233. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. E. Chan, J. M. Lazarus, R. Thadhani, and R. M. Hakim Anticoagulant and Antiplatelet Usage Associates with Mortality among Hemodialysis Patients J. Am. Soc. Nephrol., April 1, 2009; 20(4): 872 - 881. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||