OUP user menu

Administrative data outperformed single-day chart review for comorbidity measure

Jean-Christophe Luthi, Nicolas Troillet, Marie-Christine Eisenring, Hugo Sax, Bernard Burnand, Hude Quan, William Ghali
DOI: http://dx.doi.org/10.1093/intqhc/mzm017 225-231 First published online: 28 June 2007

Abstract

Objective The purpose of this article is to compare the Charlson comorbidity index derived from a rapid single-day chart review with the same index derived from administrative data to determine how well each predicted inpatient mortality and nosocomial infection.

Design Cross-sectional study.

Setting The study was conducted in the context of the Swiss Nosocomial Infection Prevalence (SNIP) study in six hospitals, canton of Valais, Switzerland, in 2002 and 2003.

Participants We included 890 adult patients hospitalized from acute care wards.

Main outcome measures The Charlson comorbidity index was recorded during one single-day for the SNIP study, and from administrative data (International Classification of Disease, 10th revision codes). Outcomes of interest were hospital mortality and nosocomial infection.

Results Out of 17 comorbidities from the Charlson index, 11 had higher prevalence in administrative data, 4 a lower and two a similar compared with the single-day chart review. Kappa values between both databases ranged from − 0.001 to 0.56. Using logistic regression to predict hospital outcomes, Charlson index derived from administrative data provided a higher C statistic compared with single-day chart review for hospital mortality (C = 0.863 and C = 0.795, respectively) and for nosocomial infection (C = 0.645 and C = 0.614, respectively).

Conclusions The Charlson index derived from administrative data was superior to the index derived from rapid single-day chart review. We suggest therefore using administrative data, instead of single-day chart review, when assessing comorbidities in the context of the evaluation of nosocomial infections.

Keywords
  • administrative data
  • comorbidities
  • death in hospital
  • nosocomial infection

Many health services researchers are measuring outcomes such as mortality or nosocomial infections. Among the different indexes available to control for comorbid conditions, the most frequently used is the Charlson index [1], which is a weighted score of 17 comorbidities that was initially used to predict in-hospital and 1-year mortality. It was then adapted for use with administrative data and in particular with the International Classification of Diseases, 9th revision, Clinical Modifications (ICD-9-CM) by different groups [25]. In the last decade, the International Classification of Disease, 10th revision (ICD-10), was introduced by the World Health Organization in many European countries, in Australia and in Canada [6]. Responding to this coding transition, Halfon et al. [7] in Switzerland and Sundararajan et al. [8] in Australia developed ICD-10 coding algorithms for the Charlson index. Subsequently, a new and more comprehensive ICD-10 coding algorithm for the Charlson index was developed by collaboration between three international research groups in Switzerland, Australia and Canada [9]. We assess this new and comprehensive algorithm in the present study.

Different approaches exist for measuring the severity of illness and predicting outcomes. Administrative data have been widely used by health service researchers in studies of outcomes, effectiveness, appropriateness and utilization of health care services [1013], often with the use of the coding algorithms described above for defining comorbidities [25, 9]. A second traditional approach is to conduct a detailed chart review after discharge. This methodology is often used to assess outcomes and processes of care after hospital stay and, in some instances, has been used by researchers to evaluate the validity of administrative data [1417]. A third approach occasionally used is to conduct a rapid single-day chart review during hospital stay. Such an approach has appeal because it can be done more quickly and at lower cost compared with a complete chart review. A second advantage is that it permits real time implementation and produces data while the patient is still in the hospital. These potential advantages lead to the use of the single-day chart review method for the measurement of comorbidity in the context of the Swiss Nosocomial Infection Prevalence (SNIP) study [18]. A single-day review conducted during patient hospitalization produces information on comorbidities that then permit appropriate risk-adjustment for comparing nosocomial infection rates across hospitals. In this study, we used the Charlson index derived from the SNIP study, since this was already available, and compared these results with the Charlson index derived from administrative data.

Of some concern, the performance of such single-day comorbidity assessments from chart reviews has, to our knowledge, never been formally assessed. Recognizing this, we conducted a formal study to assess the performance of this rapid assessment method for deriving the Charlson comorbidity index in comparison with administrative data. Our second objective is to compare the prognostic value of Charlson comorbidity index measures derived using the single-day chart review vs. administrative data for predicting the outcomes of in-hospital mortality and nosocomial infections.

Methods

Setting

The SNIP study [18], a cross-sectional 1-week period prevalence study was conducted simultaneously in six acute care hospitals of the canton of Valais, Switzerland in 2002 and 2003, respectively, from May 27 to June 6 and from May 19 to 28. The canton of Valais is located in the middle of the Alps in the western part of Switzerland. These hospitals have between 100 and 240 beds and were participating voluntarily in the Swiss-Noso Network.

Data collection (SNIP study)

All adult patients hospitalized on internal medicine, surgery, orthopaedics and gynaecology wards and those in intensive care units were included in the SNIP study. Data collection took place during any single day while patients were hospitalized. Data abstracters were infection control practitioners who attended at least three specific 1-day training sessions. A standardized documentation was provided for each data abstractor by the Swiss-Noso team. This included the study protocol, standardized case report form, written definitions of all variables and a code list.

Data collection included demographic characteristics, information on the type of nosocomial infection, type of pathogens, severity of infections, risk factors, management and treatment. Details have been reported in a previous publication [19]. The modified Center for Disease Control and Prevention criteria were used to define nosocomial infections [2021], which were recorded if present during the 7 days before the day of investigation. Data collected during the SNIP study included also the presence of all comorbidities from the Charlson index [17], which is a weighted average of selected comorbidities. While recording individual comorbidities that constitute the Charlson index, research nurses were instructed to report comorbidities retrospectively at the time of admission and specifically not to include the main cause of admission as comorbidities.

Administrative data

In the canton of Valais, Switzerland, routinely collected administrative data since the year 2000 are transmitted to the Health Observatory, by the six acute care community hospitals. The medical part of administrative data includes patients' demographic characteristics, discharge status including death, up to 10 diagnostics coded (ICD-10) and up to 10 procedures coded (International Classification of Disease-9-CM).

The Charlson comorbidity index was also calculated using administrative data and derived using the ICD-10 coding algorithm recently published by Quan et al. [9]. Information on hospital mortality was determined from administrative data as well.

Since administrative data are completely anonymous in Switzerland, we needed to merge both data sets (SNIP study and administrative data) using three variables: date of birth, date of admission and sex. We identified 1054 eligible patients (481 in 2002 and 573 in 2003) with the SNIP study. Then, after matching, 890 patients were available for the analysis (430 in 2002 and 460 in 2003). Thus, we had data from two sources for the same hospital admission.

Statistical analysis

First, descriptive statistics were employed to calculate the prevalence of individual comorbidities for data collected by infection control practitioners and administrative data. Agreement between the two datasets was measured using kappa values with corresponding 95% confidence intervals (95% CI) [22].

We then developed two logistic regression models for in-hospital mortality and two for nosocomial infection as outcome variables to assess model performance. For each outcome, one model used the Charlson comorbidity index computed from the SNIP study's single-day chart review and the second model used the Charlson index obtained from administrative data. We used the SAS procedure ‘proc logistic’ with the class option and effect coding for the Charlson index in each model. Age and sex were used as control variables. Odds ratios [23], 95% CIs, C statistics and Brier scores [24] were computed for each model in order to assess which model performed best in predicting the in-hospital mortality and nosocomial infection. The C statistic is the area under the receiver operating characteristic (ROC) curve, measuring the ability of the predictive model to discriminate among those who do or do not die at the hospital or those who do have a nosocomial infection and those who do not [2526]. All analyses were conducted using SAS, version 8.02 (SAS institute inc., Cary, NC, USA).

Results

Prevalence of comorbidities

Among 890 patients available for the analysis, the mean age (standard deviation) was 62.2 years (19.0 years), and 53.3% were female. Table 1 presents the prevalence of the 17 comorbidities from the Charlson index in administrative data and from the SNIP study. Eleven comorbidities were more frequently observed in administrative data than chart review data. Four were more frequently seen in the single-day chart review (i.e. diabetes with chronic complication, myocardial infarction, peptic ulcer disease and rheumatologic disease). The frequency of AIDS and of moderate or severe liver disease, meanwhile, was similar in both databases.

View this table:
Table 1

Measurement of agreement between administrative and single-day chart review data (SNIP study)a, N = 890

ConditionPrevalence administrative data N (%)Prevalence single-day chart review data (SNIP study)aN (%)Agreement between data sourcesKappa (95% CI)
Admin (Yes)Admin (No)
Single day (Yes) NSingle day (No) NSingle day (Yes) NSingle day (No) N
AIDS1 (0.1)1 (0.1)011888−0.001 (−0.003–0.00)
Cerebrovascular disease56 (6.3)48 (5.4)1343357990.20 (0.09–0.32)
Chronic pulmonary disease78 (8.8)51 (5.7)3543167960.51 (0.40–0.62)
Congestive heart failure84 (9.4)52 (5.8)2559277790.32 (0.21–0.43)
Dementia26 (2.9)21 (2.4)131388560.54 (0.37–0.72)
Diabetes83 (9.3)61 (6.9)4340187890.56 (0.46–0.66)
Diabetes with chronic complication30 (3.4)50 (5.6)219298310.50 (0.37–0.64)
Hemiplegia or paraplegia16 (1.8)4 (0.5)11538710.09 (−0.09–0.27)
Malignancy120 (13.5)72 (8.1)2595477230.18 (0.09–0.26)
Metastatic Solid tumour55 (6.2)30 (3.4)213498260.47 (0.34–0.60)
Mild liver disease38 (4.3)7 (0.8)53328500.21 (0.05–0.37)
Moderate or severe liver disease12 (1.4)13 (1.5)39108680.23 (0.01–0.45)
Myocardial infarction33 (3.7)60 (6.7)1617448130.31 (0.18–0.44)
Peptic ulcer disease11 (1.2)58 (6.5)38558240.07 (−0.02–0.16)
Peripheral vascular disease63 (7.1)43 (4.8)2241218060.38 (0.26–0.50)
Renal disease48 (5.4)18 (2.0)153338390.44 (0.29–0.59)
Rheumatologic disease9 (1.0)14 (1.6)36118700.25 (0.01–0.49)
  • aSNIP study, Swiss Nosocomial Infection Prevalence Study 2002 and 2003.

Agreement between data sources

The kappa value assessing the agreement between the 17 comorbidities of the Charlson index in the SNIP study compared with administrative data is represented in Table 1. For the purposes of interpretation, kappa values can be categorized into five groups: < 0.2 (poor agreement), 0.21–0.40 (fair agreement), 0.41–0.60 (moderate agreement), 0.61–0.80 (substantial agreement) and 0.81–1.00 (near perfect agreement) according to Landis and Koch [27]. Using this categorization, we found four comorbidities with poor agreement, seven with fair agreement and six with moderate agreement.

The agreement of the Charlson index score between both databases is shown in Table 2. The numbers in bold represent perfect agreements between the two data sources. The overall kappa value was 0.30 (95% CI: 0.26–0.34), and the weighted kappa value 0.44 (95% CI: 0.39–0.49).

View this table:
Table 2

Agreement of Charlson index scoresa between administrative and single-day chart review data (SNIP study)b, N = 883c

Administrative dataSingle-day chart review (SNIP study)
0123456 and overTotal
034260168305434
16660219520163
236312515115114
31616201122370
42610243229
5012521112
6 and over15773512461
Totalc4771811015322940883d,e
  • aCharlson index score = 1 × proportion of myocardial infarction + 1 × congestive heart failure + 1 × peripheral vascular disease + 1 × cerebrovascular disease + 1 × dementia + 1 × chronic pulmonary disease + 1 × rheumatologic disease + 1 × peptic ulcer disease + 1 × diabetes + 1 × mild liver disease + 2 × hemiplegia/paraplegia + 2 × renal disease + 2 × diabetes with chronic complication + 2 × malignancy + 3 × moderate or severe liver disease + 6 × metastatic solid tumour + 6 × AIDS.

  • bSNIP study, Swiss Nosocomial Infection Prevalence Study 2002 and 2003.

  • cSeven cases missing since a Charlson score was not provided during single-day chart review for these patients.

  • dKappa: 0.30 (95% CI: 0.26–0.34).

  • eWeighted kappa: 0.44 (95% CI: 0.39–0.49).

Predicting outcomes

Of the sample of 890 patients, 26 died in the hospital and 51 had a nosocomial infection. Table 3 presents the detailed number of persons who died or who had a nosocomial infection for each Charlson score calculated based on administrative data or assessed by the SNIP study single-day chart review. When the Charlson index score increases, the proportion of patients dying in hospital or getting nosocomial infections increases, in both databases, but more notably with administrative data.

View this table:
Table 3

Hospital mortality and nosocomial infection according to Charlson index scores, N = 890

MeasureCharlson scoresAdministrative data N = 890Single-day chart review data (SNIP study)aN = 883b
Percentage with event (death or infection)N event/N totalPercentage with event (death or infection)N event/N total
Hospital mortality00.231/4321.075/467
13.806/1582.925/171
21.792/1124.044/99
35.263/573.922/51
49.383/324.551/22
518.182/110.000/8
6 and over14.529/6222.509/40
Total3.0126/8653.0326/858
Nosocomial infection04.1118/4384.1920/477
13.075/1636.0811/181
26.908/1164.955/101
37.815/6411.326/53
48.823/3413.643/22
533.334/120.000/9
6 and over12.908/6215.006/40
Total5.7351/8905.7851/883
  • aSNIP study, Swiss Nosocomial Infection Prevalence Study 2002 and 2003.

  • bSeven cases missing since a Charlson score was not provided during single-day chart review for these patients.

Table 4 presents the results of the four logistic regression models with adjusted odds ratios and measure of discrimination, using hospital mortality and nosocomial infections as outcome variables. The adjusted odds ratio for in-hospital mortality for administrative data was 2.07 (95% CI: 0.88–4.87) for a Charlson score of 1, 0.39 (95% CI: 0.11–1.33) for a Charlson score of 2, 3.02 (95% CI: 1.03–8.82) for a Charlson score of 3 and 33.85 (95% CI: 16.37–69.98) for a Charlson score ≥ 4, compared with Charlson score 0. These odds ratios were higher than that for a chart review data. Further, for in-hospital mortality, the C statistic was also higher for administrative data (C = 0.863) compared with single-day chart review data (C = 0.795). The Brier score was smaller for administrative data compared with single-day chart review data; a finding that also indicates better performance for the administrative data model.

View this table:
Table 4

Adjusted odds ratio for Charlson score and measures of fit using administrative data and SNIP studya for hospital mortality and nosocomial infection, N = 890

MeasureCharlson scoresAdministrative dataSingle-day chart review data (SNIP study)a
Adjusted odds ratiob95% CIModel performance C statistic Brier ScoreAdjusted odds ratio**95% CIModel performance C statistic Brier Score
N = 865N = 858
Hospital mortality0Ref0.863Ref0.795
12.070.88–4.870.02750.560.24–1.290.0282
20.390.11–1.331.030.42–2.54
33.021.03–8.820.710.22–2.37
4 and over33.8516.37–69.9816.258.14–32.47
N = 890N = 883
Nosocomial infection0Ref0.645Ref0.614
10.220.10–0.480.05280.700.39–1.250.0537
21.220.64–2.320.450.21–0.98
31.300.60–2.852.671.27–5.62
4 and over6.463.77–11.053.511.87–6.61
  • aSNIP study, Swiss Nosocomial Infection Prevalence Study 2002 and 2003.

  • bControlling for age, sex and the Charlson comorbidity index with effect coding.

For nosocomial infections, odds ratio ranged from 0.22 to 6.46 and from 0.70 to 3.51, respectively, for administrative data and single-day chart review from the SNIP study when the Charlson score increased from 1 to ≥ 4, compared with Charlson score 0. Odds ratios were therefore close for both databases for this outcome. The areas under the ROC curve were 0.645 for administrative data and 0.614 for the single-day chart review data.

Discussion

We compared administrative data and a rapid single-day chart review data in identifying Charlson comorbidities for the purposes of risk adjustment. Our results demonstrate only poor to fair agreement for the majority of 17 Charlson comorbidities. The administrative data reported more cases and predicted hospital mortality and nosocomial infections better, compared with rapid single-day chart review. In addition, odds ratios and C statistics calculated using both databases were lower for nosocomial infections compared with hospital mortality, suggesting that the Charlson comorbidity index predicted the in-hospital mortality better than it did in nosocomial infections.

There are three possible explanations for these results. The first relates to completeness of clinical information in the chart. Single-day review data are generated based on information documented at the initial stage of hospitalization, but administrative data are generated after discharge. Conditions present at admission might be incompletely documented in the chart at the beginning of hospitalization and the chart gradually becomes ‘richer’ in clinical content, as documents such as the discharge summary, consultation notes, pathology reports, surgical reports or discharge letters are fully compiled. We performed secondary analyses to explore this hypothesis and to asses if our results were robust with respect to the timing of the single-day chart review. Kappa values between both databases were effectively higher for the majority of comorbidities if the review was performed close to discharge compared with early in the hospital stay. A second possible explanation is that in the SNIP study, abstractors were instructed to exclude the main cause of admission in the Charlson score. However, administrative data included conditions present at both stages without such distinction. Most previous publications demonstrate that administrative data generally under report Charlson comorbidities compared with entire chart review and agreement between two databases varies by condition.

A study conducted in Calgary, Canada, reported kappa values across comorbidities ranging from a high of 0.87 to a low of 0.34 [14]. In another study conducted in six general medicine wards at Yale (in New Haven, CT, USA), the kappa value for the Charlson index score was 0.35 and ranged between 0.00 and 0.83 among the different comorbidities [17]. Results of our study are slightly lower in terms of kappa values in comparison with these two North-American studies [14, 17], presumably because those studies assessed agreement between administrative data and full chart review done after discharge, whereas ours assessed agreement with the single-day chart review. In another study conducted among Medicare beneficiaries in the State of Georgia (USA), kappa values ranged from 0.12 to 0.68, with most agreement being fair to moderate, as in our study [16]. In this study, the authors found that the odds ratio for in-hospital mortality was 10.15 for the chart index and 2.06 for the ICD-9 index. Further, the area under the ROC curve was 0.697 for a multivariate model incorporating the chart index and 0.639 for the model using the administrative index; these results are much lower compared with the C statistics obtained in our study for hospital mortality as outcome [16]. Further, our results are also comparable with another published study, using complete chart review compared with administrative data [28].

The rapid single-day chart review has its value for collection of clinical information for certain conditions. We found two comorbidities (i.e. AIDS and moderate or severe liver diseases) that have been detected at the similar level by both methods and four conditions with a higher level of detection by the single-day chart review (i.e. diabetes with chronic complication, myocardial infarction, peptic ulcer diseases and rheumatologic diseases). However, administrative data predicted the hospital mortality and nosocomial infection better than rapid single-day chart review. The main advantage of this rapid single-day chart review might be that it can be implemented in less time with much less resources, when compared with a complete chart review after discharge. Therefore, results might be available more quickly. However, despite the fact that such single-day review is cheaper than complete chart review, we recommend not using this method when administrative data are available, because administrative data represent an even less expensive way to derive comorbidity information, with relatively strong performance in comorbidity measurement and outcome prediction.

One limitation is that the study was implemented in one Swiss canton, and thus might not apply to other cantons or even countries. Another potential caveat in the interpretation of the study results is the well-described limitations of administrative data. Inadequate assessment of comorbidity using administrative data has been documented previously [29]. However, most hospitals are actively working to improve the accuracy of their administrative coding systems. In the canton of Valais, Switzerland, the coding has been professionalized, first in one hospital since 2000 and then in all hospitals since 2003 with the creation of a cantonal Coding Unit, which has standardized and professionalized coding rules and methods. One study assessed the quality of coding in the canton of Valais, showing high quality coding in 2003 and major improvements since the year 2000 [30].

In conclusion, for adult patients from six community hospitals, comorbidities from the Charlson index were more often detected using administrative data compared with a rapid single-day chart review. Further, the Charlson index derived from administrative data was shown to be slightly superior to the index derived from the rapid single-day chart review as a risk predictor and, therefore, as adjustment measure. Therefore, we recommend using administrative data in order to get information on comorbidities rather than using the single-day chart review method that was in use in the SNIP study, assessing nosocomial infections rates.

Acknowledgements

The methodological development of the study was performed in the context of a Swiss National Sciences Foundation supported study (grant 3200B0-103852) that used a different dataset.

References

View Abstract