Skip Navigation



International Journal for Quality in Health Care Advance Access published online on September 16, 2007
International Journal for Quality in Health Care, doi:10.1093/intqhc/mzm039
This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
19/6/399    most recent
mzm039v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Marang-van de Mheen, P. J.
Right arrow Articles by Kievit, J.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Marang-van de Mheen, P. J.
Right arrow Articles by Kievit, J.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?

© The Author 2007. Published by Oxford University Press on behalf of International Society for Quality in Health Care; all rights reserved

Effects of study methodology on adverse outcome occurrence and mortality

Perla J. Marang-van de Mheen1, Evert-jan F. Hollander2 and Job Kievit1,2

1 Department of Medical Decision Making, Leiden University Medical Center, The Netherlands
2 Department of Surgery, Leiden University Medical Center, The Netherlands

Address reprint request to Perla J. Marang-van de Mheen, Department of Medical Decision Making, Leiden University Medical Center, The Netherlands. Tel: +31-71-5264574; Fax: +31-71-5266838; E-mail: p.j.marang{at}lumc.nl


   Abstract
Top
 Abstract
Methods
 Results
 Discussion
 Competing interest
 Acknowledgement
 References
 
Purpose. To assess the impact of variables related to setting, study design and definition on adverse outcome occurrence and mortality in hospitalized patients.

Data sources. Pubmed and Embase.

Study selection. Articles in English from 1980 onwards, in non-selected patients or surgical patients only.

Data extraction. Data were extracted independently by two authors using a predefined form. Included study methodology variables were general variables such as setting, patient variables like age, study variables like number of reviewers and definition and the resulting adverse outcome occurrence and mortality.

Results. Eleven studies reporting on 76 617 non-selected patients and 18 studies representing 136 292 surgical patients were included. Adverse outcomes were estimated to occur in 16% (12–19%) of non-selected patients and in 18% (14–22%) of surgical patients, taking into account the heterogeneity between studies. The study methodology variables were not statistically significant in explaining variation in adverse outcome occurrence, but the individual studies, when used as a random effect variable, were significant. Conversely, the study variable was not significant to explain variation in mortality rates, whereas the study methodology variables were: having more than one reviewer increased mortality by 30–80%, older study populations resulted in higher mortality and including a cause in the definition halved the mortality rate.

Conclusion. Study methodology variables do not explain differences in adverse outcome occurrence between studies. Other inter-study differences are more important. However, study methodology is an important predictor for mortality differences and should be taken into account when interpreting differences between studies.

Keywords: adverse outcome occurrence, mortality, study methodology, systematic review

The occurrence of adverse outcomes may vary between studies or over time. Part of this variation may be true, but part may also be explained by other factors such as differences in study methodology. Veen et al. recently showed that within the same surgical context, a change in the definition increased the occurrence of adverse outcomes from 7 (for adverse outcomes related to surgery only) to 27% (for all adverse outcomes, including outcomes related to underlying disease or comorbidity) [1].

Ever since the Harvard Medical Practice study [2], many studies have followed the general design and definition used in this study. Previous research has shown that methodological differences explain part of the differences in outcomes between these studies, e.g. between the Utah/Colorado and Australian study [3], but that much of the difference remained. However, all of these studies used retrospective record review as the method to identify adverse outcomes, which is not the case in e.g. the Netherlands, where adverse outcomes are usually identified as part of routine medical practice [4, 5]. Previous research has shown that this method of routine reporting is as good as record review to identify serious adverse outcomes, but slightly underestimates minor adverse outcomes [6]. A further difference is the definition of an adverse outcome which is more sensitive than the definition used in other studies in the sense that outcomes related to the natural history of the disease or comorbidity are also included [6, 7]. The impact of these differences with respect to study methodology variables on the adverse outcome occurrence is not clear, which is needed if one wants to compare the outcomes of such studies with that of previous studies.

We, therefore, carried out a systematic review of studies reporting adverse outcomes, both for non-selected in-hospital patients and surgical patients only. The purpose of this systematic review is to assess the impact of variables related to setting, study design and definition on reported adverse outcome occurrence and mortality.


   Methods
Top
 Abstract
 Methods
Results
 Discussion
 Competing interest
 Acknowledgement
 References
 
Study selection
Pubmed and Embase were searched to retrieve articles in English on adverse outcome occurrence from 1980 onwards, concerning either non-selected in-hospital patients, or surgical patients only. The following search string was used:

(‘Iatrogenic Disease/epidemiology’[MAJR] OR ‘Iatrogenic Disease/prevention and control’[MAJR] OR ‘Postoperative Complications/epidemiology’[MAJR] OR ‘Medical Errors/statistics and numerical data’[MAJR])

AND

(complication[title word] OR complications[title word] OR (adverse[title] AND (event[title word] OR events[title word])) OR error[title word] OR errors[title word] OR iatrogenic[title word])

AND

(learning [title word] OR registries OR medical records OR registry OR reporting OR measurement) OR ((complication[title word] OR complications[title word] OR adverse[title word]) AND (registries[title word] OR registry[title word] OR reporting [title word] OR measurement[title word] OR records[title] OR record[title])).

The search resulted (by November 2005) in 968 Pubmed and 776 Embase titles, constituting a total of 1545 titles. Two independent reviewers (PJMvdM and EJFH) screened the titles and read relevant abstracts to decide if the full-text articles should be obtained. Full-text articles (n = 74) were examined and selected based on the following criteria:

  • Reporting on, or providing data that allow the calculation of, occurrence of adverse outcomes.
  • Sufficient specification of what an ‘adverse outcome’ is.
  • Sufficient specification of patient population (and, if relevant, subgroups) with respect to selection, characteristics, and health care context.
Literature references were checked to minimize the risk of missing relevant articles. For duplicate papers reporting on the same study, we selected the article that reported the most complete and detailed data. This resulted in a total of 24 studies, eligible for further analysis [1, 2, 8–29].

Data extraction
Data were extracted independently by PJMvdM and EJFH by means of a predefined form. Categories in this form were

  • General variables: year of index admission, country, number of hospitals and setting (acute care, intensive care or non-selected hospitals),
  • Patient variables: average age, proportion of men,
  • Study variables: data collection method (retrospective medical record review or prospective during admission), unit of analysis (admissions, patients, or operations), number of reviewers and definition of adverse outcome (cause of adverse outcome included in definition, requiring the outcome to be causally related to the care provided—e.g. injury caused by medical management rather than by the underlying disease)
  • Study results: total number of admissions/patients/operations and number with and without adverse outcomes. Number of adverse outcomes resulting in death and number of preventable adverse outcomes were, if possible, also extracted.

Statistical analysis
Studies on adverse outcomes in surgical patients were analysed separately because these studies were far more heterogeneous in study design than studies in unselected patients, which generally followed the design of the well-known Harvard Medical Practice Study [2]. Since the number of patients varied considerably between studies, in all analyses, all estimates were weighted by the number of patients per study. We studied the following variables with respect to their association with the occurrence of adverse outcomes:

  • Setting (non-selected hospitals versus acute care/intensive care only).
  • Average age of patients in study.
  • Data collection method (prospective during admission versus retrospective medical record review).
  • Unit of analysis (patient versus admission, and operation versus admission).
  • Number of reviewers (more than one reviewer versus one reviewer).
  • Definition including cause of adverse outcome (yes versus no).
For surgical patients, most papers did not give enough details with respect to setting so that we excluded this variable from the analyses in surgical patients. Average age was not reported in one (small) study of unselected patients, and for this study we imputed the average age of the unselected patients in the other studies. For the studies in surgical patients, average age was not reported for a great number of (large) studies, so that we excluded this variable from the analyses in these patients. The proportion of men was not reported in a great number of studies in both surgical and unselected patients and was therefore excluded from the analyses.

We first tested for heterogeneity between studies, separately for studies reporting on non-selected and surgical patients, and both for adverse outcome occurrence and mortality [30]. If heterogeneity was confirmed, 95% random effect confidence intervals were reported on the proportions using the method of DerSimonian and Laird [30]. Differences in proportions between groups of studies, e.g. studies using retrospective or prospective data collection were tested using a z-test. Secondly, a multivariable model was constructed using logistic regression, in which the above variables were included to assess the independent effect of each variable adjusted for differences in each of the others. This was done for all adverse outcomes, and then for mortality only. Since there may be other unknown factors responsible for part of the variation in adverse outcomes between studies, we included a random effect for study in the logistic regression analysis.


   Results
Top
 Abstract
 Methods
 Results
Discussion
 Competing interest
 Acknowledgement
 References
 
Study characteristics and adverse outcomes
Eleven studies reported on non-selected patients, representing 76 617 patient admissions or patients. Eighteen studies reported on adverse outcomes in surgical patients, representing 16 180 patients, 12 127 patient admissions and 107 985 operations (total: 136292). The characteristics of the included studies are summarized in Tables 1 and 2, respectively. The studies greatly differ in sample size and the unit of analysis being used, from 502 non-selected patients in a single Canadian hospital [29] to 30 121 non-selected patient admissions in 51 US hospitals [2]. Similar differences between study-populations were found in surgical patients (from 192 in the Canadian study from Wanzel et al. [24] to 44 603 operations in the US study from McGuire et al. [18]). In addition, all studies differ in their methods of data collection, in the number of reviewers who have identified the adverse outcomes, and in setting from acute care hospitals and/or intensive care units to non-selected hospitals.


View this table:
[in this window]
[in a new window]

  		Table 1. Studies reporting occurrence of adverse outcomes in non-selected patients (1980–present)

 


View this table:
[in this window]
[in a new window]

  		Table 2. Studies reporting occurrence of adverse outcomes in surgical patients (1980–present)

 
The occurrence of adverse outcomes varied considerably: between 2.9 [12] and 45.8% [9] for non-selected patients in non-selected and ICU setting, respectively (average: 8.4%), and between 1.4 [16] and 39.1% [24] for surgical patients (average: 12.5%). Less variation was found in mortality: between 0.2 [12] and 1.8% [8] for non-selected patients and between 0.4 [16] and 6.1% [25] for surgical patients. Estimates with respect to the percentage of preventable adverse outcomes varied between 27.6 [2] and 51.2% [9] in non-selected patients and between 18.1 [24] and 61.6% [22] for surgical patients.

Effects of study methodology on adverse outcome occurrence and mortality
Heterogeneity in adverse outcome occurrence between studies was found both for non-selected patients (p < 0.001) and for surgical patients (p < 0.001), so that estimates were calculated with random-effect confidence intervals (to take this heterogeneity into account). Overall, adverse outcomes were estimated to occur in 15.5% (12.1–19.0%) of non-selected patients and in 17.6% (13.6–21.6%) of surgical patients. Heterogeneity between studies was also confirmed for mortality (p < 0.001 for both non-selected and surgical patients). Taking this heterogeneity into account, the overall mortality was estimated to be 0.7% (0.5–0.9%) in non-selected patients and 1.5% (1.0–1.9%) in surgical patients.

Higher adverse outcome occurrences in non-selected patients were found in studies using prospective data collection, with patients (rather than admissions) as the unit of analysis, and in which adverse outcomes were counted regardless of cause (Table 3). The results in surgical patients were in the same direction but not statistically significant (Table 3). Using the patient as the unit of analysis is likely to increase the chance of finding an adverse outcome with this patient, thereby resulting in higher adverse outcome occurrences, particularly if this patient is admitted more than once. For instance, if patients are admitted three times in 1 year with an adverse outcome occurring during one admission, this would result in an average of one adverse outcome per patient and of one adverse outcome per three admissions. Furthermore, if adverse outcomes are counted only if caused by medical management rather than the underlying disease, this will decrease the number of events that comply with this definition. The results for mortality were similar (Table 4).


View this table:
[in this window]
[in a new window]

  		Table 3. Reported adverse outcome occurrence by study methodology variables: non-selected and surgical patients

 


View this table:
[in this window]
[in a new window]

  		Table 4. Reported mortality by study methodology variables: non-selected and surgical patients

 
The multivariable logistic regression analysis showed that the independent effect of these variables on the reported adverse outcome occurrence was not statistically significant (Table 5). The effect of data collection was excluded from the model in surgical patients since it became a constant after adjustment for the other variables. Only study as a random effect variable, to account for other unknown factors responsible for part of the variation in outcomes between studies, had a statistically significant effect on the adverse outcome occurrence both in non-selected patients and surgical patients (Table 5). This suggests that other inter-study variability is more important than the differences in study methodology for the reported adverse outcome occurrence.


View this table:
[in this window]
[in a new window]

  		Table 5. Impact of study methodology on reported adverse outcome occurrence: random effect model

 
In contrast with these findings, the study methodology variables had a statistically significant effect on mortality, whereas the study variable as a random effect had no effect on reported mortality (Table 6). Having more than one reviewer increased the adverse outcome mortality by 30–80%, older study populations resulted in higher mortality, and including a cause in the definition halved the adverse outcome mortality. These differences in study methodology explain part of the variation between studies and should be taken into account when interpreting different mortality outcomes between studies.


View this table:
[in this window]
[in a new window]

  		Table 6. Impact of study methodology on reported mortality: random effect model

 

   Discussion
Top
 Abstract
 Methods
 Results
 Discussion
Competing interest
 Acknowledgement
 References
 
The results from the present study suggest that inter-study differences other than study methodology are more important to explain the variation in adverse outcome occurrence. This might mean that differences in the interpretation of definitions could be more important than the difference in definition itself. Runciman et al., for instance, demonstrated that many differences between the Quality of Australian Health Care Study and the Utah/Colorado Study are caused by lower thresholds for reporting adverse outcomes, i.e. by the inclusion of less serious events in the Australian study [31]. The Australian definition was thus more sensitive in detecting less serious events, although their definition seems identical. Furthermore, Thomas et al. [3] showed that the Quality of Australian Health Care Study still differed in many ways from the Utah/Colorado Study, although the same definition and similar research methods were used. Adjusting for these differences resulted in a smaller difference between the studies, although most of the difference remained (10.6% versus 3.2%, respectively). Apparently, other factors that differ between studies, e.g. the extent of comorbidity of patients or other patient-related variables, are far more important in explaining adverse outcome differences between studies than study methodology variables.

With respect to mortality, however, study methodology explains part of the variation between studies and is more important than other inter-study differences. This difference between adverse outcome occurrence and mortality makes sense given the fact that interpretation differences will occur more with respect to adverse outcome occurrence than with mortality. Specifically, we found that the number of reviewers was a significant predictor of adverse outcome mortality, consistent with previous research showing poor to moderate reviewer reliability [32, 33]. Furthermore, including a causal relationship in the definition of adverse outcomes halved mortality, which will explain part of the difference in outcomes between studies carried out in the Netherlands and the US studies. This is similar to the effect found within a Dutch study, where the exclusion of this causal relationship resulted in a considerable increase of adverse outcomes [1].

These results have important implications for studies using a different design and/or different definition than the US or Australian studies. When interpreting the outcomes of these studies, one should be aware of the effects of study methodology as estimated in this study, in particular for mortality. If not, then it will only lead to false positive observations of differences, which are caused by differences in methodology only. Only then will the study of adverse outcome differences do what it was initiated for in the first place; contribute to the improvement of quality of care.


   Competing interest
Top
 Abstract
 Methods
 Results
 Discussion
 Competing interest
Acknowledgement
 References
 
None


   Acknowledgement
Top
 Abstract
 Methods
 Results
 Discussion
 Competing interest
 Acknowledgement
References
 
The authors thank Dr S. Le Cessie for statistical advice on the random effect analyses.


   References
Top
 Abstract
 Methods
 Results
 Discussion
 Competing interest
 Acknowledgement
 References
 

  1. Veen EJ, Janssen-Heijnen MLG, Leenen LPH, et al. The registration of complications in surgery: A Learning Curve. World J Surg (2005) 29:402–9.[CrossRef][Web of Science][Medline]

  2. Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I N Engl J Med (1991) 324:370–6.

  3. Thomas EJ, Studdert DM, Runciman WB, et al. A comparison of iatrogenic injury studies in Australia and the USA. I: Context, methods, casemix, population, patient and hospital characteristics. Int J Qual Health Care (2000) 12:371–8.[Abstract/Free Full Text]

  4. Marang-van de Mheen PJ, Kievit J. Automated registration of adverse events in surgical patients in the Netherlands: the current status (in Dutch). Ned Tijdschr Geneeskd (2003) 147:1273–7.[Medline]

  5. Marang-van de Mheen PJ, Stadlander MC, Kievit J. Adverse outcomes in surgical patients: implementation of a nationwide reporting system. Qual Saf Health Care (2006) 15:320–4.[Abstract/Free Full Text]

  6. Marang-van de Mheen PJ, van Hanegem N, Kievit J. Effectiveness of routine reporting to identify minor and serious adverse outcomes in surgical patients. Qual Saf Health Care (2005) 14:378–82.[Abstract/Free Full Text]

  7. Kievit J, Jeekel J, Sanders FBM. Adverse outcome registration and quality improvement (in Dutch). Med Contact (1999) 54:1363–5.

  8. Steel K, Gertman PM, Crescenzi C, et al. Iatrogenic illness on a general medical service at a university hospital. N Engl J Med (1981) 304:638–42.[Abstract]

  9. Andrews LB, Stocking C, Krizek T, et al. An alternative strategy for studying adverse events in medical care. Lancet (1997) 349:309–13.[CrossRef][Web of Science][Medline]

  10. Bates DW, O'Neil AC, Petersen LA, et al. Evaluation of screening criteria for adverse outcomes in medical patients. Med Care (1995) 33:452–62.[CrossRef][Web of Science][Medline]

  11. Wilson RM, Runciman WB, Gibberd RW, et al. The quality in Australian health care study. Med J Aust (1995) 163:458–71.[Web of Science][Medline]

  12. Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care (2000) 38:261–71.[CrossRef][Web of Science][Medline]

  13. Vincent C, Neale G, Woloshynowych M. Adverse events in British hospitals: preliminary retrospective record review. BMJ (2001) 322:517–9.[Abstract/Free Full Text]

  14. Davis P, Lay-Yee R, Briant R, et al. Adverse events in New Zealand public hospitals I: occurrence and impact. N Z Med J (2002) 115.

  15. Baker GR, Norton PG, Flintoft V, et al. The Canadian Adverse events Study: The incidence of adverse events among hospital patients in Canada. Can Med Assoc J (2004) 170:1678–86.[Abstract/Free Full Text]

  16. Couch NP, Tilney NL, Rayner AA, et al. The high cost of low-frequency events. N Engl J Med (1981) 304:634–7.[Abstract]

  17. Adar R, Bass A, Walden R. Iatrogenic Complications in Surgery. Ann Surg (1982) 196:725–9.[Web of Science][Medline]

  18. McGuire HH jr, Horsley S III, Salter RD, et al. Measuring and managing quality of surgery. Arch Surg (1992) 127:733–8.[Abstract/Free Full Text]

  19. Roukema JA, van der Werken C, Leenen LPH. Registration of postoperative complications to improve surgical results (in Dutch). Ned Tijdschr Geneeskd (1996) 140:781–4.[Medline]

  20. Kable AK, Gibberd RW, Spigelman AD. Adverse events in surgical patients in Australia. Int J Qual Health Care (2002) 14:269–76.[Abstract/Free Full Text]

  21. Gawande AA, Thomas EJ, Zinner MJ, et al. The incidence and nature of surgical adverse events in Colorado and Utah in 1992. Surgery (1999) 126:66–75.[CrossRef][Web of Science][Medline]

  22. Veen MR, Lardenoye HP, Kastelein GW, et al. Recording and Classification of Compliations in a Surgical Practice. Eur J Surg (1999) 165:421–4.[CrossRef][Web of Science][Medline]

  23. Feldmann L, Barkun J, Barkun A, et al. Measuring postoperative complications in general surgery patients using an events-based strategy: comparison with complications presented at morbidity and mortality rounds. Surgery (1997) 122:711–20.[CrossRef][Web of Science][Medline]

  24. Wanzel KR, Jamieson CG, Bohnen JM. Complications on a general surgery service: incidence and reporting. Can J Surg (2000) 43:113–7.[Web of Science][Medline]

  25. Maina P, Carstensen M, Tonnesen H. Recording of postoperative complications: quantity and quality. Eur J Surg (2002) 168:736–40.[CrossRef][Web of Science][Medline]

  26. Veltkamp SC, Kemmeren JM, van der Graaf Y, et al. Prediction of serious complications in patients admitted to a surgical ward. Br J Surg (2002) 89:94–102.[CrossRef][Web of Science][Medline]

  27. Healey MA, Shackford SR, Osler TM, et al. Complications in surgical patients. Arch Surg (2002) 137:611–8.[Abstract/Free Full Text]

  28. Michel P, Quenon JL, de Sarasqueta AM, et al. Comparison of three methods for estimating rates of adverse events and rates of preventable adverse events in acute care hospitals. BMJ (2004) 328:199–203.[Abstract/Free Full Text]

  29. Forster AJ, Asmis TR, Clark HD, et al. Ottawa hospital patient safety study: incidence and timing of adverse events in patients admitted to a Canadian teaching hospital. Can Med Assoc J (2004) 170:1235–40.[Abstract/Free Full Text]

  30. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials (1986) 7:177–88.[CrossRef][Web of Science][Medline]

  31. Runciman WB, Webb RK, Helps SC, et al. A comparison of iatrogenic injury studies in Australia and the USA. II: Reviewer behaviour and quality of care. Int J Qual Health Care (2000) 12:379–88.[Abstract/Free Full Text]

  32. Thomas EJ, Lipsitz SR, Studdert DM, et al. The reliability of medical record review for estimating adverse event rates. Ann Intern Med (2002) 136:812–6.[Abstract/Free Full Text]

  33. Hayward RA, Hofer TP. Estimating hospital deaths due to medical error. Preventability is in the eye of the reviewer. JAMA (2001) 286:415–20.[Abstract/Free Full Text]

Accepted for publication August 8, 2007.


Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us    What's this?


This article has been cited by other articles:


Home page
 Int J Qual Health CareHome page
W. Mendes, M. Martins, S. Rozenfeld, and C. Travassos
The assessment of adverse events in hospitals in Brazil
Int. J. Qual. Health Care, August 1, 2009; 21(4): 279 - 284.
[Abstract] [Full Text] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
19/6/399    most recent
mzm039v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Marang-van de Mheen, P. J.
Right arrow Articles by Kievit, J.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Marang-van de Mheen, P. J.
Right arrow Articles by Kievit, J.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?