OUP user menu

Health services accreditation: what is the evidence that the benefits justify the costs?

Virginia Mumford, Kevin Forde, David Greenfield, Reece Hinchcliff, Jeffrey Braithwaite
DOI: http://dx.doi.org/10.1093/intqhc/mzt059 606-620 First published online: 13 August 2013

Abstract

Purpose To identify and analyse research on the use of economic evaluation in health services accreditation.

Data sources Seven online health and economic databases, and key accreditation agency and health department websites were searched between June and December 2011.

Study selection The selection criteria were English language and published empirical research studies on the topic of economic evaluation of health service accreditation. No formal economic evaluation of health services accreditation has been carried out to date. Empirical data on costs and benefits were analysed in 6 and 15 studies, respectively.

Data extraction Meta-analysis was unsuitable due to output variability. Attributes relating to study design, scalability and independence of outcome data were collected. For the benefit studies, we also assessed the strength of claim that accreditation improved patient safety and quality, and sources of potential bias.

Results of data synthesis The incremental costs ranged from 0.2 to 1.7% of total costs averaged over the accreditation cycle. The benefit studies were inconclusive in terms of showing clear evidence that accreditation improves patient safety and quality of care.

Conclusion The lack of formal economic appraisal makes it difficult to evaluate accreditation in comparison to other methods to improve patient safety and quality of care. The lack of a clear relationship between accreditation and the outcomes measured in the benefit studies makes it difficult to design and conduct such appraisals without a more robust and explicit understanding of the costs and benefits involved.

  • certification/accreditation of hospitals
  • external quality assessment
  • economic evaluation (cost effectiveness)
  • general methodology
  • patient safety
  • quality measurement
  • quality management

Introduction

Economic evaluation techniques are increasingly being used to ensure policy implementation is effective in achieving stated aims. For example, an analysis of relevant costs and benefits has been mandatory in the USA since 1981 [1] for regulations with an economic effect of at least US$100 million. Economic evaluation (mainly using cost-effectiveness techniques) has been widely used in health care to assess drugs and medical technologies [2, 3]. For more complex interventions, more societal, cost–benefit analysis (CBA) techniques such as Social Return on Investment [4] can be employed to ensure a more inclusive selection of stakeholders are considered, and a wider variety of benefit outcomes are measured.

One example of a complex intervention is accreditation, a form of external audit against pre-determined standards using a mixture of self-assessment and external surveys [5], which has been widely adopted by acute health services internationally [6, 7]. A literature review by Greenfield and Braithwaite [8] identified a lack of research into the costs of participating in accreditation processes, indicating that economic evaluation, in terms of an assessment and comparison of costs and benefits, may be uncommon. Given the significant investment in accreditation (such as fees paid to accreditation agencies) [9, 10]) as well questions as to whether the investment is effective [11], this review aims to determine whether economic evaluation techniques have been used to inform whether the benefits justify the costs involved [12].

Methods

Search strategies

Against this background, a review of the literature was conceptualized, planned and executed between June and December 2011. We first searched economic and medical databases for studies on the economic evaluation of health services accreditation. The databases we searched included economic databases (National Health Services Economic Evaluation Database (NHS EED), Health Economic Evaluation Database, Evidence for Policy and Practice Information Centre, National Bureau of Economic Research & EconLit) in addition to medical databases (SCOPUS and CINAHL). We used keywords appropriate to each database, for example accreditation and health for economic databases, and accreditation, economic evaluation and/or appraisal for health databases. For example, for the SCOPUS search we used: accreditation AND economic (evaluation OR appraisal), or accreditation AND cost (benefit OR utility). Secondly, we collected grey literature by searching health department websites, and contacting international accreditation agencies and key researchers in the field. Our grey literature search focussed on English speaking OECD countries, due to their history of accreditation. This part of the search process encompassed a wider variety of health service facilities to include aged care and primary care services. Thirdly, we expanded our review through a ‘snowballing’ technique by reviewing references in systematic reviews on health services accreditation [8, 13, 14] and using the internet and search engines (such as Google Scholar) to determine the level of analysis on costs and benefits separately. This, when combined with the results of our primary search, provided separate costs (n = 6) and benefits (n = 15) studies (Fig. 1).

Figure 1

Flow chart of study selection process: economic evaluation of health services accreditation.

Study selection

The selection criteria were English language and published empirical research studies on the topic of economic evaluation of health service accreditation. Although many of the results of the systematic search met the criteria for economic valuation set out by the BMJ [15] and NHS EED [16], these studies either discussed economic appraisal of specific health technologies or other types of accreditation for individual health workers. None of the papers regarding accreditation of health services met the criteria for a full economic evaluation in terms of directly assessing and comparing the costs and benefits involved.

For our extended review of the separate costs and benefits of accreditation, we included analytical studies that defined or quantified costs rather than those papers that qualitatively discussed participants' views on costs. Similarly, to assess the benefits, we included analytical studies that isolated patient safety or quality improvement outcomes as these would be easier to translate into an economic evaluation framework. We did not include primarily descriptive studies as they did not lend themselves to the appraisal framework discussed below, although the input from these studies would be critical in assessing benefits when designing an economic evaluation of health services accreditation.

Data extraction

The outcome variables in the benefit studies were not strictly comparable and included both clinical and financial outcomes. We therefore determined that conventional meta-analysis techniques would not be appropriate in this context, and created an accreditation appraisal framework to compare the different studies. For both the cost and benefit studies, we identified the papers by study design using the National Health and Medical Research Council (NHMRC) study design classifications [17] (Table 1). For scalability, we considered whether the results could be readily translated across a broader network by whether the study comprised of 5% or more of comparable facilities nationally. We also assessed the independence of the data as high (H), medium (M) or low (L). For the cost studies, this depended on whether the paper included an independent verification of the costs or whether the authors discuss methods to validate their costing. For the benefit studies, we assessed whether outcomes included measures external to accreditation (not just compliance with accreditation standards) and are assessed independently.We added two further criteria: (i) strength of claim that accreditation improved patient safety and quality, which we assessed as high (H), medium (M) or low (L) depending on the results from the study and (ii) Source of potential bias, where we reviewed the data collection and analysis techniques for sources of bias that could affect the results of each study.

View this table:
Table 1

NHMRC study design and levels of evidence

Intervention studiesAetiological studies
ISystematic review of Level II studies
IIRandomized controlled trialProspective cohort study
III-1Pseudo-randomized controlled trialAll or none
III-2Comparative study with concurrent controlsRetrospective cohort study
III-3Comparative study with no concurrent controlsCase–control study
IVCase seriesCross-sectional study or case series

Two of the authors (D.G. and R.H.) independently reviewed the selection criteria and assessments used in the framework, and any differences were resolved by discussion. During this review process, we noted that the number of studies with clinical (patient safety and quality) rather than operational (financial) measures changed following the Institute of Medicine (IOM) 2000 report To Err Is Human [18]. This report quoted two large studies in the USA indicating adverse events in 2.9 and 3.7% of hospitalizations, between 6.6 and 13.6% of these events leading to death. At the same time, the USA Inspector General for Health and Human Service [19] suggested that the structure of the Joint Commission surveys at the time made it unlikely that the accreditation surveys would identify patterns or instances of poor care. Following these reports, the Joint Commission (the main USA health service accreditation agency) increased the level of patient safety and quality assessment in their accreditation standards and developed evidence-based outcome indicators with the Agency for Healthcare Research and Quality (AHRQ), providing a valuable outcome database on which many of the later studies were based. We therefore split the benefit studies by date to illustrate this point.

Results

The results of the framework analysis are shown in Tables 2 and 3 for the cost studies and Tables 47 for the benefit studies. The number of cost studies was relatively small, and in many cases [2023], the studies were restricted to a single facility, but the results give an indication of the costs, ranging from 0.2 to 1.7% of total expenses per annum when averaged over the accreditation cycle (typically 3 years). In some cases, the costs were incomplete but most included the incremental costs of preparation and survey. However, remedial costs resulting from the external survey did not seem to be included. The lowest costs came from a single hospital study estimating costs for ongoing accreditation, whereas the highest costs came from one of the largest studies [24] looking at costs for initial accreditation with a recently introduced accreditation body, and indicated that costs were relatively higher for smaller and rural centres.

View this table:
Table 2

Summary of cost studies

StudyContextMethodologyFindingsKey messages
Bukonda et al. [51]Zambia—implementation of a nationwide accreditation programme for hospitalsDocument analysis, interviews and focus group discussions∼US$0.787 million (1.1% of annual health budget) for a full accreditation cycle (US$10 000 per hospital) is given as the main reason the programme stalled for lack of funding following USAID withdrawal from the projectAuthors suggest that setting up an accreditation system should be done on a proper financial footing with established infrastructure to support the programme, especially in a developing country
Fairbrother and Gleeson [20]Australia, 1996—review of initial accreditation survey in a teaching hospitalClose and open ended surveys to review initial accreditation. Focus on senior managers, clinical staff and department heads post accreditation survey. 88 surveys returned (44%)Costs estimated at ∼AU$1 million (in 1996) in 6 months prior to the survey, accreditation agency costs for the survey and other costs not includedAccreditation placed a high demand on organizational resources with little evidence of an impact on clinical service delivery
Foster and Gipe [21]USA—-Community Hospital (acute and sub-acute) ongoing accreditationDocument analysis and author estimates to determine ongoing accreditation costsCosts estimated at 0.6% of gross revenues. Overtime comprised of 67% of the costs (5500 h over 12 months)Authors note frustration that accreditation surveyors spent more time on processes than outcomes, and in looking through documents rather than in discussion with staff
Mihalik et al. [22]USA—mid-sized mental health facility undergoing initial accreditationAuthor estimates for incremental costs for undergoing initial NCQA (National Committee for Quality Assurance) accreditationEstimated initial accreditation costs for a mid-sized mental health facility in the USA in 2003 was US$1.3 millionThe study was based on the authors' personal experiences and would be difficult to scale up; however, the authors acknowledge the importance of being able to measure and compare outcomes
Rockwell et al. [23]USA—costs of initial JCAHO accreditation survey in a neuro-psychiatric hospitalDetailed estimation of costs using attendance rosters, staff self-reports, and meeting minutesTotal costs of 1.01% of the Hospital's operating budget in the year of the survey. Accreditation survey visit costs were 7% of total, remainder was in preparation work and ongoing costs of complying with standardsAuthors discuss the substantial opportunity costs of the survey in terms of resources re-allocated from clinical care
Zarkin et al. [24]USA—methodone outpatient treatment centres—initial accreditation under recently introduced accreditation bodySites chosen as those undergoing accreditation as part of a larger randomized controlled trial (RCT) assessing the introduction of accreditation. Surveys conducted before, during, and after accreditation site visit. Technical help and accreditation survey fees also assessedCost of preparing for and undergoing accreditation was ∼5% of annual operating costs. Majority of costs (82%) were site preparation costs, related to reviewing policies and procedures, meetings and training, and would likely be less for subsequent rounds. Costs for smaller and rural centres were much higher (∼17–19%)No significant difference in costs by accreditation outcome or whether the site had been accredited under previous accreditation systems
View this table:
Table 3

Analysis of cost studies

Main author and publication yearStudy yearNo of facilitiesAccreditation costsStudy designScalability >5% of facilities (Y/N)Level of independence of outcomes and assessment, + rationaleComments
Bukonda et al. [51]1997–2000NationalUS$0.79 million for initial accreditation cycle for all hospitals (US$10 000 per hospital) ∼1.1% of total health budget or 0.4% if 25 hospitals surveyed in any 1 year (as planned). Ongoing costs of US$7000 per hospital for subsequent cyclesIII-3YMExact methodology of cost calculation not given but outside agencies involvedAlthough costs estimated on a national basis the programme was not fully implemented
Fairbrother and Gleeson [20]NA1AU$1 million in 6 months prior to survey but not given as a comparative figureIVNMCosts analysed from self-assessment surveysNot all costs included
Foster and Gipe [21]1995–961US$202 475 at the hospital level ∼0.6% of annual revenues or 0.2% of gross revenue averaged over 3-year cycleIVNLAuthors estimates of costs at linked facilityOngoing accreditation
Mihalik et al. [22]NA1US$1.3 million—this includes 18 months of ongoing expenses—annualized cost of US$333 331 over 3-year cycle. Total hospital expenses not givenIVNLAuthors estimates of costs at linked facilityCosts not given on a comparative basis making it difficult to determine overall levels
Rockwell et al. [23]19891US$326 784 or 0.3% of total expenses averaged over 3-year cycleIVNMCosts calculated from staff self-reports46% of costs related to medical records
Zarkin et al. [24]2003102Average cost was US$48 005, ∼5% of average operating costs (1.7% of total expenses if annualized over 3-year cycle)III-3YHCosts analysed from self-assessment surveys, but team also verified a sample of costsAuthors discuss self-reporting bias. Study only considered preparation costs not potential remedial costs following survey. Given some hospitals already accredited through other bodies—initial cost unlikely to be biased on the downside
View this table:
Table 4

Summary of benefits pre-2000

StudyContextMethodologyFindingsKey messages
Barker et al. [52]USA—acute care, medication errorsProspective cohort study, stratified sample of acute and skilled nursing facilities in two states. Direct observation methods were used to detect medication errorsMean error rate was 19% (10% excluding timing errors) with 7% of errors considered potentially harmful. No significant difference between accredited and non-accredited facilitiesError rates likely to be understated given the number of facilities that declined (with the reason that they were worried about poor scores) and the obvious presence of observers
Duckett [53]Australia—acute care facilitiesSurvey of senior executives in 23 hospitals on 6 out of 18 operational areas (standards)Clinical outcomes not measured but results suggest accreditation is related to improved levels of audit, a more formal clinical organization, and a safer working environmentResults not conclusive and only limited stakeholders were consulted
Griffith et al. [54]USA—acute care facilitiesAnalysis of JC accreditation scores vs. seven outcomes (includes financial status of facility) across ∼1600 hospitals (25% of total)Only two outcomes were significantly associated with higher accreditation scores (mortality and % outpatient revenue). Other outcomes (complications, cash Flow, adjusted cost per case, adjusted length of stay and asset turnover) were not significantly relatedAuthors discuss both the criteria design (which they suggest should be linked to evidence-based outcome measures) and the subjective nature of the accreditation assessment
Hadley and McGurrin [55]USA—medium sized mental health facilitiesAnalysed data from 216 (77% of total) hospitals on seven quality outcomesOnly 18% of facilities were not accredited (or certified), but outcomes were not significantly associated with accreditation; however, authors state that these facilities generally had higher scores. Outcomes mainly process related (cost per patient, daily bed cost, bed turnover, occupancy) rather than clinical outcomesFurther analysis of the variables identified in the data could help determine the relationships
Miller et al. [35]USA—Acute Care facilitiesComparison of accreditation scores with AHRQ quality and safety outcomes. 1997–99 data from 2116 facilitiesLittle relationship between accreditation scores and AHRQ quality and patient safety outcome measures although low AHRQ post-operative measures were significantly associated with low accreditation scores. Most hospitals scored between 90 and 100% on accreditation vs. a much wider variation in AHRQ measuresAuthors discuss whether accreditation and AHRQ outcomes measure similar aspects of quality and safety of care, and suggest a mixture of structural, process and outcome measures be used
Salmon et al. [29]S. AfricaProspective RCT of 38% of public hospitals in Kwa-zulu Natal. Measured compliance with standards and eight quality indicators before and after accreditationOnly one indicator (nurse perception of quality) out of 8 showed a significant change between accreditation and controls. There was a significant increase in compliance with standards but that did not translate into improved outcome measuresAuthors suggest it may take longer for impact of accreditation to result in improved outcomes, or that the programme may improve structures and processes but not impact the outcomes used in the trial
View this table:
Table 5

Summary of benefits—post-2000

StudyContextMethodologyFindingsKey messages
Awa et al. [28]Saudi Arabia—acute care facilityAnalysis of 81 performance indicators measured over 4 years (pre- and post-Canadian accreditation) in one teaching hospital28 of the outcomes (35%) showed significant improvement after accreditation but the impact of the remaining 65% not discussedImproved performance of some key indicators but only in one hospital and other confounding factors not discussed. Sustainability an issue with many risk factors improving during accreditation but trending down following the survey
Braithwaite et al. [43]Australia—acute care facilitiesIndependent blinded assessment of accreditation performance against outcome variables in 19 facilities (5% of Australian acute care system). Team reviewed accreditation performance against organizational culture, organizational climate, consumer involvement, leadership, and clinical performanceHigher accreditation scores were positively correlated with organizational culture, leadership, a trend with clinical performance but unrelated to organizational climate and consumer involvementAuthors discuss the importance of accreditation performance in reflecting the contextual factors associated with quality of care and clinical improvement, and note the lack of consumer participation in the facilities studied
Longo et al. [56]USA—acute care facilitiesTwo factor (geographic and time) design with surveys on patient safety measures for 107 acute care facilities, 18 months apart in two different statesAfter adjusting for bed size, location and management type, accreditation was the key predictor of patient safety implementationSurvey conducted after significant increase in patient safety requirements for meeting accreditation following two critical reports on patient safety in the USA. Study looked at implementation of systems not efficacy of systems outcomes
Menachemi et al. [57]USA—Ambulatory Surgical Care (ASC) centresAnalysed difference in unexpected hospitalization post-ASC procedures for state certified vs. National accreditation ASCs in one U.S. state (Florida). Results controlled by volume, patient characteristics, severity of illness and reason for admission (DRG codes). Total of 364 ASCs reviewed, 45% state certifiedApart from a reduction in hospitalization rates 7–30 days post-surgery for colonoscopy for JACHO accredited facilities, no difference was found for other procedures between accredited and non-accredited facilitiesDifferences in assessment standards between state certification and accreditation not discussed
Sack et al. [48]GermanySurvey of 78 508 patients 4 weeks after discharge to determine whether a patient's willingness to recommend the facility to others was linked to accreditation. Results from 36 777 patients were analysed using odds ratios using gender, age, number of beds and hospital teaching status as covariates66.3% of patients across 73 hospitals recommended the hospital that they had recently received care from. However, there was no evidence of a relationship between recommendation and accreditationAuthors suggest that patient satisfaction outcomes should also be considered when considering accreditation rather than assessing for compliance with standards
Schmaltz et al. [36]USA—acute care and critical access facilitiesData analysis of 2004–08 performance statistics from the Joint Commission and CMS databases on 16 clinical performance outcomes which were grouped into four summary scores: three condition related scores (acute myocardial infarction, heart failure and pneumonia) and an overall score. The three conditions measured account for 15% of Medicare admissions. Hospitals included were either never or always accredited during the study periodBeing accredited was significantly associated with higher initial baseline scores in 2004, having larger gains over the time period, and higher scores in 2008The authors note the Joint Commission accredits over 80% of all facilities (>90% of all beds), so there may be structural or other reasons for the difference with non-accredited facilities, and paired controls were not used
Sekimoto et al. [58]JapanSurvey of infection control (IC) practices in teaching hospitals in Japan. Data analysed from 335 hospitals (out of 638) who responded to both surveysHigher scores IC associated with accreditation (especially first year of accreditation) but implementation of effective IC activities was poor and not linked with accreditation. Fee for service structure does not encourage IC activityAuthors suggest that fee for service structure does not encourage IC activity, and that recent changes to a more punitive approach may encourage cosmetic changes
Shaw [59]EuropeSite visits of 89 hospitals across 7 European countries to complete a hospital assessment tool comprising of 229 criteria over 6 dimensions. Data from 71 hospitals was analysed, 34 were accredited, 10 had ISO certification, and 27 had neither (18 were excluded)Overall scores for four dimensions (management, patient safety, clinical organization and clinical practice) were significantly higher for accredited hospitals than for ISO 9001 certified hospitals and also for those not accredited or certifiedQuality and safety structures and procedures are more in evidence in hospitals that have been externally assessed
Thornlow and Merwin [60]USA—acute care facilitiesData analysis of 1.4 million inpatient discharge records from 115 hospitals across 20 different states, using data from the 2002 Nationwide Inpatient Sample and patient safety scores from the 2002 Joint Commission accreditation surveys. Four patient safety outcomes were measured and analysed against various demographic and clinical variablesThe results for accreditation show a relationship between lower patient safety practice use and higher infection rates and decubitus ulcers in two of the four subsets—but no relationship with post-op respiratory failure or failure to rescueAuthors discuss whether accreditation scores are capturing practices and procedures that ensure safe patient care, but also discuss the evidence base for their indicator selection. Authors note the clustering effect of accreditation scores and used a subset of accreditation performance scores to measure use of patient safety practices
View this table:
Table 6

Analysis of benefits studies—pre-2000

Main author and publication yearStudy yearNo. of facilities studiedNo of outcomes measuredStudy designScalability >5% of facilities (Y/N)Level of independence of outcomes and assessment, + rationaleLevel of claim that accreditation improves patient safety and quality of care, + rationaleSource of potential bias
Barker et al. [52]1999361III-3NHExternal, independent assessment of medication errors on drug roundsLLower rate of medication errors (nearly 1 in 5 doses), not associated with accreditationData from independent reviewers, but potential bias from high number of facilities refusing to take part in the study. In addition, non-accredited facilities tended to be smaller than controls
Duckett [53]1978–90236IVNLQuestions based on accreditation standardsLResults discussed qualitatively not quantitatively, and not conclusiveNot all sub-groups had control groups making comparisons difficult. Statistical analysis of results not given
Griffith et al. [54]1996–9815967IVYHOutcomes independently chosenL50% of accreditation scores were between 91 and 97, creating a ceiling effect. Only 2 of 7 outcomes (higher mortality and higher outpatient activity) were significantly related to lower accreditation scoresAuthors caution against self-selection as hospitals can suppress accreditation information
Hadley and McGurrin [55]19832167III-3YMOutcomes not designed to measure quality of care, and selection method not detailed—but outcomes were independent of accreditationLNo statistical difference in quality of care indicators between accredited or certified hospitals and those with neitherLarge data sample but statistical analysis not detailed
Miller et al. [35]1997–99211633IVYHClinical outcomes independently developed by AHRQLAccreditation scores not significantly correlated with individual Inpatient Quality Indicators. The results for Patient Safety Indicators were more mixed with two factors significantly associated with higher scores and two with lower. Combined scores post-op were associated with lower accreditation scoresIssues with ceiling effects in terms of clustering of accreditation scores. Authors also discuss problems of accuracy in using hospital data (coding errors, etc.) but counter that the specificity of the outcomes should help offset this
Salmon et al. [29]1998–2000208IINMCompliance with standards not an independent measure. Quality indicators determined by committee but many process basedLCompliance with standards and quality indicators measured at two points in time. Only one outcome measure showed a significant relationship with accreditationScalability across the province but not necessarily countrywide. Data collection issues in terms of timing, sampling procedures and indicators used
View this table:
Table 7

Analysis of benefits studies—post-2000

Main author and publication yearStudy yearNo. of facilities studiedNo. of outcomes measuredStudy designScalability >5% of facilities (Y/N)Level of independence of outcomes and assessment, + rationaleLevel of claim that accreditation improves patient safety and quality of care, + rationaleSource of potential bias
Awa et al. [28]2006–09181III-3NMOutcomes independent of accreditation but method for outcome selection not givenMOnly 35% of outcomes positively associated with accreditation (results of others not discussed)Only one hospital measured, results only partially discussed
Braithwaite et al. [43]2005–07195IVYMTeams measuring contextual measures were blinded but use of indicators is user-determined creating self-selection biasM2 out of 7 indicators show statistically significant relationship to accreditation performanceAuthors discuss limited power of the study and lack of controls due to widespread use of accreditation so size of control group not known
Longo et al. [56]2002–041078III-2NLOutcomes related to accreditation standardsMAuthors discuss accreditation as a predictor of patient safety implementation systems rather than outcomesBreakdown of number of accredited and non-accredited hospitals not given
Menachemi et al. [57]20043641III-3YHOne outcome but over two time periods and five procedures and independently assessedMAlthough a significant difference in unanticipated re-admissions was only found for one out of five procedures (colonoscopy) this comprised of 43% of all procedures measuredResults applied to raw data and controlled for facility volume and patient characteristics
Sack et al. [48]200778 508 patients from 73 hospitals1III-3YHOne outcome but independent of accreditation complianceLNo evidence that accreditation was linked to patient satisfaction as measured by patients’ willingness to recommend the hospital they had recently attendedLarge survey but only one “indirect” aspect of patient safety examined
Schmaltz et al. [36]2004–08367916III-2YHEvidence-based independent clinical outcomesH73.7% of non-accredited hospitals had a 90% or better performance overall score in 2004, and an 11.8% improvement by 2008—but accredited hospitals showed higher initial and composite scoresCharacteristics of excluded hospitals could have biased results as 75% of excluded hospitals were non-accredited
Sekimoto et al. [58]2004–053351III-3YLQuestions linked to IC parts of accreditation surveyLAccreditation associated with higher infection control scores but not necessarily clinically or statistically significantHigh degree of scalability across teaching hospitals but not necessarily across non-teaching facilities. Authors discuss issues around self-assessment
Shaw [50]2006–07716 (229 criteria)III-3NMAssessment tool designed for the study—but some measures mirror accreditation standardsMOverall scores are significantly higher (P < 0.05) for accredited hospitals vs. certified hospitals or those with neitherScalability an issue due to small sample size in some countries resulting in a lack of controls, and lack of direct comparisons between accreditation agencies. Authors attempt to overcome self-assessment issues by direct observation of criteria via hospital visits
Thornlow and Merwin [60]20021154IVNHClinical outcomes independent of accreditation, and collected by independent bodyLOnly 2 out of 16 potential outcomes (four accreditation sub-scores and four indicators) showed a significant associationSample hospitals had slightly different characteristics (size, location, and ownership) to national averages. Sample sizes also differ between outcomes

The single hospital studies were not independently verified and not necessarily representative, but they do make a useful contribution to estimates of the probable level of overall system costs. The more detailed costing studies in the grey literature [25, 26] used the activity-based costing methods and champion the use of incremental costs. This includes those costs that would not have occurred in the absence of accreditation rather than reviewing all the costs related to compliance with existing rules and regulations. This approach has a solid basis in economic theory [27]. Some of the studies also touch on the opportunity cost of accreditation in terms of the time not spent on clinical care, but the remedial costs of accreditation are not widely discussed or estimated.

Only two of the benefit studies were experimental or interventional in design [28, 29]; one of which [29], although not strictly a peer reviewed article, was included as it was widely cited and the subject of a Cochrane review [30]. The number of aetiological studies is in keeping with the implementation of accreditation usually being outside the control of research teams given the scale of the intervention involved, but it does create problems of being able to identify a viable control group, which we saw as one of the main sources of bias. We made the assumption that the main benefits of accreditation were improved patient safety and quality of care, as this was a common theme in the studies. As discussed, the strength of claim assessments were lower for the pre-2000 studies but more mixed for the post-2000 studies. This could be a due to a number of factors: publication bias, availability of outcome measures, accreditation standards becoming more geared towards outcomes, or changes in health care (whether or not related to accreditation) creating improvements in patient safety and quality of care.

Discussion

The wide variety of outputs measured, and inconclusive results, highlight the difficulties of selecting appropriate accreditation related outcomes in a health care setting. One issue is that health services may have a choice of accreditation process or agency and are therefore being measured to different standards. In addition, the length of the accreditation cycle means that health services will be at different stages of implementation at any point in time during analysis. Shaw [31] suggests that the lack of clear outcomes in terms of accreditation benefits is often due to confusion over the ‘endpoints’ of accreditation, especially for government mandated programmes where accreditation is often used more for regulation and public accountability rather than as a means of voluntary self-assessment and quality improvement. Donabedian's work [32] on reviewing quality processes is insightful in breaking down the process into quality assessment, quality monitoring and quality assurance. He describes assessment as being an epidemiological review of management, looking at quality from both a provider and patient perspective. Monitoring comprises more of an administrative approach to ensure the health system is achieving its objectives, whereas assurance has a broader mandate and includes education, resources, financing and legal frameworks. Although accreditation has historically been more of an assessment and monitoring process through measuring compliance against standards, changes since the IOM report have resulted in a more outcomes based, continuous quality improvement approach. This fusing of the audit and quality outcome components suggested by Shaw [31] makes it difficult to define and isolate the benefits of accreditation from other quality and safety improvements.

For health economic evaluations, Shiell et al. [33] differentiate complex systems and complex interventions, suggesting that resources and outcomes can be evaluated without needing to understand how the intervention works if there is enough clarity (in terms of inputs–outputs and boundaries) to ensure that changes can be measured and valued, but the challenge comes from trying to determine what the outputs are, as well as how they can best be measured. Chassin et al. [34] address this last point by recommending that outcomes must meet four key accountability criteria: robust evidence (Research), whether the process has been carried out satisfactorily (Accuracy), a clear and direct link between accreditation and the outcome (Proximity) and no unintended or unwanted actions (No Adverse Effects). These criteria have been adopted by the Joint Commission in determining the outcomes measured by accredited hospitals and are used in several studies [35, 36]. For the final criteria, the unintended consequences of publishing outcome measures have been widely debated [3740] in terms of whether accreditation outcomes should be made public and also what sort of rewards or incentives should be offered.

The observational nature of these studies makes it difficult to provide statistically robust evidence of the efficacy of accreditation or be able to show causality, but is in line with the MRC guidelines [41] on evaluating complex interventions which recommend using non-experimental methodologies where conventional randomized controlled trials are not feasible. Many of the benefit studies include qualitative surveys [26, 42, 43] to investigate the organizational and management factors that lead to clinical and organizational improvement. In particular, Braithwaite et al. [43] highlight the non-clinical nature of potential benefits and found that accreditation performance was significantly associated with a positive culture and demonstrated leadership. Further studies could look at outcome measures to determine this empirically, perhaps through staff turnover data, work related injury claims or staff satisfaction surveys. Several studies [36, 43] describe the lack of control groups as a major problem where accreditation is either mandatory or so widely implemented as to create possible bias in the control group.

In practice, there are considerable difficulties involved in the economic evaluation of health care programmes that influence the entire system. These arguments have been well articulated by Drummond [44] and McIntosh et al. [45], for example there are often no clear start and end points to the intervention; the desired outcomes are not always well articulated, defined, or measured; market pricing is often absent and analysis relies on shadow pricing or willingness to pay estimates; externalities such as waiting lists and adverse events may shape consumer preferences and clinical heterogeneity is likely to influence outcomes. Despite these difficulties, Drummond [44] emphasizes the benefits of a framework that requires a systematic and explicit approach to analysing the benefits foregone as well as benefits obtained. He argues that articulating the arguments and constructing a sensitivity analysis can help inform policy makers in terms of discussing the explicit assumptions used in the analysis. However, the problems of measuring benefits should not be underestimated and Dawson et al. [46] stress that although economic theory has caught up with current medical practice in terms of providing methods for analysis, data collection can still be an issue, especially for recording patient experiences in health care services as the data are often not available.

Two reports from the grey literature stand out in terms of costs. The Australian Productivity Commission [25] looked at administrative costs for general practice (GP) and concluded that GP accreditation costs came to about 1.1% of total costs (∼AU$49 million in 2005 (∼US$37 million at 2003 monthly average rates)). However, this was offset by accredited practices being eligible for practice incentive payments—with costs estimated at between 5 and 25% of these potential benefits—and rural loading designed to offset the higher costs for rural practices. Doyle et al. [26] provide detailed activity-based costing for two Irish hospitals over different accreditation cycles (average of 0.29% of annual budget averaged over two cycles) but highlight the time intensive nature of the analysis and the difficulties face obtaining data in a financially constrained environment. The Australian Commission on Safety and Quality in Health Care produced a regulatory impact statement on the implementation of proposed new accreditation standards [47] but only estimated implementation costs from the perspective of the accrediting agencies rather than a wider stakeholder base. However, more significantly, the paper explicitly states the expected benefits: reducing harm to patients and reducing the cost of care, improving system and consumer productivity and improving consumer trust in the health system. Although these are neither measured nor monetized in the report, the potential benefits provided for each of the 10 new standards provides a starting point to establish a baseline for a more comprehensive review of accreditation benefits.

Conclusion

The perceived gap in the literature for economic appraisal studies in health services accreditation has been confirmed by this review which also highlights the lack of a clear-cut relationship between accreditation and improved patient safety and quality of care outcomes identified in the benefit studies. Several studies [36, 43] discuss the inherent difficulties in study design as accreditation, in most of the countries where it is implemented, is either mandatory or quasi-mandatory (e.g. by being linked to funding) which precludes both a randomized trial and a suitable control group. Although the scale of several studies was impressive, the lack of control groups creates complications in measuring both causality and the overall impact of accreditation, especially given the clustering effect of accreditation scores, difficulties in isolating accreditation from other safety and quality measures, difficulties of selecting indicators with full accountability criteria [34], and the timing of accreditation surveys with the effects of other institutional changes. In addition, many of the studies were focussed on clinical outcomes and only one study focussed on patient experience of care [48]. With significant investment indicated by the cost studies and the current debate about the role of accreditation in healthcare [49, 50], a formal economic evaluation is needed to create a baseline point of reference and for measuring and monitoring any reforms in accreditation processes by providing a more robust and explicit understanding of the costs and benefits involved. A clearer definition of the expected benefits would enable measurement and monetization to determine whether the benefits do outweigh the costs. This type of CBA, whereby a wide variety of potential benefits are monetized to provide a common denominator, can also be used to provide a comparison with other quality and safety measures and determine whether there are other, more cost effective, ways of achieving improvements in patient safety and quality of care.

Funding

This work was supported under the Australian Research Council's Linkage Projects funding scheme (project number LP100200586) and NHMRC Program (grant number 568612).

Conflict of interest

None declared.

References

View Abstract