OUP user menu

Healthcare accreditation systems: further perspectives on performance measures

Ebrahim Jaafaripooyan, Dila Agrizzi, Faizollah Akbari-Haghighi
DOI: http://dx.doi.org/10.1093/intqhc/mzr063 645-656 First published online: 27 September 2011

Abstract

Objective The purpose of this paper is to identify and suggest a number of performance measures to facilitate the evaluation of accreditation programs in healthcare.

Methods The paper is based on an exploratory research which has used qualitative methods, including snowball sampling technique, email interview and thematic content analysis.

Participants Respondents (experts and professionals) were selected from a diverse spectrum ranging from healthcare organizations, universities and accreditation-associated institutions.

Results The analysis of the data provided key measures to be considered in the evaluation of accreditation programs' impact at macro and micro levels as well as their nature and operations. The measures can be used to, for example, assess the degree of stakeholders' reliance on accreditation results, measure the cost of accreditation for participating organizations and serve as a formal mechanism for accredited organizations to appeal accreditation decisions.

Conclusions This paper has brought together a number of generic, yet influential and workable, measures which could be utilized for assessing the overall performance of an accreditation program in healthcare. The application of these measures depends on the features of given accreditation program and the context in which the program operates. Therefore, the next step/steps in the assessment of an accreditation program might be choosing the measures suiting that program.

  • performance measures
  • accreditation programs
  • performance measurement and improvement
  • healthcare

Introduction

It is argued that accreditation is one of the influential mechanisms for assessing the performance of healthcare organizations (HCOs) and improving the quality and safety of healthcare services [1, 2]. Accreditation has been defined as an external evaluation mechanism which assesses the performance of HCOs through investigating their compliance with a series of pre-defined, explicitly written standards [35]. Its aim is to encourage continuous improvement of quality rather than simply maintaining minimal levels of performance. It is also described as the public recognition emanating from the achievement of specific standards by a HCO, which is demonstrated after an independent external assessment of the organization's performance [6]. Specific features of accreditation have presumably made this system more preferable for regulators, providers, third parties and customers in healthcare, vis-à-vis other similar programs transposed from industrial sector [710]. These characteristics include [8, 11, 12]:

  1. The comprehensive and multidisciplinary nature of the assessment.

  2. The fit of the assessment method to the unique features of healthcare.

  3. Inclusion of improvement as a goal of the assessment.

  4. The use of highly trained surveyors with experience in healthcare.

While accreditation programs are set up to evaluate HCOs, their performance, as such, needs to be assessed to both maintain their alignment with the initially determined objectives and improve their merits and capabilities to continuously detect the deficiencies and malpractices in HCOs [13, 14]. Critical outcomes [15] and information asymmetry between providers and consumers in healthcare [16] have added to the sensitivity and importance of these programs in this area of the public sector. Their high cost for both those running and being evaluated by accreditation programs has augmented this importance [5, 17]. Despite this apparent urgency, research into the effectiveness of these programs is still at an embryonic stage [18, 19] and fewer studies than the area deserves have embarked on an examination of their performance [13, 2022].

There is a lack of empirical evidence in the literature exploring performance-related measures for assessment of accreditation programs. Existing efforts have largely focused on a single aspect of accreditation's performance and impact [cf. 18]. Accordingly, this paper seeks to provide valuable insights in direct relation to the assessment of these evaluatory mechanisms, by suggesting a number of generic measures for their evaluation. This is an important contribution to the literature and practice for two main reasons. First, the provided measures could facilitate the challenging task of assessing accreditation programs' performance [14, 23]. Some of the measures could also be used to support the decisions by authorities and policymakers to gauge the success of their programs in a specific period. Second, the paper reflects the voice of groups (i.e. academics and professionals), which have not been considered in this variety by previous studies.

The paper is structured in the following fashion. The first section provides a background on the related literature concerning the performance measurement of healthcare accreditation. The second section is devoted to describing the research methods employed. The empirical findings are presented in the next part, followed by discussion of the results.

Background

Previous research on the performance and impact of healthcare accreditation on HCOs has shown mixed and inconsistent results [18, 19]. Existing literature abounds with various studies showing either confirmatory [e.g. 2427] or neutral [e.g. 2831] evidence regarding the effects of accreditation programs on HCOs, with no consistent results. In their extensive review of health sector accrediation research, Greenfield and Braithwaite [18] could only find evidence for capacity of these programs to promote change and professional development in HCOs. Some studies on accreditation have also been of limited support for its effectiveness, due to their anecdotal nature or small-scale coverage [20]. Therefore, there has always been a call for more research to investigate the effectiveness and performance of accreditation programs [e.g. 14, 18, 20, 3235].

The efforts to examine the impact of accreditation program have assumed different forms in the literature. Various aspects of accreditation have been discretely addressed. For instance, Greenfield and Braithwaite [18] have clustered the exsiting studies in terms of their focus on accrediation. They found the cases in relation to accreditation's financial impact, consumers and professions' attitudes to accreditation, promoting change and professional development and public disclosure by these systems. As such, Sunol et al. [36, p. 27] have located the prior literature on the impact of these programs in three distinct areas:

  1. The impact of accreditation on the quality and safety of healthcare delivery.

  2. The efficiency of accreditation tools and systems for providing feedback with reliable information both to the accreditation organizations as well as all key stakeholders.

  3. The impact on the capacity development of systems.

Braithwaite et al. [20] argue that only a multi-method, multidisciplinary, multi-level research design is capable of providing reliable evidence on the performance and impact of accreditation. Scrivens' approaches of ‘experience/perception and objective indicator’ could provide a valuable lens to look critically into and make sense of the current trends in studying accreditation programs [37, p. 6]. The first approach requires that the perceptions and experiences of various groups, involved in or related to accreditation (e.g. HCOs' professionals and stakeholders) are elicited toward the different aspects and functions of the program. Public sector abounds with those studies adopting this approach; in the healthcare [e.g. 38, 39] and the education sector [e.g. 40]. In these studies, the perceptions of healthcare professionals and accreditation surveyors have been solicited upon the performance of their running accreditation system in terms of accreditation standards, surveyors and implementation processes.

The objective indicator approach, instead, calls for identifying and developing tangible and intangible measures of success (i.e. hospital performance indicators, patient satisfaction, etc.) in connection with accreditation in HCOs. In line with this approach, any change (e.g. in the quality of services) in accredited HCOs is quantitatively investigated and the positive effects are attributed to the effective function and performance of the accreditation programs. This approach subsequently sees the changes as a confirmatory sign of the programs' impact on the organizations. The respective studies include those, for instance, that sift through the relationship between accreditation and clinical indicators [41, 42], patient [43] or provider satisfaction [21] and the changes triggered by these programs in subject organizations [e.g. 32, 44].

Both approaches have arguably their own strengths and weaknesses. The perception approach has been criticized for being superficial and judgmental [37]. Deficiency of the objective approach lies mostly in the difficulty of measuring performance in healthcare, such as long-lasting, multi-factor and probable outcomes [4547]. Øvretveit and Gustafson [48] point to the methodological challenges in measuring healthcare outcomes and conceiving causality between accreditation and its possible outcomes. de Walcque et al. [5] refer to possible disagreement of the programs' stakeholders on intended outcomes as a real cause of this challenge. As such, Shaw [14, p. 455] has expressed concerns about the difficulty of defining ‘endpoints’ of an accreditation program and their change based on the expectations of users and observers. Given these challenges, the results of different studies that assume the objective indicator approach could be prone to producing inconsistent and biased results. This approach thus always carries concerns over its capability to render a valid evaluation of accreditation programs' performance in view of those challenges: just as Shaw [14] notifies of the difficulty in evaluating these programs when compared with clinical technology. Therefore, owing to the complex nature of the health sector creating the foregoing challenges, the perception approach might be considered safer from this perspective, despite its deficiencies [37, 49]. A fairly wide range of the current literature has utilized this approach as mentioned earlier [e.g. 50, 51]. The most cautious, yet credible, outlook might be to synthesize two approaches.

Following from this debate and considering the mixed results in the previous literature upon the performance and impact of accreditation programs on HCOs, this paper casts further light on the examination of these mechanisms in healthcare. It aims to draw upon the ‘perceptions and experiences’ of both experts and professionals to explore workable guidelines for the assessment and improvement of these programs. The term ‘expert’ here is used for both academics and practitioners, whose research or work is in a way related to healthcare accreditation. ‘Professional’ refers to those working inside HCOs.

Methods

This paper is based on an exploratory qualitative research approach which has utilized both semi-structured interviews and open-ended questionnaires in order to collect its related data in a two-stage process [52]. For the first stage of the research, a number of experts in healthcare accreditation from a variety of universities (e.g. the USA and UK) and healthcare accreditation-associated institutions such as Joint Commission on Accreditation of Healthcare Organizations (JCAHO), Australian Council on Healthcare Standards (ACHS) and International Society for Quality (ISQua) were interviewed. For the second stage, a group of hospitals' professionals in a developing country (i.e. Iran) was selected, in order to introduce more practical and complementary views on identification of accreditation programs' measures. The significance of professionals' perspectives is highlighted by Barker et al. [53], indicating that the mechanisms for judging the delivery of a service must have the confidence of those being evaluated.

To minimize bias related to country-specific variation in accreditation programs, questions were designed to emphasize definition and features common to most accreditation programs (see Appendix) [5, 14, 37, 54].

Selection of respondents

A purposive sample, which is considered to be the most common sampling technique for qualitative studies, was used to select potential participants [55]. Inclusion criteria for respondents at this stage were developed based on their publications (i.e. mainly books and papers in the accreditation-related refereed journals). Although there was no asserted limitation and mandate for selecting the participants from a specific country, the intention was to consult the experts from those countries which have comparably settled and successful accreditation systems (e.g. the USA, Canada, Australia and France). A ‘snowball sampling’ technique was used at a later stage in order to identify additional experts, in a way that at the end of the interviews, respondents were asked to nominate other people who would be relevant for the purposes of the research [56, p. 17]. Snowballing was helpful, because the nominator was used as a form of reference to enhance the credibility of this research. In addition, the new respondents were supposed to be capable of answering the questions of the research [56]. Overall, 35 experts were contacted, of whom 25 replied and consequently were interviewed by email during 3 months starting from May 2008.

For the second stage, the senior managerial and clinical members (i.e. hospital managers, matrons, supervisors and heads of Emergency Department, Laboratory, Radiology, Medical Record and Pharmacy) of the provincial hospitals, amounting to 150, were selected. The professionals' familiarity with the accreditation was the underlying inclusion criterion at this stage. The group was sent an open-ended questionnaire asking their perspectives and comments on the evaluation of accreditation programs. Of those contacted, 120 replied. The selection of the professionals from this country was based on two main grounds: First, the country has had a national accreditation program in place for a number of years, meaning the professionals were familiar with the overall concept of healthcare accreditation. The second reason was the familiarity with and the ease of access to the HCOs for the researchers. Moreover, the email interview (method used for the first stage) was not easily applicable to the professionals as it was with the other groups (i.e. experts), given their time constraints and the different nature of their work.

Data collection

Given the fact that experts were geographically located in different parts of the world, conducting a conventional interview could be highly expensive and time-consuming; thus the ‘email interview’ technique was adopted in this study. The main reasons for selecting this particular technique follow [57]:

  1. Potential participants were spread out in different countries and not limited to a country or an organization (as explained earlier).

  2. Given time limitations, this method was convincingly suitable, because they could respond in their own time and without any pressure, which might have positively reflected in the quality of their responses.

  3. They all were supposed to have sufficient access to the internet because of their position and job. This was proved in the later stages of the research as the majority of respondents replied to the emails.

The potential advantages of this technique have made it highly capable for fulfilling the objectives of this study. Strengths such as being less costly, quick and more freedom for respondents, could be further attribute to this technique [see 57, p. 93]. However, operationalization of this technique requires access to the internet by all respondents.

The experts were asked to answer the open-ended questions in the interview [52], drafted based on the review of the relevant literature [e.g. 5, 6, 37] (see Appendix). They were mainly directed at ascertaining ‘the measures, elements and dimensions to be utilized for evaluating accreditation programs in healthcare’. The purpose was to build up general guidelines for assessing the performance of healthcare accreditation programs. The questions were also followed by a statement asking for respondents' other related comments and leaving room for their suggestions to improve the questions. Two follow-up emails were sent as reminders to those respondents who did not respond within the deadline. This raised the response rate significantly. Further emails were exchanged in order to clarify any ambiguity seen in the responses.

In the case of professionals a self-administered questionnaire including similar questions, as in the previous stage, was distributed by the researchers through mail among the respondents. A phone reminder was also made for unreturned questionnaire after 2 weeks of initial distribution. There was no tangible incentive anticipated for the respondents, except that a copy of the results could be sent for those interested. Overall, 120 questionnaires were returned.

It should be also mentioned that since the study did not include the vulnerable groups as its respondents, the interview protocol was only once reviewed by a number of academics for its validity, and not by any institutional review board or human subjects research committee, before its application to the empirical field.

Data analysis

Data analysis was conducted using Thematic Content Analysis, which is a widely used methodology for analyzing qualitative data [58]. It identifies and categorizes the recurring themes that emerged from the data collection and, consequently, reports the key elements addressed by the respondents [59].

Drawing on this method, the data analysis was conducted by the authors and comprised five distinct stages [56]. The first stage (the preparation of data) consisted of collating the information from the questionnaires and emails. During the second phase (familiarity with data), the researchers extensively reviewed and coded the content of data based on their relevance to different aspects of accreditation programs. At the next stage of the analysis (interpreting the data), the similar codes were combined which formed a number of general themes (e.g. cost of accreditation for HCOs). At this stage, the codes were compared, refined and formulated into the themes (i.e. measures). Such themes were then tested against respondents at the final stage (verifying the data) in an attempt to increase the validity of the results. Some of the themes were removed and/or refined and polished, according to respondents' validation. At the final stage (representing the data), three main groups of measures were achieved and represented (see Tables 13).

View this table:
Table 1

Societal (macro) impact of accreditation programs

MeasureExplanation
Uptake of accreditation programs by HCOsDemand for (voluntary) accreditation programs in the health sector, in comparison with other external evaluation and improvement initiatives such as ISO and EFQM
Retention rate of HCOs by accreditation programsThe rate of HCOs that have stayed returned or switched to a (voluntary) accreditation program from ISO, EFQM, etc. during a specific period because of its quality improvement merits
The level of community (i.e. HCOs) awareness of accreditation programsThis might be an indicator of (voluntary) accreditation programs' success and dominance in a society
Reflection of local healthcare priorities in the standards of accreditation programsThis is important in the case of national programs, which they give priority to the evaluation of locally delivered services
Representative governance structure for accreditation programsRepresentatives from different stakeholders, especially vulnerable groups (e.g. patients) and third parties, in the programs' governing structure (and surveying team). This could minimize the influence of powerful stakeholders (e.g. states) in the evaluation process
Independence of accreditation programsAn independent body for accreditation of HCOs which is not influenced by powerful stakeholders such as states and evaluates them based exclusively on their merits
A formal mechanism to appeal accreditation decisionHCOs are formally allowed to appeal and receive a convincing response in the event of complaints
Disclosure and clarity of accreditation resultsAccreditation programs communicate the ranking of HCOs to the public which is discernible for the public
Cost of accreditation programs for HCOsAll costs (e.g. application fee) of the programs for HCOs as opposed to similar initiatives such as ISO
The degree of stakeholders' reliance on accreditation results in their decision-making process
  • This could refer to the satisfaction of different stakeholders with the performance of the accreditation programs, validating the evaluation ranking made by these programs

  • Stakeholders such as third-party organizations seek accreditation ranking of HCOs to set their service contracts with them

  • Governments rely on this ranking to allocate resources to HCOs

  • Patients in those health systems with the ‘choose and book’ procedure might rely on the ranking of the hospitals to look for their intended services

Transparency and consistency of accreditation programs
  • All stages and processes of accreditation programs are crystal clear for HCOs (i.e. they know what they are asked by accreditation programs) in a way that the HCOs are satisfied with the transparency of all stages and requirements of accreditation programs

  • HCOs are informed of the changes in accreditation programs regularly and in advance

Responsiveness and accountability of accreditation programs
  • Willingness (obligation) of accreditation programs to justify their decisions about the performance of HCOs and take the responsibility, if required

  • Their intention and willingness to take into account the capabilities and ‘out of hand’ problems of HCOs while meeting the requirements of the programs

A reliable communication networkAccreditation programs have a regularly updated website for disseminating relevant information about their process and standards for HCOs
Valuation of stakeholders' feedbackSystematic feedback from stakeholders concerning all stages of accreditation is periodically sought
View this table:
Table 2

Methodology (survey and standards) of accreditation programs

MeasureExplanation
Trained, experienced and healthcare- (hospital) oriented surveyors
  • The use of specifically tailored training programs for surveyors

  • Regular examination of surveyors' selection and training processes

Appraisal of surveyors' performanceA mechanism for regular appraisal of surveyors' performance
Surveyor replacement and turnover rateThis could reduce the risk of surveys turning into an informal and routinized process and introduce fresh eyes to the process (low turnover times might eliminate experience element)
Presence of HCOs' members (as observers) during surveyors' evaluationThis could both validate the assessment process and justify the results to the members and create more clarification on HCOs' activities to surveyors
Ongoing monitoring of HCOs during the intervening time between two accreditation surveysIn light of long intervening times (e.g. three years), monitoring HCOs (via announced and unannounced visits) during this time could keep HCOs loyal to the requirements of (mandatory) accreditation programs and safe for patients
Cross-check of the survey report with HCOs before final submission for rankingThis might increase the validity and acceptability of the results to HCOs
Access of the HCOs to survey report after evaluationApart from confidential aspects, availability of the reports for HCOs after evaluation might show how they can attend to and improve the details of their operations
Report turnaround timeTime between the onsite visit and delivery of final survey report and recommendations to the hospitals (e.g. it was claimed by the professionals that the shorter this time the sooner the hospitals could identify and make an effort to rectify the problems)
Consultatively driven standard development processA significant input from all stakeholders (e.g. providers of care, consumers and purchasers, government, insurers and healthcare administrators) specifically into accreditation programs' standard development process
Different patterns of evaluation or standards for/in accordance with various hospitalsIdentical or different types of standards are applied for evaluating different hospitals. Given the different mission of these hospitals, similar standards might not generate a fair evaluation of their all activities
Feasibility of standardsWhether HCOs are able to fulfill the accreditation standards in practice
Clarity of standardsThe intention, meaning and interpretation of the standards are understandable for all participating groups (i.e. HCOs, surveyors)
Communication of accreditation standards to HCOs by accreditation programsHCOs have access to the standards against which their performance is to be assessed
Consideration of documenting requirements in accreditation standardsThe programs insist on the documentation of the services (i.e. encouraging evidence-based assessment)
The scope of the standardsThe coverage of all type of activities and services of HCOs by accreditation standards, including hotel-type services and administrative and financial activities
Inclusion of ‘outcome-related’ metrics in accreditation standardsGiven the argument that assuming a linear relationship between processes and outcomes in healthcare is thought to be challenging, attention to outcome-related indicators seems necessary, in addition to processes and structures
Consideration of structure and process standards by accreditation programs in developing countriesUnlike developed countries with well-organized structures and processes, developing countries should still focus more on their structures and processes, along with outcome indicators
Inclusion of ‘core clinical activities’ of HCOs in accreditation standardsThe majority and core of the services rendered in HCOs are clinical, which are argued to be more influential in care delivery processes
Regular review and update of accreditation standards
  • A regular review and update system set up specifically for the standards

  • The frequency of reviewing and updating process in a specific period

Participation (attraction) of HCOs' staff (especially clinical people) in accreditation processes
  • This could increase the credibility of the accreditation programs' requirements and facilitate their operationalization in HCOs

  • This might be undertaken in the self-evaluation stage of HCOs (i.e. preparation for the accreditation)

A sound and reliable scoring systemAn evidence-based rather than judgemental system for allocation of the scores to the related activities (more reliance on evidence for scoring)
The use of a self-evaluation system by accreditation programs to ensure their continued relevance to current practice of HCOsAccreditation of HCOs based on newly emerged methods and development of standards for new services afoot in HCOs
View this table:
Table 3

Organizational (micro) impact of accreditation programs

MeasureExplanation
Outputs of accreditation programs
  • Increased compliance with accreditation standards: the number of HCOs met the (mandatory) accreditation standards vis-à-vis previous accreditation periods-measured based on a HCOs' accreditation rank

  • Immediate tangible actions taken by HCOs before and after an accreditation program to meet its requirements (e.g. holding meetings, setting up purpose-built internal committees): This might refer to the degree of importance given to accreditation programs by HCOs and could be measured through identifying those efforts of the HCOs which are directed toward these requirements (e.g. during the preparation phase)

  • Recommendations offered to HCOs by accreditation programs for improvement: The percentage of initially failed HCOs which are subsequently successful as a result of the guidance of (mandatory) accreditation programs

Outcomes of accreditation programsThe substantive outcomes of accreditation programs for HCOs over time (pre/post-accreditation) or in terms of accredited/non-accredited HCOs:
  1. Improved patients' care

  2. Standardization of care processes in HCOs

  3. Enhanced patient and staff satisfaction

  4. Greater safety for patients and staff

  5. Improved financial performance of HCOs

  6. Improved inter-/intra-HCOs' communication and information flows

  7. Professional development (learning outcomes of the accreditation programs for the hospitals' staff)

  8. Coherent organizational culture (encouraging same norms and language of communication and shared knowledge in HCOs)

Minimal unintended and dysfunctional effects
  • HCOs undergo a low rate of adverse effects from/through the process of accreditation (e.g. tunnel vision, stress and tension)

  • HCOs are not driven towards dysfunctions (e.g. gaming) because of accreditation programs

Indirect benefits of accreditation programs for HCOs
  • Financial benefits for HCOs (increased tariffs for services)

  • Legitimacy for the HCOs, granted by accreditation programs in the form of ranking, which might be construed in two ways (for a HCO):

    • As a permission to continue its activity in its current state;

    • As an element of reputation

  • Encouragement of HCOs to imitate their higher ranked similar organizations

At the end of each stage, results were discussed and agreed upon by the authors, in order to minimize the judgment bias and maintain the consistency throughout the analysis process. Adopting Thematic Content Analysis, the researchers proceeded from categorizing and coding the data to reflect on how the codes associate with each other, leading to a more profound understanding of the issue.

Results

The findings of the study could be categorized under three main headings based on the relevance of the emerged themes to the different aspects of accreditation programs.

  1. Societal impact.

  2. Methodology (i.e. survey and standard).

  3. Organizational impact.

These general concepts, elaborated in Tables 13, underline a broad range of ‘highly specified, quantitative measures, principles and guidelines’ in connection with the ‘performance measurement and improvement’ of accreditation programs in health sector. The tables both illustrate the measures and provide an explanation as to their nature and the way of their measurement, where required. The majority of the measures put forth have a conceptual nature, yet very few hard measures are also identified, such as surveyor training days and report turnaround time.

Discussion

The conceptual measures displayed in the tables are de facto the result of synthesizing the perspectives of various accreditation-concerned groups, geared to evaluating the merits and worth of accreditation schemes in healthcare. They are oriented toward a generic framework of accreditation, irrespective of their voluntary/mandatory fashion. However, those with obvious relevance to either of these modes are also emphasized.

The underlying intention of the study is to identify general measures for assessing the performance of the accreditation programs. Therefore, its original focus is mostly on the common grounds in the data gathered from the different respondents. As such, the strictly context-related measures (i.e. those highly associated with a specific accreditation programs, particularly in relation to the professionals' views) were removed from the indicators. The principal rationale for considering and mixing the various groups' perspectives is to include the most possibly diverse range of relevant views on the performance of the programs and render an integrated assortment of the measures.

Some measures exclusively refer to accreditation programs' nature and operations without carrying any positive/negative value. For example, there is a debate about the virtue of ‘disclosing accreditation results to public’ in the literature [60]. Hence, these measures might be considered either as a weakness or strength for these programs. Nevertheless, most of the measures are stated positively in that their exact opposite could be also assumed as relevant in investigating the dysfunctional effects of an accreditation program. The following section will briefly provide a discussion of the main points presented in the tables.

Societal impact of accreditation programs

Table 1 demonstrates the measures which reflect the impact of accreditation programs at a macro and societal level. Accreditation has undeniably been an influential mechanism for protecting society in the sense of safeguarding pubic access to quality and safe healthcare [2]. The accreditation programs play a key role in monitoring the reflection of healthcare values (i.e. quality and excellence) by HCOs. Such tasks are argued to be undertaken by ‘societal steering mechanisms’ from a macro perspective [61, p. 13]. Based on this reasoning, accreditation could be conceived as a steering mechanism in healthcare with its respective impact at the societal level. The measures presented in this table are expected to reflect societal aspects of the programs' performance. For instance, ‘society's demand for a voluntary accreditation program’ could be a sign of performance excellence and success for these programs. As such, the ‘retention rate of HCOs by a voluntary program’ accentuates the beneficial, or otherwise, consequences of accreditation programs when compared with other quality assessment and improvement systems in a society. Efforts to operationalize public involvement in the governance of these systems provided greater objectivity and credibility to the accreditation process [62]. Moreover, it has been argued that the provision of information to the public about HCOs' performance, given the current ease of access to the information sources, has become a key factor for successful accreditation programs [63]. Regardless of the possible consequences such as the misinterpretation of accreditation reports or overreaction to negative data, the existing evidence backs the positive effects of publicly releasing the accreditation results on HCOs' evaluation results, besides enhancing public accountability and quality of care [60]. The discernability of the accreditation rating could be advantageous for the public to easily identify the well-performed HCOs [64].

Different stakeholders may rely on accreditation results in their decision-making. A growing number of state-run accreditation programs could allude to the increased reliance of governments on accreditation results to safeguard the public access to quality healthcare [14]. Moreover, the introduction of a ‘choose and book’ procedure by some advanced health systems (e.g. NHS) warrants patients to think of specific criteria against which they would select their desired provider [65]. The information provided through accreditation could serve as such criteria. By the same token, the transparency of accreditation programs' various stages for accreditees and the responsiveness of the programs for their decisions could turn them into an evidence-based program [35].

Flexibility of an accreditation program to adapt to the changes in its environment and accommodate the feedback of different stakeholders could ensure its sustainability and relevance. This openness to changes could transfer the programs into a learning organization which always incorporates feedback in its development process and stays up-to-date.

Methodology of accreditation programs

Table 2 provides related ‘performance measures’ for judging the functionality of accreditation. Surveyors and standards are two main elements of any accreditation scheme [66]. Surveyors are envisioned as the ‘eyes, ears and hands’ of any accrediting organization, without which the accreditation process is argued to be unsustainable [67, p. 1]. Therefore, as such, a detailed training program for new surveyors along with setting surveyor selection criteria are claimed to enhance the reliability of accreditation [68]. As Pickering [69] indicates, the training is essential in guiding the surveyors to detect the gaming in the hospitals.

Standards are similarly a key element for accreditation, against which HCOs' performance is assessed. The choice of standards, their focus and the level at which they are set are crucial in determining the tone, acceptability and nature of the accreditation programs [49]. Therefore, it is important that the standards are concomitantly reviewed and kept aligned with advances in healthcare and relevant to the services or organizations under their evaluation. There are various dimensions that should also be taken into consideration while evaluating accreditation standards. For instance, ‘the rate of clarity and feasibility of standards for HCOs’ implies that standards should be ‘understandable’ at first sight by those who perform accreditation (i.e. surveyors) and are accredited (i.e. HCOs). The Accreditation Canada institution believes in optimal, but achievable (within the current state of the art) and surveyable standards within the confines of resource constraints [66]. Application of a ‘consensual process’ for developing the accreditation standards was also recommended by the experts. Incorporation of ‘stakeholders’ voice' in different stages of accreditation is highly stressed [63] and is receiving growing attention among accreditation agencies [62].

‘Outcomes performance measures’ are increasingly important in healthcare. Lack of evidence supporting the assumption that appropriately organized inputs could certainly lead to desired outcomes in healthcare has accelerated the movement towards using outcome indicators in evaluating hospitals [38, 70]. de Walcque et al. [5] refer to ‘outcome measures’ as the determinant of the ultimate impact of an accreditation program. However, given the intangible nature of these indicators, process and structure measures might be preferred [70]. Despite the importance of outcomes, achieving quality outcomes through poor structures and processes might also appear unlikely, as could be the case in developing countries [14].

‘Inclusion of clinical indicators’ in the accreditation standards could increase the clinician's involvement, which is vital for successful introduction and implementation of accreditation programs [71], in different stages of the accreditation process [41]. This is noticed and operationalized by various programs. For instance, in the USA, since 1997, JCAHO has linked clinical outcomes indicators to accreditation process through ORYX-initiative integrating outcomes and other performance measurement data into the accreditation process [72]. ACHS has developed the performance and outcome service to increase the clinical components and indicators in its new accreditation program [73, 74].

The existence of a regular review and update system for the entire accreditation process, specifically the standards, was widely reflected by the respondents. In JCAHO, standards are reviewed every year for hospitals and every 2 years for other HCOs, and Accreditation Canada reviews its standards every 2 years. The self-evaluation system for accreditation programs ensures the relevance of their standards to the newly emerging activities of HCOs. It further helps assimilate the advancements in the structure and development of accreditation programs.

Emphasis on ‘documenting’ by HCOs in accreditation standards could be a key pointer to ‘evidence-based’ evaluation of HCOs. ‘Documentation requirements’ for HCOs by these programs are a key element indicative of ‘reliability’ of accreditation processes [68, p. 112].

Organizational impact of accreditation programs

The impact of accreditation programs on HCOs at the micro level is explained in Table 3. It comprises all forms of immediate (outputs), long-lasting (outcomes), indirect and dysfunctional effects of the programs on HCOs. Enhanced compliance of HCOs with the requirements of accreditation programs is the most tangible sign of their effectiveness [30]. Comparing this reaction (with the exclusion of symbolic compliance) over a specific period or different hospitals might signify that the HCOs have improved their ability to meet the standards of the programs (i.e. the boosted capabilities of HCOs for achieving accreditation standards). This might be referred to as the ‘capacity development’ role of accreditation programs.

Tangible actions and efforts of HCOs (e.g. holding meetings and setting up a specific internal committees) to meet the programs' requirements could signal the importance given by HCOs to satisfying the requirements of accreditation programs. These actions de facto orchestrate the circumstances for further and prolonged improvements in the HCOs. Recommendations offered by accrediting groups for boosting quality in HCOs serve to fulfill ‘improvement and change’ intentions of these systems in organizations under evaluation [75, p. 161].

Despite these initial effects, real impact of accreditation should be seen through its capacity to lead to sustainable improvements in patient care quality and safety [24, 76], from two perspectives; within HCOs through pre- and post-accreditation stages and between accredited and non-accredited HCOs.

There have always been concerns over how to enhance the effectiveness and efficiency of evaluation and control mechanisms while maximizing their intended consequences on organizations. Policymakers are increasingly becoming worried about dysfunctional and unintended consequences of quality improvement initiatives [77]. Therefore, the merits of accreditation programs' could be also scrutinized for their unintended and dysfunctional effects on HCOs (e.g. the programs may cause physicians to focus mostly on measured care at the expense of unmeasured care).

Accreditation could be seen as a source of economic gain and legitimacy for HCOs. These organizations are permitted to work in some communities on the condition that they receive an accreditation award (i.e. legitimacy). Alternatively, they might be allowed to charge higher tariffs or have service contracts with third-party organizations based on the level of their ranking. In addition, these programs provide a chance for HCOs to identify and emulate those organizations which have received the highest ranking in accreditation.

Final considerations

With the paucity of research investigating performance evaluation of accreditation programs, this study brings together a number of generic, yet workable, measures which could be utilized for assessing the value and merits of an accreditation scheme in healthcare [cf. 78, 79]. The measures are somewhat in the form of general pathways laying the groundwork for further examination of the programs' performance and effectiveness, which could be considered as a departure point for future research on accreditation.

This study can be conceived as unique and novel for its direct approach to identifying and suggesting performance measures for evaluating accreditation programs in healthcare. Some measures could be utilized as a decision support by authorities and policymakers to appraise their programs over a specific period. Alternatively, some might be also used to guide the initiation of a new program in a country. Data from both academic and professional contexts were included, in order to yield more comprehensive and plausible results and reflect diverse and complementary perspectives and experiences on accreditation. In terms of the application of the measures developed, the features of given accreditation program and the context in which the program operates should be considered. First step/steps in the assessment of an accreditation program, for example, could be identifying relevant measures suiting that specific program and context.

Along with the modest effort of this study to contribute to the scant knowledge of performance measurement and improvement of accreditation in healthcare, the following limitations should be also recognized for this research. First, although we conducted a rather comprehensive search for experts, it is always possible for some to be missed, particularly with our snowball sampling. Second, the measures are the result of a qualitative and perceptual process and, therefore, they are on the verge of being judgmental and not generalizable in quantitative sense. Third, selecting the group of professionals from only one country is a limiting factor in comparison with other two extended groups. Such a selection was due, on one hand, to the intention of the study to form a diverse participant group and, on the other hand, to the data access restraints, as explained earlier. Fourth, despite the fact that a generic form of the accreditation was conceptualized for the respondents, an intrinsic weakness in seeking opinions from three sources, which may have very different orientations and experience of accreditation, remains. Since the data from the various groups were not intended to be compared, but combined, the effect of contextual differences could be arguably alleviated. Finally, the study is not claimed to be all-encompassing in its coverage of measures related to the performance of accreditation programs and other indicators might be explored. Accordingly, it is essential that further and persistent empirical research is performed in an attempt to seek confirmatory signs and build additional understanding of these performance measures.

Funding

This study has been financially supported by the Iranian Ministry of Health and Medical Education (MOHME).

Acknowledgments

The authors would like to acknowledge the Iranian Ministry of Health and Medical Education and Tehran University of Medical Sciences by which this study was supported. Special thanks should also go to two anonymous reviewers who provided valuable comments on this paper.

Appendix

Embedded Image

References

View Abstract