OUP user menu

Proposed standards for the design and conduct of a national clinical audit or quality improvement study

Nancy Dixon
DOI: http://dx.doi.org/10.1093/intqhc/mzt037 357-365 First published online: 21 May 2013

Abstract

Purpose The purposes were to find and synthesize available literature on explicit or implicit standards for the design and conduct of a national activity that involves measuring and facilitating improvement of the quality of patient care, such as a national clinical audit or a quality improvement (QI) study, and to develop proposed standards for the design and conduct of the national activity.

Data sources, selection and analysis The literature was searched to identify key aspects of good practice in the conduct of national or international clinical audits, QI studies, performance or quality indicator measurements or equivalent national initiatives that have the purpose of driving improvement in the quality of care provided in a healthcare system. Key aspects of good practice in design or operation of these activities were abstracted from the literature, and organized logically into standard statements according to the stages in the design or conduct of such an activity.

Results Thirty standards for the design and conduct of a national clinical audit or QI study were derived from the published literature. The standards are on structural, process and outcome aspects of any national activity that involves measuring and improving healthcare services. Most of the standards focus on measurement processes.

Conclusion It is hoped that these proposed standards for a national clinical audit or QI study will facilitate debate on how to assure the quality of these national activities. Activities that meet accepted standards may be more effective in influencing participating sites to achieve improvements in the quality of care.

  • clinical audit
  • national clinical audit
  • national quality improvement study
  • national quality indicator monitoring
  • national performance indicator measurement

Introduction

Clinical audit has been defined as a quality improvement (QI) process intended to measure and improve the care and outcomes patients experience [1]. Clinical audits are normally carried out by a group of peers working in the same healthcare service or organization. National groups in the UK have carried out national clinical audits since 1994 [2]. In England in 2012, there were as many as 70 national clinical audits being conducted; National Health Service (NHS) organizations are mandated to participate in 29 of these audits [3]. In other countries in which clinical audit does not have a high profile, equivalent activities, such as international and national quality indicator projects, QI studies, performance measurement or monitoring projects or registries, have been carried out for several years.

Some individual countries have set out guidance for the design of registries [4, 5]. The SQUIRE (Standards of Quality Improvement Reporting Excellence) guidelines [6] were developed in recognition of the variation in content and quality of QI work in health care. The guidelines describe what should be included in published reports on QI studies, and good practice for carrying out a QI project is embedded in the guidelines.

However, guidelines or standards against which national clinical audits or QI studies can be assessed and assured have not been available internationally. In the absence of agreement on best practice for these activities, variation in their potential effectiveness and cost–benefit is inevitable. For example, if staff in participating sites are unable to reliably collect the data required or do not receive timely reports comparing their performance with equivalent sites, they may be less likely to make changes in their clinical practice based on the findings.

Purpose

The purposes of this work were to find and synthesize available literature on explicit or implicit standards for the design and conduct of a national activity that involves measuring and facilitating the improvement of the quality of patient care, such as a national clinical audit or a QI study, and to develop proposed standards for the design and conduct of such a national activity.

Method

Search strategy

For purposes of identifying publications describing good practice relating to the design and conduct of a national clinical audit or QI study, any activity that involves the measurement of an aspect of the quality of patient care, including clinical processes or outcomes, was included. The types of activities involved included national clinical audits, national outcome studies, national performance indicator monitoring, national or international quality indicators measurement, national QI studies and registries.

Data sources and selection

The following sources were used: Medline and CINAHL databases using PubMed, Ovid and EBSCOHost as search engines; websites of organizations known to carry out or report on national clinical audits or QI studies; journals known to publish national or international studies related to measuring and improving the quality; professional organizations known to be concerned with measuring and improving the quality of healthcare services; and Google and Yahoo search engines.

In combination with the range of activities specified, the terms used for searches included: criteria or standards or indicators or measures with clinical audit or QI or outcomes; performance or quality indicators or measures; scope of work or statement of work and national quality indicators and registry. Where limits could be set, the years searched were limited to 2000 to 2011. Reference lists in publications retrieved or on websites searched were scanned for additional relevant sources.

Limitations of the search and the literature

The literature on the conduct of national studies of the quality of care is not uniform or systematic in the description of methodology used in carrying out the work. Much of the published material about how to design and carry out a clinical audit or QI study is written for clinicians carrying out their own projects in individual clinical services in single healthcare organizations. This material was scanned and included only if a point made was thought by the reviewers to be relevant to the conduct of a national project. Descriptions of national clinical audits or QI studies or performance indicator projects were not included unless they included details relating to quality of the performance of the project.

Data extraction, analysis and synthesis

Documents that described a performance expectation on any aspect of the design or operation of a national audit or equivalent activity were identified and screened. One reviewer screened documents retrieved and abstracted information from relevant documents and a second reviewer validated the screening of documents and abstracts. Key themes among the performance expectations were identified. Standard statements were derived from references that related to each theme. The themes and standard statements were arranged in a logical sequence in accordance with the stages in designing and conducting a national project.

Results: proposed standards for a national clinical audit

A summary of proposed standards for a national clinical audit or equivalent activity derived from available literature is in Table 1. The standards cover structure, process and outcome aspects of such a project.

View this table:
Table 1

Standards for a national clinical audit or equivalent activity

No.TopicStandard
Structural aspects
 1Ethical basisThe subject of a national clinical audit has an ethically based justification for the conduct of the audit as a means of achieving improvement in the quality of care provided to patients
 2GovernanceThere is continuous and effective governance of a national clinical audit by a group that assumes accountability for the effectiveness of the audit
 3Stakeholder involvementAll key stakeholders in a national clinical audit are identified and actively involved in all aspects of the design and conduct of the audit. Key stakeholders include patients or service users or their representatives
 4ResourcesResources allocated to a national clinical audit at a national level are appropriate for the valid and reliable measurement and proactive facilitation of improvement of the quality of patient care for the subject of the audit
 5Roles and responsibilities and project planRoles and responsibilities for the design and conduct of the national clinical audit at national and participating site levels are comprehensively described and there is a published project plan for the execution of all aspects of the audit
Process aspects
 6RecruitmentThe national clinical audit is continuously proactive in recruiting the participation of all sites intended for inclusion
 7Improvement-driven aims and objectivesThe stated aims and objectives of a national clinical audit reflect an intention to drive improvements in the quality of patient care
 8Population or sampleThe population or sample selected for a national clinical audit is suitable for drawing valid conclusions in relation to the stated objectives of the audit and for minimizing bias in the findings
 9Data collection strategyThe strategy selected for collecting data for a national clinical audit is suitable for the stated objectives of the audit
 10Quality-of-care measuresQuality-of-care measures used in a national clinical audit meet the criteria for such measures that are established by the governance board and are capable of producing valid and reliable data about the quality of care
 11Data elements and data sourcesData specified for the collection for a national clinical audit permit judging compliance with the quality-of-care measures in accordance with the objectives of the audit, using currently available data to the extent possible to avoid an unreasonable data collection burden on participating sites
 12Data collection and handling protocol or manualA comprehensive protocol describes in detail all aspects of data collection and handling for a national clinical audit and is accessible by all stakeholders and all participants in the audit
 13Protection of patient identitySystems and processes are designed and fully implemented to protect the identity of patients whose care is assessed in a national clinical audit consistent with applicable national laws and regulations
 14Identification and handling of ethical issuesAny ethics-related issues in a national clinical audit are recognized quickly and acted on appropriately
 15Training and support for data collectorsData collectors for a national clinical audit are trained to follow the data collection protocol and are supported to supply complete, accurate, reliable and timely data
 16Pilot testingThe design, the data collection process, protocol and tools to be used in a national clinical audit are pilot tested before all intended sites are invited to participate and action is taken on the lessons learned from the pilot test
 17Reliability testingThe reliability of the data collected for a national clinical audit is tested and the findings are included in reports
 18Data linkagesIf a national clinical audit relies on linkages with electronic records or databases that contain information about individual patients, systems are implemented to ensure the security of the linkages
 19Data quality management, analysis and assuranceData submitted for a national clinical audit are managed, analysed appropriately and quality assured. Findings related to data quality and their implications are reported
 20Preliminary data and peer reviewPreliminary data are available on a timely basis to participating sites to enable a peer review and feedback process in participating sites before findings are published. The local peer review includes analysis of findings and any cases not consistent with good practice to identify and feed back to the national organization any clinically acceptable exceptions not previously acknowledged in data collection
 21Identification of good practice and shortcomings in the quality of careAreas of good practice and shortcomings between what is done and what should be done in the delivery of care are identified and reported
 22Analysis of causes of shortcomings in the qualityParticipating sites are supported in identifying root causes of any shortcomings in the quality of care
 23Facilitation of improvementsA strategy is adopted and implemented for proactively facilitating the implementation of improvements in the quality
 24ReportsComplete, timely and useful reports are provided for all stakeholders
 25CommunicationCommunication with all stakeholders is maintained throughout all stages of the clinical audit
Outcome aspects
 26Level of participationA target is set and met for the number of eligible organizations and services that participate in a national clinical audit
 27Reliability of dataReliability testing or independent validation of data collection demonstrates that the reliability of data collected for the clinical audit meets or exceeds an established target
 28Timeliness of reports on preliminary dataReports of preliminary data collected are supplied to participating sites promptly following the completion of data submission, that is, within a short number of weeks of the deadline for submission of the data
 29Timeliness of complete reportsReports of final data collected are provided promptly, that is, within a short number of weeks of the deadline for submission by participating sites of any additional data following review of preliminary findings and cases that did not appear to meet the quality-of-care measures
 30Evidence of improvements in the quality of careImprovements in the quality of care from one time period, for example, 1 year, to the next of a national clinical audit are sufficient to justify continuation of the audit

Summary of literature supporting the proposed standards

Structural aspects

Ethical basis

Benefits and risks to patients in the current healthcare delivery system and whether or not the proposed audit responds to the risk–benefit analysis should be determined [7, 8]. A range of considerations can be included in the risk–benefit analysis [9, 10], including the incidence or prevalence of the condition that is the subject of the work; the impact the condition has for patients, families and the health and social care systems; availability of evidence that current quality of care is not consistent with best practice or there is significant variation in practice among healthcare organizations or evidence that meaningful improvements in the quality of care can be achieved.

Governance

Governance should provide for transparency to all stakeholders in the audit on its operation, accountability for decision-making and reporting of results [4, 5]. A named governance board for a national clinical audit should include representatives of all key stakeholders in the audit including patients or their immediate representatives [4] and representatives of participating sites.

Stakeholder involvement

All relevant stakeholders, including patients and their representatives, should be involved from the beginning of the clinical audit through to the proactive facilitation of improvements [9, 11].

Resources

The resources needed to operate an audit at a national level and in participating sites should be considered carefully when the audit is designed [4, 5]. The most efficient methods that are likely to be effective in meeting the audit objectives should be used. Population-based audits are not justified when extensive time is required for data collection and validation, particularly when appropriate statistically determined samples could be used. Resources allocated should consider the need to facilitate changes in practice to improve the quality of patient care in participating sites.

Roles and responsibilities and project plan

The roles and responsibilities of everyone involved in an audit at national and site levels should be defined comprehensively [5], including demonstration of commitment to and leadership of participation in the audit and implementation of improvements in care [9]. The project plan should include timelines for the national organization and participating sites and the published timelines should be met routinely [11].

Process aspects

Recruitment

Goals and processes for recruitment and retention of participating sites should be described clearly and their effectiveness assessed periodically. Variations between intended and actual levels of recruitment and participation should be continuously evaluated for their risk of introducing bias in the audit findings [4]. Incentives for participation [4, 12] and reasons for non-participation and non-continuation in the audit should be identified and acted on by the governance board. The recruitment process should provide for early and full disclosure of the requirements for participation to potential participating sites, including access to the complete project plan, defined roles and responsibilities, data collection tools to be used and a realistic estimate of resources needed for participation.

Improvement-driven aims and objectives

The stated aims and objectives of a national clinical audit should be explicit about the intentions to have participating sites use the audit to drive improvements in the quality of care and the organization operating the audit to lead and support the implementation of needed improvements in the quality. The design and conduct of the audit should use several strategies that are likely to facilitate improvements in the quality of care in participating sites [13].

Population or sample

The population or sample should be clearly specified [1315] and selected to approximate the relevant characteristics of the target population as much as possible [4]. The desired number of cases should consider the magnitude of an expected, clinically meaningful effect; practicality; cost; and the intention to show the effects of interventions to achieve improvement from one data collection period to another [4]. Also, the population or sample, along with the data collection strategy and data sources should be selected to enhance representativeness and minimize potential sources of bias [4], and if consistent with the audit's objectives, enable international comparisons. The number of cases in the population or sample should not place an unnecessary burden of data collection on participating sites [13].

Data collection strategy

The data collection strategy should be suitable for the audit objectives and the availability and accessibility of data needed, and should not put an undue burden on those supplying data.

Quality-of-care measures

Measures enable numeric quantification, usually reported as percentages or ratios, of healthcare quality for the subject and objectives of a clinical audit [16]. The governance board for the national audit should establish criteria for the selection of quality-of-care measures to be used in the audit, such as: [1623] evidence based [5, 17, 24] or agreed by consensus in the absence of valid evidence [2529]; scientifically acceptable, for example, that the measures have been tested and demonstrated to produce valid and reliable information about the quality of care [17, 25, 3033]; important to measure and report; usable and relevant; feasible to collect; efficient and avoiding risks to patients [17]. The validity [32, 3439] and sensitivity or positive predictive value (PPV) of the measures should be determined [4047]. Cases identified as ‘false positives’ should be reviewed individually to identify how the measures need to be improved [4047].

Data elements and data sources

Data to be collected should be derived from the quality-of-care measures, be carefully defined [32, 48, 49] and permit judging compliance with quality-of-care measures in accordance with the audit objectives, using currently available data to the extent possible to avoid an undue burden on participating sites [5, 9, 13]. Data elements should be able to be used to support QI efforts at local level [13]. The collection of ‘nice to know’ or optional data should be kept to a minimum [4]. The reliability of any data retrieved from secondary sources should be tested [32, 50]. Measurement of patient-reported outcomes or patient experiences should use tools that have been designed and validated for this purpose [5].

Data collection protocol or manual

All aspects of data collection, collation, analysis and control for a national audit should be described in a protocol or manual [5, 9, 25]. The protocol or manual should be easily accessible to all stakeholders in an audit, particularly to healthcare organizations prior to their need to register for participation in the audit.

Protection of patient identity

The governance board of a national clinical audit must assure that the systems designed to protect the identity of patients whose care is assessed in the audit are robust and consistent with applicable national laws and regulations, and that they are continuously implemented in all sites participating in the audit.

Identification and handling of ethical issues

Provision should be made to handle ethical issues that emerge at each of these stages of a national audit [8, 51, 52]: the design of the audit; when the findings are available; and when findings from repeat data collection are available. The governance board should assume responsibility for decisions made on the handling of any ethical issues in the audit [8].

Training and support for data collectors

Training on carrying out data collection and submission for a national audit should be available for all clinical or non-clinical staff collecting and submitting data [4, 5, 13, 25, 32, 5355]. Training manuals and live support should be easily accessible [4]. Feedback from data collectors about the value of the training, materials and support provided should be regularly sought and the findings acted on.

Pilot testing

The design, data collection process, protocol and tools to be used in a national audit should be thoroughly pilot tested in several sites eligible for participation in the audit before the audit involves all intended sites [4, 25, 32, 33, 53, 56]. Action should be taken on the lessons learned from the pilot test.

Reliability testing

Data collected for a national clinical audit should be tested for reliability, sometimes referred to as re-abstracting data [25, 32, 54, 55, 5762], independently by the national clinical audit [5, 54, 55, 57, 58, 63]. The reliability of data should be reported routinely when the findings are reported [55, 57, 60].

Data linkages

When a national clinical audit uses linkages with electronic records or databases to capture patient care information, robust systems must be implemented to ensure the continuous security of the linkages [5, 9].

Data quality management, analysis and assurance

The quality of data submitted for a national audit should be monitored and assured [5], preferably using systems that use front-end logic to check data quality [4, 5, 11, 13]. Findings of quality assurance checks should be reported to the governance board, used to make changes in the conduct of the audit as needed [4], and included in reports.

Preliminary data and peer review of cases

Participating sites should have access to and review their preliminary data and cases not consistent with good practice, through a local peer group process [5, 54], and prepare for implementing any needed improvements [64, 65]. Sites should be able to correct their data in the national audit prior to publication, if evidence of error or inappropriate judgement during data collection is supplied to the national audit.

Identification of good practice and shortcomings in the quality of patient care

A national audit should identify areas of good practice as demonstrated by compliance with the quality-of-care measures used. Gaps or shortcomings between the actual quality of care and best practice as represented by the measures also should be explicitly identified [5, 6668]. Participating sites should easily be able to determine their levels of performance in comparison with other comparable sites.

Analyses of causes of shortcomings in quality

A national clinical audit should help participating sites to identify and understand the causes of their shortcomings in care through providing examples of analysis of variation [69, 70], root cause analysis of problems revealed and case studies [31, 71].

Facilitation of improvements

A national clinical audit should adopt and implement a framework for spread of good practice [72, 73], identify what works best among improvement initiatives and encourage the rapid adoption of those initiatives [13, 54, 55, 74]. The national audit should create and market practical tools to help healthcare organizations improve their performance [13, 28, 75]. Examples of improved performance should be disseminated through professional meetings and awards recognizing substantially improved practice [13, 54, 55, 76].

Timely, complete and informative reports

Reports should be timely and reflect the SQUIRE guidelines to the extent feasible [6]. They should be designed to help sites quickly and easily identify their areas of needed improvement in relation to their peers [13]. Reports on national clinical audits should be provided for groups of identified stakeholders, including patients [9].

Communication among all stakeholders

Communication with all stakeholders should be maintained throughout every stage of the clinical audit [9], particularly with management, clinicians and data collectors in participating sites.

Outcome aspects

Level of participation

A target level of participation in a national clinical audit by eligible sites should be established by the governance board, which should publish the participation rate in comparison with the target.

Reliability of data

A target level of reliability of data collected should be established by the governance board, which should publish the findings of reliability testing of data collected. Reliability testing or independent validation of data collection should demonstrate that the reliability of data collected for the clinical audit is at least 90% or an equivalent kappa value [55, 77].

Timeliness of reports on preliminary data and complete reports

Reports of preliminary data collected should be supplied to participating sites for local review in real time or within weeks following the submission of the data. Complete reports of national clinical audits should be supplied to participating sites and other stakeholders within weeks of the review of preliminary data by participating sites.

Evidence of improvements in quality

The governance board should establish and apply measures for judging the effectiveness of the audit. Depending on the audit findings, the board should determine how data are being used to drive QI or make changes in the audit to influence the implementation of improvements needed in participating sites.

Discussion

The Cochrane systematic review on audit and feedback concluded that ‘audit and feedback generally leads to small but potentially important improvements in professional practice. The effectiveness of audit and feedback seems to depend on baseline performance and how the feedback is provided’ [78]. This conclusion is based on a rigorous review of published trials of audit and feedback, which is defined as ‘a summary of clinical performance over a specified period of time’.

The review does not specifically address national clinical audits or QI studies. However, in the absence of standardization of the work carried out, particularly the quality of the processes used to measure the quality and provide information to participating sites on their own performance, it is unclear what conclusions can be drawn about the potential effectiveness of national projects that are intended to measure and improve the quality of health care. Given the substantial economic resources involved in these projects and the constraints on funding of healthcare systems around the world, it is essential that these activities provide optimal cost–benefit to healthcare systems. Their potential value should be judged only when they are carried out to the highest standard.

The proposed standards should be achievable by organizations carrying out national clinical audits or QI studies because they represent current practice as described in the relevant literature. The biggest issue is that potentially not every national audit or QI activity complies with every standard. A major shortcoming is likely to be the informality of many of the processes involved, such as training data collectors, and the lack of full documentation of many of the measurement processes such as a complete data collection and quality control manual and a realistic project plan for both the national organization operating the activity and participating sites.

Conclusion

National clinical audits or QI studies have been carried out in the UK and other countries for over 15 years. In the absence of consensus agreement on best practice in the design and conduct of these activities, the audits or QI projects can vary in their effectiveness and cost–benefit, particularly at participating site level. Synthesis of available literature indicates that there are many aspects of the design and operation of national clinical audits or QI studies for which good practice has been defined and some published examples of standards for the design and operation of an audit or QI study being achieved. A set of 30 standards for the design and conduct of a national clinical audit or equivalent activity is proposed, based on published literature and examples of best practice. It is hoped that the standards will be debated among stakeholders in national clinical audits and equivalent activities as the basis for undertaking these projects in the future.

Funding

The work was supported in entirety by Healthcare Quality Quest Ltd, prompted by a request from Parkinson's UK to evaluate the Parkinson's national clinical audit.

Acknowledgements

Mary Pearce of Healthcare Quality Quest carried out the searches of literature for this work and validated the abstracting of the evidence.

References

View Abstract