International Journal for Quality in Health Care Advance Access originally published online on April 10, 2008
International Journal for Quality in Health Care 2008 20(4):297-303; doi:10.1093/intqhc/mzn012
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Developing clinical indicators for the secondary health system in India
1 School of Health Systems Studies, Tata Institute of Social Sciences, Deonar, Mumbai, India
2 Directorate of Health Services, Government of Maharashtra, India
3 Center for Global Health Research, St. John's Research Institute (SJRI), Koramangala, Bangalore 560034, India
* Address reprint requests to: Harshad Thakur. E-mail: harshad{at}tiss.edu; harshad_thakur2000{at}yahoo.co.in
 |
Abstract |
|---|
 Top Abstract Quality problem or issueInitial assessmentChoice of solutionImplementationEvaluation and lessons learnedFundingReferences |
|---|
Quality problem or issue. One of the prime goals of any health system is to deliver good and competent quality of healthcare. Through World Bank-assisted Maharashtra Health Systems Development Project, Government of Maharashtra in India developed and implemented clinical indicators to improve quality.
Initial assessment. During this, clinical areas eligible for monitoring quality of care and roles of health staff working at various levels were identified.
Choice of solution. Brainstorming discussion sessions were conducted to refine list of potential clinical indicators and to identify implementation problems.
Implementation. It was implemented in four stages. (a) Self-explanatory tool of record, standard operating procedures and training manual were prepared during tools preparation stage. (b) Pilot implementation was done to monitor the usefulness of indicators, document the experiences and standardize the system accordingly. (c) The final selection of indicators was done taking into consideration points like data reliability, indicator usefulness etc. For final implementation, 15 indicators for district and 6 indicators for rural hospitals were selected. (d) Transfer of skills was done through training of various hospital functionaries.
Evaluation and lessons learned. Selection and prioritization of clinical indicators is the most crucial part. Active participation of local employees is essential for sustainability of the scheme. It is also important to ensure that data recorded/reported is both reliable and valid, to conduct monthly review of the scheme at various levels and to link it with the quality improvement programme.
Keywords: development, healthcare, indicators, quality
|
Quality problem or issue |
|---|
|
TopAbstractQuality problem or issueInitial assessmentChoice of solutionImplementationEvaluation and lessons learnedFundingReferences |
|---|
One of the prime goals of any health system is to deliver good and competent quality of healthcare. Quality of healthcare is a multi-dimensional concept encompassing accessibility, accountability, acceptability (to both users and providers), effectiveness, efficiency and safety. Monitoring healthcare quality is impossible without the use of indicators. They create the basis for accountability, quality improvement, prioritization and transparency in the healthcare system [1]. The challenge is to develop and implement quality indicators that uncover as much as possible of true quality [2].
A clinical indicator is defined as a quantitative measure of an aspect of patient care. It is not a direct measure of quality. It is a flag that indicates areas for more detailed analysis. It is a measurement tool used as a guide to monitor, evaluate and improve the quality [3, 4]. They provide a basis for clinicians, organizations and planners aiming to achieve improvement in care and the processes by which patient care are provided [5]. They allow individual providers to monitor and improve their quality of care and also provide comparative information across all providers [6].
Before assessment can begin, there is a need to decide how quality is to be defined. Though we know much about assessing quality, much remains to be known [7]. Measuring performance will assist a health service in making systematic progress towards meeting its goals and objectives of providing services of a high quality [8].
Developing and reporting clinical indicators has become a widely accepted method of improving both quality and accountability in many countries. There is also a worldwide upsurge in the production and use of clinical indicators with a bewildering array of approaches [9–11]. Two particular challenges have emerged from the international experience, namely: producing technically robust and interpretable indicators; and embedding indicators in systems where this information is used to improve patient care [12].
Over the last few years, there are many examples of successful development of clinical indicators. For example, the International Quality Indicator Project® (IQIP) is the largest international data set of quality indicators in the USA. It provides quarterly feedback of comparative indicator data and support for effective use of these data within the participants' own quality improvement programmes [13]. The UK Quality Indicator Project® (UKQIP) is the largest component of the IQIP. The UKQIP began as a pilot project in the National Health Service public sector in 1991. It now includes about two-thirds of UK private sector acute hospitals [14].
In Europe, Scotland has led the way in the production and public release of clinical indicators. In 1992, the Clinical Outcomes Working Group was set up to produce comparative clinical outcome indicators for Scotland. Clinical indicators were first produced in 1993 and an indicator's report is published annually. The main purpose of producing these indicators is to supply information that local clinicians and managers can use to improve the quality of patient care [12].
In 2001, the Taiwan Healthcare Executive College developed a comprehensive performance assessment system called Taiwan Healthcare Indicator Series that included four categories of indicators, namely: outpatient, inpatient, emergency care and intensive care, and had 139 items in total [15].
Compared with above examples, in India, there was not a single example of development and implementation of clinical indicators in public hospitals. Considering this, as a part of the Maharashtra Health Systems Development Project in 2001, it was decided to develop quality standards along with guidelines for medical staff. This involved formulating new clinical indicators for assessment and improvement of the existing system. It proposed to measure the performance of the secondary referral healthcare institutions in Maharashtra State.
Maharashtra is one of the largest and developed states in India. It is administratively divided into 35 districts. The rising life expectancy, declining birth and death rates and comparatively low disease burden in Maharashtra could be attributed to effective functioning of existing health infrastructure. For the provision of public health services, the state is administratively divided into eight circles. Deputy Director of Health Services heads a circle, which covers three to five districts. For provision of secondary health care services, each district has district hospital, sub-district hospitals and the rural hospitals. While Civil Surgeon is the head of district hospital, Medical Superintendents looks after sub-district and rural hospitals.
Public Health Department, Government of Maharashtra, implemented a World Bank-assisted Maharashtra Health Systems Development Project (1998–2006) in order to improve and strengthen the secondary health system. Under this, 136 health facilities in 33 districts were provided with infrastructure and administrative inputs. The project assisted the government to improve efficiency in the allocation and use of health resources through policy and institutional development. It improved the performance of its health system by enhancing the quality, effectiveness and coverage levels.
Through the project on development and implementation of clinical indicators, for the first time in Maharashtra and also in India, the performance of the government healthcare institutions was monitored and evaluated by clinical indicators. It served to sensitize the end functionaries towards importance of quality improvement. The overall objective was to develop the system of quality improvement measures using limited resources judiciously. The steps in developing and implementing clinical indicators are available in literature and are still evolving [1, 16]. We studied and followed them but occasionally had to deviate depending upon the local requirements.
|
Initial assessment |
|---|
|
TopAbstractQuality problem or issueInitial assessmentChoice of solutionImplementationEvaluation and lessons learnedFundingReferences |
|---|
During initial assessment, roles of health staff working at various levels and clinical areas eligible for monitoring quality of care were identified. Three hospitals (one district, sub-district and rural hospital each) were selected from each of the eight circles. Thus, totally 24 hospitals were selected for initial assessment. During the field visits, meeting was conducted with staff followed by hospitals rounds for situation analysis. The clinicians were interviewed individually to document their views and suggestions regarding quality of care. This was followed by brainstorming session with all the clinicians and hospital authorities together to develop an inter-speciality consensus. This resulted in identification of comprehensive list of clinical indicators (88 indicators for district and 68 indicators for sub-district and rural hospitals), which had potential to be implemented in various hospitals.
|
Choice of solution |
|---|
|
TopAbstractQuality problem or issueInitial assessmentChoice of solutionImplementationEvaluation and lessons learnedFundingReferences |
|---|
Brainstorming discussion sessions were conducted at circle level to refine list of potential clinical indicators, to discuss the feasibility, monitoring and acceptability issues and to identify implementation problems. The Deputy Director Health Services (circle in-charge) along with Civil Surgeons and Clinical Resident Medical Officers of all district hospitals within that circle were involved. Districts and potential trainers from the project hospitals for pilot implementation were identified.
A need for separate set of indicators for district and rural hospitals, respectively, was realized. Rural hospital indicators can be implemented in both sub-district and rural hospitals. A provisional list of 32 indicators for district and 13 indicators for rural hospitals was short listed. Each indicator was analysed according to its relationship with quality, possible ways for successful implementation and potential issues in local monitoring.
The Clinical Resident Medical Officer at the district level was identified as the trainer during the training of trainees and implementation of the indicators. The concept of Quality Circle, which would consist of Assistant Director of Health Services, Civil Surgeons and Clinical Resident Medical Officers in each circle, was also envisaged to oversee and supervise the implementation of the project.
|
Implementation |
|---|
|
TopAbstractQuality problem or issueInitial assessmentChoice of solutionImplementationEvaluation and lessons learnedFundingReferences |
|---|
It was conducted in four stages: tools preparation, pilot implementation, final selection of indicators and transfer of skills.
Tools preparation
The existing recording and reporting system in the district and rural hospitals was extensively studied. Each indicator was reviewed regarding aspects like precise definitions of numerator and denominator, identifying people concerned with recording and reporting of the data, and quality issues that need to be reported for specific indicators.
Tool of record, standard operating procedures and training manual were prepared after reviewing each clinical indicator. The tool of record provided the ledgers and tables to record the data on a daily/monthly basis and then report to higher administrative levels. Standard operating procedures detailed the method of recording and reporting of each event in easily understood manner. The training manual served as a final instruction tool comprising the details of each indicator. A pre-pilot implementation was done to finalize the tools. Importance of training of the personnel and interpretation of indicator in the light of existing infrastructure was realized.
Pilot implementation
Here, the objective was to monitor the usefulness of indicators, document the experiences and standardize the system accordingly. The pilot implementation was done in two districts and two rural hospitals for 2 months. Two mid-terms and then one final assessment were done. The data collected was used for final selection of indicators.
The brief results of pilot implementation for finally selected indicators along with their definitions are presented in Table 1. It shows wide variation because the pilot implementation was done only for 2 months and that too only in two districts and rural hospitals each. The variation is even wider in district hospitals when compared with rural hospitals for indicator like call book response time because in district hospitals the patient load is quite heavy and they are big hospitals when compared with rural hospitals. For indicators such as perineal tears during delivery and surgical wound sepsis, the variation is wider in rural hospitals because of lesser patient load. Also in spite of training, there was some variation in interpretation of the definitions accounting for further variation in results.
|
Final selection of indicators
Each indicator was analysed using two scores: Data Reliability Score (average of the reliability scores calculated for each data element—separately for numerator and denominator) and Indicator Usefulness Score (usefulness of the indicator evaluated assuming the data is reliable). Table 2 presents attributes used for evaluation of data elements and indicators [4]. Each attribute was given a score on a scale of 1–5 where 1 is extremely bad and 5 extremely good. The scoring was done by the field investigators involved in the process.
|
On the basis of these scores, the indicators were grouped into four classes (A, B, C and D). In class A, both Indicator Usefulness Score and Data Reliability Score for the indicators are high (
3). In class B, Indicator Usefulness Score is high but Data Reliability Score is low (<3). Indicators in class C have a low Indicator Usefulness Score but high Data Reliability Score. In class D, both the scores are low. This implies that on the basis of the analysis of reliable data, indicators in class A were found to be useful. Hence, they were considered as the best candidates for final selection. The indicators were further evaluated taking following points into consideration in addition to data reliability and indicator usefulness: suggestions by World Bank officials, inputs received from the project management, interpretive value of the indicator and feasibility of actual implementation. Thus, 15 indicators for district hospitals and 6 indicators for rural hospitals were selected for final implementation. Table 3 presents list of indicators rejected for final implementation.
|
Transfer of skills
Effective training (transfer of skills) is the first step to ensure successful implementation of any programme. The objective was to conduct training of various hospital functionaries. Tool of record, standard operating procedures and training manual were finalized. Training guidelines were also developed with active involvement of the project management. The training manual brought forth the ideology and methodology of the project in a manner easily understood by the people who would be implementing it. It served as a guide and dissolved queries arising during the implementation.
The responsibility of implementing the project in hospitals within a district was given to Civil Surgeon, Clinical Resident Medical Officer and Matron of the District Hospital. The responsibility for the entire Circle was given to Assistant Director of Health Services of the Circle. They were also identified as key trainers. The training of key trainers was carried out in four batches. Totally 92 key trainers were trained in four batches. Participatory teaching skills and subject related skills were imparted, which enabled them to conduct training of the staff working under them, developed the skill to compile, analyse, interpret and report the findings and also developed the ability to take local level corrective measures for performance improvement. The trainees to be trained were all the medical officers, sister in-charges, staff nurses and Radiology Assistant within the district and rural hospitals.
|
Evaluation and lessons learned |
|---|
|
TopAbstractQuality problem or issueInitial assessmentChoice of solutionImplementationEvaluation and lessons learnedFundingReferences |
|---|
The entire project was completed in
20 months (started in December 2001 and completed in October 2003). We developed the system of clinical indicators and then handed it over to the project management for implementation. It was initially implemented only in project hospitals and later on over the entire secondary health system by the government. The quarterly review of district provides an opportunity to review the progress. The experience of development and implementation of clinical indicators on such a large scale was quite interesting and informative. Though the formal evaluation was not done by us, we learned many useful lessons during the period. Prioritizing among clinical areas for assessment may be based on various criteria, including the importance of the health care problem and the opportunity for interventions based on evidence of quality of care. In the Danish National Indicator Project, stroke and lung cancer were selected based on evidence-based clinical guidelines [1, 17]. Baylor Health Care System (BHCS), an integrated health care delivery organization in Dallas-Fort Worth, TX, USA, implemented a clinical indicator system focused on measures of health care underuse, overuse and misuse. These indicators demonstrated the accomplishments of specific process of care improvements throughout BHCS [18].
The Health Care Quality Indicators (HCQI) Project by Organization for Economic Co-operation and Development (OECD) was an initiative for the development and implementation of quality indicators at the international level. The participating countries and organizations selected five priority areas (cardiac care, diabetes, mental health, patient safety and primary care/prevention). The experience of the HCQI Project demonstrates that international consensus can be achieved in how to measure the quality of care in priority areas [19]. Conceptual framework of HCQI focuses on the quality of health care, maintains a broader perspective on health and its determinants, and recognizes the key aims of health policy [20]. The experience of the OECD Health Care Quality Indicators Project and the US National Healthcare Quality Report shows that conceptual frameworks should be established to guide the selection of indicators. Also choices should be made early on in the process to focus on a wide range of clinical conditions or to report on a few priority areas. Methods should be developed to add and subtract indicators while maintaining a stable set of indicators to track over time [21].
The training manual should be translated into local language for better understanding by staff. The good quality registers are important for durability. More manpower might be needed for record maintenance. The clerical staff specifically involved in the reporting system could also be trained. If possible, the clinical indicator reporting should be merged with the existing format. This will simplify the reporting system and streamline the workload at local, circle and state level. Web-based reporting of data by each hospital will reduce the time lag between reporting and feedback and eliminate calculation errors. BHCS experience demonstrates that despite implementing web-enabled error reporting systems, indicators of medication misuse still continue to be in a formative stage [18]. The local level managers should take more responsibility for implementation and analysis of project in their hospitals. They should analyse monthly report in terms of patient, personnel and institutional factors affecting performance and see the trend over the period.
Monitoring, benchmarking and revising of indicators should be done depending on the experience and feedback from the field. At least 6 months experience is required for rational benchmarking at state level. It is important to spot opportunities for improvement. There is also a need to organize regular refresher training programmes.
Local employees should be actively involved for smooth transition and sustaining the scheme over the long period. This ensures that data entry, compilation and analysis are error-free, reliable and valid. Regular monitoring and supervision over recording and reporting of data, introduction of incentives and disincentives, conducting monthly review of the scheme at local and circle levels and linking it with other quality improvement programme are few measures to make it sustainable. Resources should be allocated to communication strategies and how best to present data results to diverse audiences. It is essential to put in place mechanisms to maintain project momentum [21].
The Taiwan study shows that participating hospitals increased from 45 in 2001 to 227 in 2006 and constituted 50% of the total hospital population in Taiwan. The reporting rate averaged 77.7% in 2004. How the data are interpreted and how data interpretation can lead to quality improvement was the principal concern of participating hospitals. The Bureau of National Health Insurance of Taiwan proposed participation in the series as being one of the criteria to be reimbursed for quality [15].
The local participation of employees is crucial for successful implementation and sustainability of the programme. Even the objectives of the Australian Council on Healthcare Standards (ACHS) indicator program were to increase the involvement of clinicians in evaluation and quality improvement activities, and to facilitate the collection of national data on the processes and outcomes of patient care [22].
The ACHS clinical indicators are mostly not evidence-based, and they do not adequately represent the sub-specialties within the many disciplines. Also forced use of externally derived clinical indicators removes clinical ownership and makes their use for quality improvement less likely. Benchmarking against a standard can have the effect of encouraging complacency once the benchmark is reached, which is at variance with the continuing quality improvement ethos [23]. Even OECD experience shows that much additional work remains necessary before the project can supply policymakers and researchers with ongoing, comprehensive and reliable data on the quality of care [19].
Overall, the project in Maharashtra has been pathfinder and set a new paradigm of accountability in secondary health system. State Health Systems Development Projects of other states in India like West Bengal and Karnataka also tried the development of indicators, but the approach and process was not that one adopted by Maharashtra.
|
Funding |
|---|
|
TopAbstractQuality problem or issueInitial assessmentChoice of solutionImplementationEvaluation and lessons learnedFundingReferences |
|---|
This study was funded by the World Bank-assisted Maharashtra Health Systems Development Project (MHSDP) in the state of Maharashtra, India.
|
References |
|---|
|
TopAbstractQuality problem or issueInitial assessmentChoice of solutionImplementationEvaluation and lessons learnedFundingReferences |
|---|
- Mainz J. Developing evidence-based clinical indicators: a state of the art methods primer. Int J Qual Health Care (2003) 15:5–11.
[Free Full Text] - Mainz J. Editorial: Quality indicators: essential for quality improvement. Int J Qual Health Care (2004) 16:1–2.
[Free Full Text] - JCAHO (Joint Commission on Accreditation of Healthcare Organizations). Characteristics of clinical indicators. Qual Rev Bull (1989) 11:330–9.
- Canadian Council on Health Services Accreditation. A Guide to the Development and Use of Performance Indicators (1996) Ottawa: CCHSA.
- Mainz J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care (2003) 15:523–30.
[Abstract/Free Full Text] - Gibberd R, Hancock S, Howley P, et al. Using indicators to quantify the potential to improve the quality of health care. Int J Qual Health Care (2004) 16:37–43.[CrossRef]
- Donabedian A. The quality of care. How can it be assessed? J Am Med Assoc (1988) 260:1743–8.
[Abstract/Free Full Text] - Dental Health Services Victoria. A Guide to the Use of Quality Clinical Indicators in the Community Dental Program. 3. (Revised: October 2006).
- World Health Organization. Health systems: improving performance. In: World Health Report 2000 (2000) Geneva: WHO.
- Loeb JM. The current state of performance measurement in health care. Int J Qual Health Care (2004) 16:5–9.[CrossRef]
- Mattke S, Kelley E, Scherer P, et al. Health care quality indicators project: initial indicators report. OECD Health Working Papers (2006) No. 23.
- A Draft National Strategy for Clinical Indicators in Scotland. NHS Quality Improvement Scotland 2006. http://www.nhshealthquality.org (5 December 2007, date last accessed).
- Kazandjian VA, Wood P, Lawthers J. Balancing science and practice in indicator development: the Maryland Hospital Association Quality Indicator (QI) Project. Int J Qual Health Care (1995) 7:39–46.
[Abstract/Free Full Text] - Thomson R, Taber S, Lally J, et al. UK Quality Indicator Project® (UK QIP) and the UK independent health care sector: a new development. Int J Qual Health Care (2004) 16:51–6.
[Abstract/Free Full Text] - Chiu W, Yang C, Lin H, et al. Development and implementation of a nationwide health care quality indicator system in Taiwan. Int J Qual Health Care (2007) 19:21–8.
[Abstract/Free Full Text] - Rubin HR, Pronovost P, Diette G, et al. From a process of care to a measure: the development and testing of a quality indicator. Int J Qual Health Care (2001) 13:489–96.
[Abstract/Free Full Text] - Mainz J, Bartels PD, Laustsen S, et al. The National Indicator. Project to monitoring and improving of the medical technical care. Ugeskr Laeger (2001) 163:6401–6.[Medline]
- Ballard DJ. Indicators to improve clinical quality across an integrated health care system. Int J Qual Health Care (2003) 15:13–23.[CrossRef]
- Mattke S, Epstein AM, Leatherman S. The OECD Health Care Quality Indicators Project: history and background. Int J Qual Health Care (2006) 9. 1–4.
- Arah OA, Westert GP, Hurst J, Klazinga NS. A conceptual framework for the OECD Health Care Quality Indicators Project. Int J Qual Health Care (2006) 9. 5–13.
- Kelley ET, Arispe I, Holmes J. Beyond the initial indicators: lessons from the OECD Health Care Quality Indicators Project and the US National Healthcare Quality Report. Int J Qual Health Care (2006) 9. 45–51.
- The Australian Council on Healthcare Standards. Clinical Indicators Users Manual 2006 (2006) Ultimo, NSW: ACHS. http://www.achs.org.au (February 2008, date last accessed).
- Buchanan J, Pelkowitz A, Seddon M. Quality improvement in New Zealand healthcare. Part 4: achieving effective care through clinical indicators. J N Z Med Assoc (2006) 8. 119.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||