International Journal for Quality in Health Care Advance Access originally published online on August 26, 2006
International Journal for Quality in Health Care 2006 18(5):336-345; doi:10.1093/intqhc/mzl033
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Evaluating treatment process redesign by applying the EFQM Excellence Model
1 Amsterdam Institute for Addiction Research, Academic Medical Center University of Amsterdam, 2 The Jellinek Centre, Innovation and Prevention Departement, Amsterdam, and 3 Amsterdam Institute for Addiction Research, Academic Medical Center University of Amsterdam, The Netherlands
Objective. To evaluate a treatment process redesign programme implementing evidence-based treatment as part of a total quality management in a Dutch addiction treatment centre.
Method. Quality management was monitored over a period of more than 10 years in an addiction treatment centre with 550 professionals. Changes are evaluated, comparing the scores on the nine criteria of the European Foundation for Quality Management (EFQM) Excellence Model before and after a major redesign of treatment processes and ISO certification.
Results. In the course of 10 years, most intake, care, and cure processes were reorganized, the support processes were restructured and ISO certified, 29 evidence-based treatment protocols were developed and implemented, and patient follow-up measuring was established to make clinical outcomes transparent. Comparing the situation before and after the changes shows that the client satisfaction scores are stable, that the evaluation by personnel and society is inconsistent, and that clinical, production, and financial outcomes are positive. The overall EFQM assessment by external assessors in 2004 shows much higher scores on the nine criteria than the assessment in 1994.
Conclusion. Evidence-based treatment can successfully be implemented in addiction treatment centres through treatment process redesign as part of a total quality management strategy, but not all results are positive.
Keywords: addiction treatment centre, EFQM, evidence-based treatment, quality assurance, total quality management, treatment process redesign
Address reprint requests to Udo Nabitz, AIAR Jellinek Centre, Postbox 3907, 1001 AS Amsterdam, The Netherlands. E-mail: unabitz{at}jellinek.nl
Accepted for publication July 19, 2006.
There is increasing evidence that proves that total quality management is a powerful strategy for profit organizations to make changes and be innovative. Literature reviews [1], empirical studies, and opinion leaders [2,3] show that total quality management is effective. However, these studies are limited to profit organizations, measuring only financial outcomes. In health care, there is also converging evidence that an overall quality management strategy can be a successful way to be innovative [4], but strong empirical evidence is lacking [58].
As part of a large, nationwide innovation programme in the addiction field, one of the Dutch treatment centres adopted a total quality management strategy for several years. The centre had formulated an overall quality policy, implemented evidence-based treatment protocols by redesigning the primary processes, acquired the ISO certificate, and started to measure clinical outcome [911]. The goal of the strategy was to reduce the gap between evidence and practice and to make the outcomes more transparent [12]. This endeavour was evaluated by using the nine criteria of the European Foundation for Quality Management (EFQM) Excellence Model as a multi-dimensional framework. The research question of the study was whether this strategy is reflected in the quality improvement of the organization applying the nine criteria from the EFQM Excellence Model before and after intervention.
| Method |
|---|
|
|
|---|
Setting
The addiction treatment centre used in this study is responsible for almost monopolistically providing a wide range of free services for alcohol- and drug-dependent patients in the urban region of Amsterdam in the Netherlands. Annually, the centre treats approximately 3500 clients, and approximately 5000 persons are supported by Internet services. Next to cure and care, the centre provides a diversity of prevention, probation, dental, and laboratory services. The staff consists of 550 persons, and the annual budget is
32 million. The centre has the reputation of being a leading edge organization for quality management. In the 1980s, the first formal quality strategy was developed [13,14]. In 1994, the first external EFQM assessment was conducted followed by several improvement projects. In 1996, the improvements were assessed again and the centre received the Dutch Quality Price. This success was one of the reasons to start with a fundamental overall redesign programme.
Design
The centres total quality management strategy was initiated in 1994, by an EFQM self-assessment and improvement projects. The treatment process redesign programme was carried out in the period 19972003. An evaluation of the programme was made possible by comparing the first submission report on the state of the centre in 1994 with the EFQM submission report of 2004, which can be seen as a prepost quasi-experimental design [15].
Intervention
Total quality management.
Total quality management is a structured, systematic approach for creating continuous quality improvements in all processes by all people of an organization. This means applying methods such as multi-level self-assessment, auditing, improvement projects, plan-do-check-act cycles, certification, result and customer orientation, permanent education, and process redesign [16].
After conducting several improvement projects, the treatment centre introduced total quality management [17] in the early 1990s including the development of a quality management system that became ISO certified [18]. In the subsequent phase, the attention switched from support processes to treatment processes, including evidence-based protocols and redesigning the processes. Related to the process redesign was the measurement of clinical outcome at follow-up to make them more transparent and to stimulate clinicians to learn from feedback [19].
Evidence-based treatment of addiction.
Evidence-based treatment of addiction means that the research findings on the effectiveness of treatment modalities are used and translated into clinical practice [16]. Although addiction treatment has quite a research tradition, the first reviews on the effectiveness of treatments for alcohol and drugs were published in the 1980s and 1990s [20]. The university research institute related to the treatment centre used the reviews and evaluated the actual treatment modalities of the centre, which led to the conclusion that hardly any of the treatment modalities were evidence based. This initiated a radical redesign of the treatment processes. The research institute suggested combining the effective treatment modalities with the concepts of the disease model of alcoholism and stepped care for treatment planning [2123].
Treatment process redesign programme.
Treatment process redesign or re-engineering is the application of Business Process Redesign in health care organizations. It means that treatment processes are fundamentally reviewed and radically redesigned to achieve dramatic improvements in performance measures [24,25]. Process redesign, customer orientation, case management, and the use of information technology are the focus of the approach.
The process redesign programme was brought into practice by a steering group under the leadership of a dedicated, determined, and transformational oriented directorate. The group met every 2 weeks, directed the programme for the whole period and decided on the protocols, the trajectory model, the reorganization of the centre, the new housing facilities, and the technical infrastructure such as an electronic patient file.
A department for Quality and Innovation was established and equipped with fresh staff, external advisors, and a quality manager. In close cooperation with the research institute and within the realms of a national innovation programme [9], a formal design method in six steps was developed. Furthermore, all personnel of the centre were engaged in a diversity of quality projects. Middle management participated in monthly conferences, where progress was presented and a broad communication strategy was developed to inform, motivate, and stimulate clinicians, with emphasis on measuring and learning.
Main outcome measures
The EFQM quality assessment.
The nine EFQM criteria are a well-known and frequently applied framework for evaluating the quality level of a health care organization [1,26]. The criteria were originally developed to identify the European Quality Award winners but are now used to identify the quality level of many organizations. The criteria are detailed in 32 criterion parts, which are scored by assessors using the RADAR scoring matrix with scales from 0 to 100% [27].
Comparing two assessments.
In 1994, a submission report of the treatment centre was composed based on documents with quantitative and qualitative data from various sources [28]. Three independent Dutch quality assessors evaluated the report and the material, visited the centre, and reached a consensus on the final scores of the nine EFQM criteria [13]. These scores represent the status of the centre before the intervention and are seen for this study as the first assessment.
In 2004, a new EFQM submission report was composed, which reflected the situation of the centre in 2004 [29]. Seventeen members of the European EFQM Health Sector Group, among whom were nine EFQM assessors, studied the report, used the RADAR scoring matrix, and determined the final scores of the nine EFQM criteria in a consensus meeting. The findings of the group represent the status of the centre in 2004 and are seen as the second assessment.
Comparison in the radar graph.
The nine criteria of the EFQM Excellence Model and the assessment procedure remained consistent over the years, although the scoring system changed in 1999. In 1994, a simple scoring matrix was used for 28 criterion parts. In 2004, a complex RADAR scoring matrix was introduced with 32 criterion parts. Among the EFQM assessors, there is consensus that the old scoring system is an overestimation of the quality level of about 20%. For the purpose of this study, and after consulting two independent assessors who are experienced with the scoring systems, the scores of the nine criteria were adjusted (C. Amelun, personal communication; W. J. Veldt van der, personal communication). The 1994 scores were reduced by 10% and the 2004 scores enhanced by 10%. These adjustments allow a fair an methodological sound comparison of the 1994 and 2004 assessments.
| Results |
|---|
|
|
|---|
The results illustrate the situation of the treatment centre in 1994 and 2004 along the nine EFQM criteria. Table1 provides an impression of the innovations and changes in the centre and allows a qualitative comparison. The radar graph is based on the judgement of the assessors and allows a quantitative comparison.
|
Organizational improvements: EFQM criteria 14
Table 1 summarizes the most important changes. These are (i) the restructuring of the organization in three treatment circuits; (ii) intake, intensive cure, and complex care; and (iii) the use of an annual planning cycle, an extensive schooling programme, a trajectory management, new housing, and an advanced technical infrastructure.
Process improvements: EFQM criterion 5
In 1994, the work instructions, documentation, and protocols of all departments within the treatment centre were collected. This amounted to several hundred documents of which only a few were related to care and cure processes. The documents had no systematic structure and were incomplete, fragmented, limited, and not authorized. The protocols were neither evidence based nor did they meet the standards required for ISO certification.
In 2004, two types of protocols had been introduced in the treatment centre. Twenty-nine evidence-based treatment protocols were developed, 21 were implemented, and 11 were evaluated (Tables 2 and 3). A quality management system with eight groups of ISO-certified procedures was established covering over 100 structured, authorized, and certified documents.
|
|
Based on the idea of stepped care, a logical structure for the treatment protocols was developed, distinguishing eight trajectories and an Internet self-help path (Figure 1).
|
Outcome indicators: EFQM criteria 69
Client satisfaction: EFQM criterion 6.
In 1994, the centre participated in the first national mental health organization satisfaction survey. A total of 156 outpatient and 94 in-patient clients participated in the exit interviews. This was about 10% of all treated clients. The results show that there was high satisfaction concerning the professionals and low satisfaction concerning housing and some types of treatment. In 2004, the in-house call centre continuously collects the treatment outcome of approximately 1200 clients per year concerning three validated clinical scales, including the report card score. The findings show, per trajectory, the percentage of clients who were satisfied with three aspects of the treatment. On all but one of the scales, more than 50% are satisfied. The most positive rating shows that 91% were happy with the decision-making in the day treatment. The report card figure ranges from 6,7 (satisfactory) to 7,9 (good). For the trajectory Internet self-help, which was evaluated in 2004 by 522 clients, 50% were satisfied with the information, decision-making, and outcome, and 75% stated that the Internet self-help can be recommended to other people. The report card score was 6 (passed).
Personnel satisfaction: EFQM criterion 7.
In 1994, the first ad hoc personnel satisfaction measurement was conducted among 34 outpatient service personnel. There was a high satisfaction rate (85% and higher) signalling meaningful, pleasant, useful, and interesting work. A low satisfaction rate was given to career possibilities, schooling, and work stress 59% and lower. During the process redesign programme, two systematic surveys among the relevant teams were conducted to measure work satisfaction. The results of the first survey among more than 300 personnel showed high scores on the scales autonomy, responsibility, support by colleagues, and the meaningfulness of the work. However, the personnel negatively rated internal structure, workflow, and the work stress. Teams that participated in the process redesign programme scored unfavourably. The second survey in 2004 was part of the national work stress-monitoring programme. The scores show high work pleasure and strong commitment of the majority of the staff, but about half of the personnel indicate that they need time for recovery. The personnel still see as a negative point the structuring of the work processes, the mental and emotional burden, and the limited career and learning possibilities. However, the ratio of personnel on sick leave had decreased over the preceding 3 years and was lower than the national benchmark [30].
Society satisfaction: EFQM criterion 8.
In 1994, there was only a vague impression available of societys appreciation of the addiction centres achievements. Society as a whole was not yet seen as a stakeholder of the centre, and facts about their opinion were not available. Since 1996, a national population survey is conducted every 3 years. The image of the centre, measured with 13 items, is overall positive, but items such as innovation, costs, and flexibility are rated lower than the average and there are signs of a negative trend [31].
Clinical, productivity, and financial indicators: EFQM criterion 9.
In 1994, data on clinical outcome of the treatment programmes were not available. Table 4 summarizes the clinical outcomes, assessed by the call centre 9 months after intake. In 2004, 895 clients were interviewed, and complete data of the six trajectories became available. This is a response rate of 50%.
|
The abstinence rate varies between 31 and 68%. On average, about 50% of the clients were abstinent 9 months after intake, and about 20% controlled their consumption on an acceptable level. The tables allow to take into account the situation of the client at intake.
In 1994, the available data on productivity were rather limited. A total of 2044 clients entered the centre. Outpatient contacts exceeded 12000, and the occupation of the beds was 98%. In 2004, data on the productivity can be segmented in input, throughput, and output figures. A total of 3812 clients had an intake in 2004. A total of 5489 clients logged in for Internet support (input). A total of 3283 clients followed a treatment trajectory, of which 3044 finished the trajectory (throughput). In general, the output figures meet the formulated goals for 2004 and show an increase over the last 10 years.
The financial data for the whole centre show a gradual increase in total volume from
20 million in 1994 to
32 million in 2004. The balance has been positive for most of the years, but there were years with small negative annual results. The reserves that were generated during the preceding 4 years were more than 10% of the total turnover.
EFQM assessment: criteria 19.
In Figure 2, the scoring of the two EFQM assessor groups of 1994 and 2004 are graphically represented [29,32]. The dotted line of 1994 shows low quality (level 1 or lower than 20%) for the EFQM criteria: Processes (5), Customer Results (6), and People Results (7).
|
The line representing 2004 illustrates that the assessors are much more positive about the quality of the centre. All criteria are at level 2 or higher (above 40%). Based on the judgement of the assessors, four EFQM criteria have improved more than 20%: Processes (criterion 5), Partnerships and Resources (criterion 4), People (criterion 3), and Customer Results (criterion 6). Leadership (criterion 1) remained the same. Policy and Strategy (criterion 2) and the results criteria People Results (criterion 7), Key Performance Results (criterion 9), and Society Results (criterion 8) only show some improvement.
| Discussion and conclusions |
|---|
|
|
|---|
This study demonstrates that over a 10-year period, substantive and positive changes had been achieved by applying a total quality management strategy with the focus on treatment process redesign. A trajectory model, which constitutes the framework for intake, cure, and care, has been established; evidence-based treatment protocols have been introduced; the quality management system has been certified; and follow-up treatment outcomes have been constantly evaluated by an in-house call centre. Many additional innovations such as the housing, technical infrastructure, and in-service training that did not exist in 1994 were introduced by 2004. Consequently, the EFQM assessors testify a higher quality level of the treatment centre visualized in the radar graph.
There are only a few publications about similar projects in health care organizations. In the Basque Country in Spain, all 32 health care services used a total quality management approach and reported positive results concerning all nine EFQM criteria [7]. The findings of Sanchez et al. [7] also showed that the personnel satisfaction radically improved, which shows that innovations can also lead to higher personnel satisfaction, which was not achieved in this study of the addiction treatment centre. A publication about a dental clinic in Switzerland with a staff of about 20 professionals also shows very positive results. The clinic worked with a total quality management strategy and carried out a large variety of creative improvement projects and was honoured with the European Quality Award [33]. Both studies proved that a total quality management strategy takes time before sustainable effects become visible. The well-known study of Hendricks and Singhal [34] in which more than 800 profit organizations were compared using an experimental design shows that it took at least 6 years before the experimental group, which had followed a total quality management strategy proved to be superior to the control group.
Studies of complex changes and improvements of treatment processes over a period of 10 years have to cope with many methodological constraints. The three most important ones in this study were the stability of the scoring system, the availability of the data, and the compatibility of the assessment carried out.
In this study, the criteria for the scoring remained consistent, which were the nine criteria of the EFQM Excellence Model. But the criterion parts and the scoring system had changed. There is a broad understanding among assessors that the new RADAR scoring matrix is more demanding. Consequently, the improvements are underestimated. To compensate for this bias, the scores had to be readjusted. Guidelines on how to compare old and new EFQM scores should be developed.
There were quite some data available for a statistically prepost comparisons, but in most cases, the data were incomplete, the samples were not comparable because of new structures and new teams, and often the definitions of 1994 and 2004 did not match. Therefore, a statistical analysis was not possible, and the changes were based on the judgement of the assessors.
The basic approach along which the assessors evaluated the submission reports was consistent. The assessors were trained and experienced and used the standard procedure to reach consensus. But the assessors of 2004 did not have the chance to make a site visit, whereas the assessors of 1994 did.
These methodological problems were discussed extensively by the authors, and the best solution was chosen with the goal to make positive and negative changes transparent. However, it was a challenge to find appropriate solutions. Guidelines for a pragmatic science as proposed by Berwick are needed.
The goal of the addiction treatment centre used in this study was to reduce the gap between scientific evidence and everyday practice and to make the results of the centre more transparent. It can be concluded that this goal has been achieved. However, the implementation of the evidence protocols is still ongoing, and the dramatic improvement in performance, as predicted by Hammer and Champy [13,25] has not been demonstrated. The client satisfaction and the clinical results are not better than in other studies [35], and People Results (criterion 7) and Society Results (criterion 8) show no clear measurable improvement. It is very likely that only new effective treatment modalities in combination with a total quality management strategy can lead to a real breakthrough in the performance of addiction treatment.
| Acknowledgements |
|---|
|
|
|---|
The treatment process redesign programme was carried out in the Jellinek centre in Amsterdam, the Netherlands. In 2004, the Board of Directors were Petra van Dam and Christian Krappel. The innovation programme was initiated by Jan Walburg then CEO of the centre. During the programme, the Board of Directors comprised of Jan Walburg, Astrid van Dijk, and Petra van Dam. This study was supported by the Amsterdam Institute for Addiction Research of the Academic Medical Centre Amsterdam and was funded by the Zorgkantoor AGIS Amerfoort and the Ministry of Health, Welfare and Sports, The Hague.
| References |
|---|
|
|
|---|
- Malorny C. TQM Umsetzen. [Implement TQM.]. Stuttgart: Schaffer-Poeschel Verlag, 1999.
- Garvin D. How the baldrige award really works. Harv Bus Rev 1991; December: 8093.
- Buelens M. Managementprofeten. Amsterdam: Nieuwezijds, 2000.
- Shortell S, OBrien J, Carman J et al. Assessing the impact of continuous quality improvement or total quality management: concept versus implementation. Health Serv Res 1995; 30 (377): 401.
- Institute of Medicine. Crossing the Quality Chasm. Washington, DC: Institute of Medicine, 2002.
- Moeller J.The EFQM Excellence Model. German experiences with the EFQM approach in health care. Int J Qual Health Care 2001; 13 (1): 4549.
[Abstract/Free Full Text] - Sanchez E, Letona J, Gonzalez R et al. A descriptive study of the implementation of the EFQM excellence model and underlying tools in the Basque Heath Service. Int J Qual Health Care 2006; 18 (1): 5865.
[Abstract/Free Full Text] - Grol R.Improving the quality of medical care. Building bridges among professional pride, payer profit and patient satisfaction. J Am Med Assoc 2001; 286 (20): 25782585.
[Abstract/Free Full Text] - Schippers GM, Schramade M, Walburg JA. Reforming Dutch substance abuse treatment services. Addict Behav 2002; 27 (6): 9951007.[CrossRef][Web of Science][Medline]
- Walburg JA. Integrale Kwaliteit in de Gezondheidszorg. [Total Quatlity Management in Health Care]. Deventer: Kluwer BedrijfsInformatie, 1997.
- Schippers G, Es van M, Mulder E, Dijk van A. The addiction services go professional. Evaluation of a large-scale service innovation project. Maandblad Geestelijke Volksgezondheid 2005; 60 (5): 527539.
- Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organisations: systematic review and recommendations. Milbank Q 2004; 82 (4): 581629.[CrossRef][Web of Science][Medline]
- The Jellinek Centre. Terugmeldingsrapport Nederlandse Kwaliteit [Feedbackreport Dutch Quality]. Amsterdam: The Jellinek Centre, 1994.
- Walburg J. Staan voor kwaliteit [We Garante Quality]. Amsterdam: Jellinek, 1990.
- Cook TD, Campbell DT. Quasi-Experimentation. Design and Analysis Issues for Field Settings. Boston: Houghton Mifflin Company, 1979.
- Joint Commission Mission. Lexikon: Dictionary of Health Care Terms, Organisations and Acronyms. Oakbrook Terrace: Joint Commission on Accreditation of Healthcare Organisations, 1998.
- Nabitz U, Klazinga N, Walburg J. The EFQM excellence model: European and Dutch experiences with the EFQM approach in health care. European Foundation for Quality Management. Int J Health Care Qual Assur 2000 June; 12 (3): 191201.
- Bouwens R. Invoering van een kwaliteitsmanagementsyteem dat voldoet aan de norm ISO 9001:2000 [Introduction of a quality management system along the ISO 9001:2000 serie]. Handb Addict (Handboek Verslaving) 2001; 28 (B9010): 324.
- Mugford M, Banfield P, OHanlon M. Effects of feedback of information on clinical practice: a review. BMJ 1991; 303: 650651.
[Free Full Text] - Miller W, Andrews N, Wilbourne P, Bennet M. A wealth of alternatives. Effectives treatments for alcohol problems. In Treating Addictive Behaviors. New York: Plenum Press, 1986: 203229.
- Brink WVD. Effectieve interventies bij alcoholproblemen [Effective interventions for alcohol problems]. In Brink van den W, Schippers GM, eds. Behandelingsstrategieën bij alcoholproblemen. Houten: Bohm Stafleu Van Loghum, 1996: 2547.
- Institut of Medicine. Broadening the Base of Treatment of Alcohol Problems. Washington DC: National Academic Press, 1990.
- Sobel M, Sobel L. Controlled drinking after 25 years. How important was the great debate? Addiction 1995; 90: 11491153.[CrossRef][Web of Science][Medline]
- Johansson H, Mchuch P, Pendlebury A, Wheeler W, Bate P. Business Process Reenginering: Breakpoint, Strategies for Market Dominance. West Sussex, England: John Wiley & Sons, 1993.
- Hammer M, Champy J. Reengineering the Cooperation. London: Nicolas Brealy, 1993.
- Brandt E. Qualitätsmanagement Gesundheitsförderung im Krankenhaus. Handbuch zur EFQM-Einführung [Quality Management in Health Promoting Hospitals. Manual to Introduce EFQM]. Neuwied: Luchterhand, 2001.
- European Foundation for Quality Management. EFQM Excellence Model. Public and Voluntary Sector Version. Brussels: European Fundation for Quality Management, 2003.
- The Jellinek Centre. Aanmeldingsrapport Nederlandse Kwaliteitsonderscheiding [Submission report for the Dutch Quality Award]. Amsterdam: The Jellinek Centre, 1994.
- Schramade M, Nabitz U, Osseman D, Visser G. EQA European Quality Award 2005. Concept Submission Report. Amsterdam: Jellinek, 2004.
- Schramade M, Nabitz U. Ondernemend in kwaliteit. Een intergrale en resultaatgerichte aanpak. De casus Jellinek [The Enterprize of Quality. An integral resultoriented approach of the Jellinek case]. Amsterdam: SWP, 2005.
- Nabitz U, Vis C, Brink van den W. Wat vindt de Nederlandse bevolking van de GGZ en de verslavingszorg? [What is the opinion of the Dutch citizen about mental health and addiction treatment services?] Maandblad Geestelijke Volksgezondheid 2001; 56 (9): 795805.
- Dorren PLL, Stevens RWF, Wiersema MI. Terugmeldrapport. De Jellinek [Audit Report: The Jellinek Centre]. Eindhoven: Nederlandse Kwaliteit, 1994.
- Harr R. TQM in dental practice. Int J Health Care Qual Assur 2001; 14 (23): 6981.
- Hendricks KB, Singhal VR. Does implementing an effective TQM program actually improve operating performance? Empirical evidence form firms that have won quality awards. Manag Sci 1997; 43 (9): 12581274.
- Küfner H, Feurlein W. In-Patient Treatment for Alcoholism. A Multi-Centre Evaluation Study. Berlin: Springer Verlag, 1989.
This article has been cited by other articles:
![]() |
P. Vallejo, A. Ruiz-Sancho, M. Dominguez, M. J. Ayuso, L. Mendez, J. Romo, L. Ontoria, P. Rodriquez, C. Serina, and C. Arango Improving quality at the hospital psychiatric ward level through the use of the EFQM model Int. J. Qual. Health Care, April 1, 2007; 19(2): 74 - 79. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||


