International Journal for Quality in Health Care Advance Access published online on March 2, 2007
International Journal for Quality in Health Care, doi:10.1093/intqhc/mzm004
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Understanding what works—and why—in quality improvement: the need for theory-driven evaluation
Centre for Public Policy and Management, Manchester Business School, University of Manchester, Booth Street West, Manchester, M15 6PB, UK
Address reprint request to Kieran Walshe, Manchester centre for Healthcare Management, University of Manchester, Manchester, UK; Tel: (0161) 275 3852. Fax: (0161) 273 5245. E-mail: kieran.walshe{at}man.ac.uk
Accepted for publication January 15, 2007.
Clinicians who are asked to participate in quality improvement programmes in healthcare organizations are often heard to ask for the evidence that they ‘work’. By that, they often mean they want randomized controlled trials, which show that accreditation, or credentialing, or criterion-based audit, or adverse event monitoring, or continuous quality improvement programmes, or whatever approach is being used cause meaningful and worthwhile improvements in the quality of care[1]. When they learn that there are relatively few experimental studies of quality improvement interventions[2], and those which do exist often show weak or moderate effects at best, this state of affairs is sometimes used to argue that it is not worthwhile investing time and effort in quality improvement. After all, the argument goes, we should not embark on using a new clinical intervention such as a drug or a surgical procedure without solid experimental evidence of its effectiveness, so why should we have a lower threshold for the adoption of organizational interventions like quality improvement programmes? Surely, they too should be proven to ‘work’ before they are adopted or implemented widely?
This point of view needs to be challenged. It is founded on a fundamental misunderstanding of the place of experimental methods in investigating and understanding complex social interventions, which is commonplace particularly among clinicians and biomedical researchers and which can seriously hamper those both researching and implementing quality improvement in healthcare.
In all research, our starting point should be to match our research methods to the questions or issues being investigated (and not the other way around). Before selecting an experimental research design, we should ask whether it fits the intervention we want to study. In particular, we need to consider the context in which the intervention is used, the content of the intervention itself, the process by which it is applied and the nature of its results or outcomes. In each of these domains, we may find either low variance (homogeneity) or high variance (heterogeneity). Table 1 offers some examples by way of illustration.
|
When an intervention has low variance across these domains, then the experimental method elegantly eliminates all potential biases and confounders, proves causality beyond dispute and quantifies effect. The theoretical basis for the intervention is a secondary consideration—our focus is on its empirical performance. However, when we find high variance in one or more domains, the value of the experimental method is less clear, because the variance reduces or even eliminates our ability to generalize empirically about the impact of an intervention from any specific experiment, done with a certain context, content and application combination, and to draw conclusions about how it might work in a different context, with a different content and different application. Then, the theoretical basis for the intervention (why and how it works) becomes more important than its empirical performance (whether it works) in any particular study.
As the natural heterogeneity of an intervention increases, experimental methods become progressively less helpful in understanding its effectiveness. We see this in the findings from experimental research into some clinical interventions, such as controlled trials of surgical procedures where application variance can be high, or of behavioural therapy interventions where content, application and outcome variance are significant. In such studies, the findings are more difficult to interpret and apply to clinical practice than those for other clinical interventions where variance is low—for example, many pharmaceutical therapies.
But quality improvement initiatives are complex social interventions, for which high levels of variance in context, content and application are often inherent and desired characteristics of the initiative[3]. For example, the responses of different healthcare organizations to a continuous quality improvement programme, or a system for adverse event reporting and investigation, will be quite different—and the programme or system will rightly be tailored or modified to make it work better in the individual organizational setting. Attempts to ‘standardize’ or control such interventions to make them fit an experimental paradigm completely miss the point—that their multiple outcomes are a complex co-product of context, content and application variables. A different approach to evaluating effectiveness is needed.
In researching healthcare quality improvement, we should learn from those in other settings—such as education[4] and criminal justice[5]—who have decades of experience in researching complex social interventions. Programmes designed to teach children to read, to rehabilitate offenders or to mentor disaffected teenagers are just as challenging to evaluate. Researchers in these fields have largely abandoned the experimental method, in favour of theory-driven approaches to evaluation[6]. In brief, theory-driven evaluation first attempts to map out the programme theory lying behind the intervention and then designs a research evaluation to test out that theory. The aim is not to find out ‘whether it works’, as the answer to that question is almost always ‘yes, sometimes’. The purpose is to establish when, how and why the intervention works, to unpick the complex relationship between context, content, application and outcomes, and to develop a necessarily contingent and situational understanding of effectiveness. The researchers seek theoretical rather than empirical generalizability—the ability to transfer theories from the research setting and bring them to bear in often quite different combinations of context, content and application[7].
In conclusion, I do not argue that there is no place for experimental research methods in testing the effectiveness of quality improvement interventions. Far from it, I fully acknowledge the power of experiments in demonstrating causality and quantifying effect, but I also assert the need for a theoretically driven approach to understanding complex social interventions and their effects, and the greater explanatory power of a more contingent and situationally sensitive approach to research.
The last word can be left to the researchers who, over 60 years ago, undertook the famous Hawthorne experiments, exploring the effects on the productivity of workers at a Western Electric plant of various changes in working conditions. Although the rigour of these studies has since been criticized, the researchers acknowledged at the time the methodological problems they faced. They found that their efforts to use controlled experiments to test for the effects of single variables (like pace of work, or rest periods) were practically impossible, and resulted in them altering the very thing they were trying to research. The researchers found that experimentation created ‘[not] an ordinary shop situation, but a socially contrived situation of their own making. With this realisation, the inquiry changed its character. No longer were the investigators interested in testing for the effects of single variables. In the place of a controlled experiment, they substituted the notion of a social situation which needed to be described and understood as a system of interdependent elements.’[8] Similarly, healthcare quality improvement programmes are complex social interventions that can only be properly evaluated if their interconnected context, content, application and outcomes are understood.
 |
References |
|---|
 Top References |
|---|
- Grimshaw J, McAuley LM, Bero LA, et al. (2003) Systematic review of the effectiveness of quality improvement strategies and programmes. Qual Saf Health Care 12:298–303.
[Abstract/Free Full Text] - Perneger T. (2006) Ten reasons to conduct a randomized study in quality improvement. Int J Qual Health Care 18:395–6.
[Free Full Text] - Walshe K and Freeman T. (2002) Effectiveness of quality improvement: learning from evaluations. Qual Saf Health Care 11:85–87.
[Abstract/Free Full Text] - Brown AL. (1992) Design experiments: theoretical and methodological challenges in creating complex interventions in classroom settings. J Learn Sci 2:141–78.
- Pawson R and Tilley N. (1994) What works in evaluation research. Br J Criminol 34:291–306.
[Abstract/Free Full Text] - Pawson R and Tilley N. (1997) Realistic Evaluation(Sage Publications, London).
- Pawson R, Greenhalgh P, Harvey G, Walshe K. (2005) Realist review – a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy 10:21–34.
- Roethlisberger FS and Dickson WJ. (1939) Management and the Worker(Harvard University Press, Cambridge, MA).
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  F. Davidoff Heterogeneity Is Not Always Noise: Lessons From Improvement JAMA, December 16, 2009; 302(23): 2580 - 2586. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. D. Ralston, D. P. Martin, M. L. Anderson, P. A. Fishman, D. A. Conrad, E. B. Larson, and D. Grembowski Group Health Cooperative's Transformation Toward Patient-Centered Access Med Care Res Rev, December 1, 2009; 66(6): 703 - 724. [Abstract] [PDF]
|
|
|
|
 |
|
 A. K Rowe, F. Onikpo, M. Lama, and M. S Deming The rise and fall of supervision in a project designed to strengthen supervision of Integrated Management of Childhood Illness in Benin Health Policy Plan., November 18, 2009; (2009) czp054v1. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 B. Andersson-Gare Improvement education by improving education: a model for integration of teaching, learning and research while practice is improved Qual. Saf. Health Care, August 1, 2009; 18(4): 246 - 247. [Full Text] [PDF]
|
|
|
|
 |
|
 G. Filardo, D. Nicewander, J. Herrin, J. Edwards, P. Galimbertti, M. Tietze, S. Mcbride, J. Gunderson, A. Collinsworth, Z. Haydar, et al. A hospital-randomized controlled trial of a formal quality improvement educational program in rural and small community Texas hospitals: one year results Int. J. Qual. Health Care, August 1, 2009; 21(4): 225 - 232. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. K. Rowe Potential of Integrated Continuous Surveys and Quality Management to Support Monitoring, Evaluation, and the Scale-Up of Health Interventions in Developing Countries Am J Trop Med Hyg, June 1, 2009; 80(6): 971 - 979. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. Walshe Pseudoinnovation: the development and spread of healthcare quality improvement methodologies Int. J. Qual. Health Care, June 1, 2009; 21(3): 153 - 159. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. Waring Getting to the 'roots' of patient safety Int. J. Qual. Health Care, October 1, 2007; 19(5): 257 - 258. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||