Comparing cost effects of two quality strategies to improve test ordering in primary care: a randomized trial
)1 Centre for Quality of Care Research (WOK), Care and Public Health Research Unit (CAPHRI), Department of General Practice, Maastricht University, Maastricht, 2 Centre for Diagnostics and Consultation, St Jans Hospital, Weert, 3 Department of Health Organisation Policy and Economics, Maastricht University, Maastricht, the Netherlands, 4 Research Unit, Department of Public Health, Ottawa University, Ottawa, Canada
Objective. To determine the costs and cost reductions of an innovative strategy aimed at improving test ordering routines of primary care physicians, compared with a traditional strategy.
Design. Multicenter randomized controlled trial with randomization at the local primary care physicians group level.
Setting. Primary care: local primary care physicians groups in five regions of the Netherlands with diagnostic centers.
Study participants. Twenty-seven existing local primary care physicians groups, including 194 primary care physicians.
Intervention. The test ordering strategy was developed systematically, and combined feedback, education on guidelines, and quality improvement sessions in small groups. In regular quality meetings in local groups, primary care physicians discussed each others’ test ordering behavior, related it to guidelines, and made individual and/or group plans for change. Thirteen groups engaged in the entire strategy (complete intervention arm), while 14 groups received feedback only (feedback arm).
Main outcome measure. Running costs, development costs, and research costs were calculated for the intervention period per primary care physician per 6 months. The mean costs of tests ordered per primary care physician per 6 months were assessed at baseline and follow-up.
Results. The new strategy was found to cost 
702.00, while the feedback strategy cost 58.00. When including running costs only, the intervention was found to cost 554.70, compared with 17.10 per primary care physician per 6 months in the feedback arm. When excluding opportunity costs for the physicians’ time spent, the intervention was found to cost 92.70 per physician per 6 months in the complete intervention arm. The mean costs reduction that physicians in that arm achieved by reducing unnecessary tests was 144 larger per physician per 6 months than the physicians in the feedback arm (P = 0.048).
Conclusion. On the basis of our findings, including the expected non-monetary benefits, we recommend further long-term effect and cost-effect studies on the implementation of the quality strategy.
Keywords: costs, costs analysis, diagnostic tests, feedback, health care, quality assurance, routine
Address reprint requests to Wim H. J. M. Verstappen, Department of General Practice, Maastricht University, PO Box 616, 6200 MD Maastricht, The Netherlands. E-mail: wim.verstappen{at}HAG.unimaas.nl
Accepted for publication June 7, 2004.
In times of limited resources for health care, it is necessary to evaluate not only the cost-effectiveness of new treatments or procedures for patient care, but also the cost-effectiveness of new strategies to improve the quality of health care delivery. Economic evaluations of interventions aimed at changing the behavior of primary care physicians assess the balance between benefits attained and resources needed [1–4]. Many strategies have been developed to improve test ordering behavior of primary care physicians, because the numbers of tests ordered are growing in many countries, even though established guidelines regard many of these tests as unnecessary [5,6]. Rigorous studies of the effects of strategies such as educational materials, reminders, feedback, small group quality meetings, and financial incentives have so far produced heterogeneous results [7–9]. A few studies investigating costs have also yielded contradictory outcomes [10–15]. We initiated an economic evaluation study to evaluate the costs and the effects of a strategy that combines a traditional feedback strategy with a multifaceted strategy, including feedback, dissemination of and group education on evidence-based guidelines, and small group quality improvement meetings in a local primary care physicians group, using social influence as an important motivator for change [16,17]. A genuine effect of this innovative, multi-faceted strategy has been observed and presented elsewhere [18].
The present paper provides a method for cost analyses of such quality improvement strategies, and compares the costs and cost reductions of the new strategy with one of its elements, ‘classic’ feedback, to assess whether implementation of the innovative test ordering quality strategy on a national scale would be worthwhile, depending not only on its effectiveness, but also on the costs involved and the savings achieved.
 |
Methods |
|---|
 Top Methods ResultsDiscussionReferences |
|---|
Setting
The strategy was applied in five regions of The Netherlands with a diagnostic center, which is an institute, usually associated with a hospital, where primary care physicians can order tests without referring patients to the hospital. Our strategy aimed at local primary care physicians groups, an existing infrastructure of Dutch primary care physicians collaborating in a specific region. These groups share patient care outside office hours and many of them also engage as a group in small group quality improvement activities, e.g. prescription quality circles. Local groups with a link to one of these five diagnostic centers were eligible for the study. The medical coordinator of the diagnostic center provided the test ordering data needed, distributed the feedback reports, and supervised the small group quality improvement meetings.
Design and measurements
The new strategy was tested in a multicenter, randomized controlled trial, with randomization at local primary care physicians group level. The original complete design consisted of three arms. Two of the original three arms will be reported in this paper: one of the complete intervention arms (complete intervention arm), also reported in JAMA [18], and the control arm groups that only received feedback (feedback arm).
Numbers of tests ordered were assessed over a period of 6 months before the intervention (the baseline period) and a period of 6 months after the intervention (the follow-up period). The 6-month intervention took place in 1999. Participating local groups were randomized centrally, and stratified by the size of the local group and the region in order to spread the workload of the medical coordinators of the diagnostic center.
Intervention
The complete intervention consisted of the following elements: a graphical feedback report including a comparison of personal test ordering data with those of colleagues, dissemination of and group education on national, evidence-based guidelines, and quality improvement meetings in small groups. The improvement strategy concentrated on three specific clinical subjects (cardiovascular topics, upper abdominal complaints, and lower abdominal complaints), and the laboratory, imaging, and function tests used for these clinical problems. Table 1 describes the clinical problems and the associated laboratory, imaging, and function tests that were included in the experiment. Each report was followed by a standardized small group quality improvement meeting, at which the feedback data relating to one of the clinical problems and the guidelines were discussed. At the end of the session concrete plans for change, both at individual and local primary care physicians group level, were established.
|
Effect measures and measuring instruments
Measuring costs. All costs of producing the feedback reports and organizing the small group quality meetings were calculated. Costs were divided into the following categories:
1. Running costs
1.1. Costs of the feedback reports. Staff members of the diagnostic centers extracted and edited the data. The production costs partially depended on the number of primary care physicians who participated; more physicians meant more written reports, and hence more production time and more postage costs. Secretarial time and paper costs were calculated per feedback report.
1.2. Costs of the quality meetings. Secretarial time spent organizing the meeting and the time spent by the medical coordinator preparing and chairing the sessions were calculated per meeting per physician.
Since each meeting lasted 1.5 hours, and we assumed half an hour for preparation and travelling, one meeting took 2 hours of the primary care physician’s time. Physician fees were derived from the Dutch Government’s annual care review. Total national expenditure for curative primary care physician care in 1998 was 1 023 227 100, which corresponds to an hourly rate of 77. These costs were opportunity costs; in the time the physician attended the meetings, he/she could not ‘produce’ other work.
2. Development costs.
These costs covered activities for the continuation of the project, e.g. administration, organization, and the development and updating of concise guideline information. A software company developed software for the production of the feedback reports, and their costs were included as well.
3. Research costs.
Scientific development of the strategy, expert meetings, the financial compensation physicians received for participating in this study with related activities, e.g. completing evaluation forms, and working up the questionnaires and evaluation forms were counted as research activities with related costs.
Registration forms measuring the time needed to extract data and to produce and send feedback reports were completed by the staff members. Costs were then calculated on the basis of the salary scales of staff members at the diagnostic center and the research department.
Measuring cost reductions. Cost reductions were calculated using existing standard tariffs per test (Table 1). In The Netherlands, costs of laboratory tests are reimbursed according to standard prices for tests. Reimbursement for imaging and function tests includes hospital costs and specialists’ fees. Cost reductions were determined by assessing the mean difference in the costs of tests ordered per primary care physician and per 6 months between the follow-up period and the baseline period, and comparing this difference between the two arms. Cost reductions of laboratory tests were analyzed separately, because although they are a minor part of the cost reductions, they constitute the great majority of tests.
Consistent with the national, evidence-based test-ordering guidelines, which have limited indication for ordering tests for the three clinical problems included in the study, combined with the well known effect that overuse occurs, a decrease in the number of tests was considered to represent improved patient care.
Analysis
Costs of the intervention and the feedback strategy were calculated per primary care physician per 6 months. Since the unit of randomization was the local primary care physicians group, the unit of analysis also had to be the local primary care physicians group. Therefore, multilevel analyses were applied to evaluate whether the local primary care physicians groups were important determinants of the effects of the intervention. A three-level model was used, with the primary care physicians group as level 3, the primary care physicians as level 2, and the numbers of tests as level 1. This model was analyzed using SAS PROC MIXED. Multilevel baseline analyses showed that analyses could be performed without the local primary care physicians groups. All effects were analyzed with analyses of covariance using the costs of tests during the follow-up period as the dependent variable and the costs of tests at baseline and the region, which appeared to be an important determinant, as independent variables. A sensitivity analysis was performed by varying the inclusion of the various cost categories and cost reductions.
|
Results |
|---|
|
TopMethodsResultsDiscussionReferences |
|---|
A total of 38 local primary care physicians groups were invited by open recruitment to take part in this trial. Twenty-seven local groups with 194 physicians immediately expressed their willingness to participate. After randomization, the complete intervention arm included 13 local primary care physicians groups and the feedback arm included 14. Figure 1 illustrates the study design and shows that follow-up data were unavailable for 20 physicians. Table 2 shows that there were no differences in individual characteristics between the two arms. There was a large, but statistically not significant difference in numbers and costs of laboratory tests and all tests ordered per primary care physician between the two arms at baseline.
|
|
Costs of the strategy and cost reductions in test ordering
Table 3 shows the total costs of the intervention. Concerning the running costs of the strategy, the cost of one feedback report per primary care physicians was 5.70. The costs per physician per quality meeting were 25.20 for 4.25 hours of coordination time, including secretarial, preparation, meeting, and travelling time. The opportunity cost of the physicians’ time spent attending the meetings was 2 hours x 77 = 154 per physician per meeting. With respect to the development costs, guidelines were only used in the complete intervention arm and costs were 4 per primary care physician. The total cost of the intervention was 65 998: 702 per physician per 6 months for the complete intervention arm and 58 per physician per 6 months for the feedback arm. If only part of the running costs are counted (the opportunity costs of primary care physicians for the time spent on the quality meetings are excluded), the total running costs amount to 92.70 per physician for the complete intervention arm, and 17.10 per physician for the feedback arm per 6 months.
|
Table 4 shows that the costs of laboratory tests, as well as all tests, decreased in both arms, but significantly more so in the complete intervention arm than in the feedback arm. The total cost reduction in the complete intervention arm per primary care physician per 6 months was 144 more than in the feedback arm.
|
Table 5 shows the results of a sensitivity analysis. When including opportunity costs for primary care physicians’ attending time, the costs for the complete intervention arm exceeded the cost reductions. The cost reductions of the complete intervention arm exceeded the costs, with 208.30 (301 – 92.70) per physician per 6 months, with only part of the running costs included (excluding the opportunity costs). The cost reductions of the feedback arm were larger than its costs for all cost categories, and introducing the feedback strategy would save 143.90 (161 – 17.10) per physician per 6 months when including only running costs.
|
|
Discussion |
|---|
|
TopMethodsResultsDiscussionReferences |
|---|
The present paper evaluates the costs and cost reductions of an innovative strategy to improve primary care physicians’ test ordering, involving feedback, education on guidelines, peer interaction, and social influence, by comparing it with a traditional approach involving only the provision of feedback. The new strategy improved test ordering more substantially and consistently, and, besides the favorable clinical effects, appeared to bring about more cost reduction than feedback alone when not counting the opportunity costs [18–20]. Introducing this effective strategy in The Netherlands, with its
7100 practicing primary care physicians, would then save 1 478 930 (7100 x 208.30) in the first 6 months. There are some methodological aspects of our study that need to be considered. Unfortunately, clinical outcome data were not available, and outcome measures were data on costs of tests ordered; therefore, the quality of the test ordering could not be measured. In general, the focus of our intervention was on a decrease in the volume of tests, in accordance with the national evidence-based guidelines. This means that there was a potential danger of the intervention resulting in the underuse of tests. However, the physicians discussed their feedback data, and if it appeared that a physician clearly ordered fewer tests than his/her colleagues, he/she made plans for ordering more tests. Moreover, there is empirical evidence that a general reduction in test use in primary care does not lead to more referrals or substitution of care [21].
Concerning cost reductions, a reduction in the number of laboratory tests ordered does not always influence laboratory costs; for example, a diagnostic apparatus performing fewer tests costs the same amount of money, and only a large reduction can mean that fewer laboratory staff are needed. We could not include such potential cost reductions. For instance, not performing a redundant test also implies that a patient does not have to take time off work. More importantly, we were unable to assess the cost reductions achieved by not performing tests that would result in false-positive findings. Such test results may lead to a cascade of further testing, or inappropriate treatment or referrals, and as a result of better diagnosing patients costs are saved in the long run. The limited time frame of the study prevented us from studying these effects, since patients included in our study would have needed to be monitored for several years. Finally, for the same reason we were unable to assess possible learning effects, which could mean that quality activities may become less time-consuming over time, even if the approach is directed to other clinical problems.
Our study deals with some interesting and important topics for costs analyses of quality improvement studies. As in many quality improvement studies, only intermediate effect measures instead of patient outcome measures were available. These kinds of cost analyses can be seen as cost minimization analyses, since, by applying evidence-based test ordering guidelines, negative effects on patient’ outcome are not expected in the quality strategy [22]. The analyses were done from a societal perspective, but the perspective of the primary care physicians involved may also be important. Further, we focused on the costs and cost reductions, expressed in monetary units, but with our new strategy we may also expect non-monetary benefits related to the strategy, such as improvement of the physicians’ clinical knowledge and job satisfaction. Of course it is difficult to quantify these important benefits in such a cost analysis. There is some empirical evidence that participating in such quality improvement activities may increase job satisfaction [23,24]. Moreover, we calculated the opportunity costs for the time spent by the physicians in attending the quality meetings, and in our study these costs were found to be the largest cost type. Activities, among them educational and quality ones, compete with each other for available time. In general, these opportunity costs should be included because they weigh (in monetary units) the time needed for conducting the activities considered in this study, and also the time that is no longer available for other activities, which is why they are named opportunity costs. Furthermore, it was found to be difficult to differentiate between development and research costs, and we decided to define only the costs of the expert meetings and the scientific effect evaluations, including the compensation for the research activities of participating physicians, as research costs. Nevertheless, it is debatable whether these costs have to be accounted for, and researchers have to state their choice explicitly. In cost analysis research, such costs are usually excluded, and it is debatable how to handle development costs because some of these costs will be necessary when implementing a strategy on a broader scale. In conclusion, we evaluated costs and cost reductions of our strategy without counting the research and development costs. However, including the development costs in our sensitivity analysis did not change the results.
Surprisingly, cost effects have usually not been evaluated in quality improvement studies, perhaps because, as was mentioned above, many problems can be expected [25]. Recently, Mason et al. provided a framework for exploring the economics of improving quality of care by means of influencing physicians’ behavior, using clinical outcome data at the patient level [26]. The present paper provides a method for cost analyses of quality improvement strategies, where it is difficult or even impossible to perform a real cost-effectiveness study due to lack of clinical data.
In conclusion, in The Netherlands, the innovative test ordering strategy reveals considerable cost reductions in the first 6 months when not counting the opportunity costs for the time spent by primary care physicians. Because, contrary to the feedback strategy, non-monetary benefits can also be expected, we suggest that primary care physicians organizations encourage primary care physicians to participate in this new strategy.
|
Footnotes |
|---|
(
) Deceased. 
|
References |
|---|
|
TopMethodsResultsDiscussionReferences |
|---|
- Mason J, Eccles M, Freemantle N et al. A framework for incorporating cost-effectiveness in evidence-based clinical practice guidelines. Health Policy 1999; 47: 37–52.[CrossRef][Web of Science][Medline]
- Sculpher M. Evaluating the cost-effectiveness of interventions designed to increase the utilization of evidence-based guidelines. Fam Pract 2000; 17: 26–31.
- Mauskopf JA, Paul JE, Grant DM et al. The role of cost-consequence analysis in healthcare decision-making. Pharmacoeconomics 1998; 13: 277–288.[CrossRef][Web of Science][Medline]
- McIntosh E, Donaldson C, Ryan M. Recent advances in the methods of cost-benefit analysis in healthcare. Matching the art to the science. Pharmacoeconomics 1999; 15: 357–367.[CrossRef][Web of Science][Medline]
- Leurquin P, Van Casteren V, De Maeseneer J. Use of blood tests in general practice: a collaborative study in eight European countries. Eurosentinel Study Group. Br J Gen Pract 1995; 45: 21–25.[Web of Science][Medline]
- Kristiansen IS, Hjortdahl P. The general practitioner and laboratory utilization: why does it vary? Fam Pract 1992; 9: 22–27.
[Abstract/Free Full Text] - Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. Can Med Assoc J 1995; 153: 1423–1431.[Abstract]
- Bero LA, Grilli R, Grimshaw JM et al. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group. Br Med J 1998; 317: 465–468.
[Free Full Text] - Solomon DH, Hashimoto H, Daltroy L et al. Techniques to improve physicians’ use of diagnostic tests. A new conceptual framework. JAMA 1998; 280: 2020–2027.
[Abstract/Free Full Text] - Etter JF, Perneger TV. Health care expenditures after introduction of a gatekeeper and a global budget in a Swiss health insurance plan. J Epidemiol Commun Health 1998; 52: 370–376.[Abstract]
- Helgesen F, Andersson SO, Gustafsson O et al. Follow-up of prostate cancer patients by on-demand contacts with a specialist nurse: a randomized study. Scand J Urol Nephrol 2000; 34: 55–61.[CrossRef][Web of Science][Medline]
- Larsson A, Biom S, Wernroth ML et al. Effects of an education programme to change clinical laboratory testing habits in primary care. Scand J Prim Health Care 1999; 17: 238–243.[CrossRef][Web of Science][Medline]
- Wensing M, Grol R. Single and combined strategies for implementing changes in primary care: A literature review. Int J Health Care 1994; 6: 115–1132.
- Wensing M, Van der Weijden T, Grol R. Implementing guidelines and innovations in general practice: which interventions are effective? Br J Gen Pract 1998; 48: 991–997.[Web of Science][Medline]
- Thomson O’Brien MA, Oxman AD, Davis DA et al. Audit and feedback versus alternative strategies: effects on professional practice and health care outcomes. Cochrane Libr 2000; 2: CD000260.
- Mittman B, Tonesk X, Jacobson P. Implementing clinical practice guidelines: social influence strategies and practitioner behavior change. Qual Rev Bull 1992; 18: 413–422.
- Grol R. Successes and failures in the implementation of evidence-based guidelines for clinical practice. Med Care 2001; 39: 46–54.
- Verstappen WH, van der Weijden T, Sijbrandij J et al. Effect of a practice-based strategy on test ordering performance of primary care physicians: a randomized trial. JAMA 2003; 289: 2407–2412.
[Abstract/Free Full Text] - Little P, Cantrell T, Roberts L et al. Why do GPs perform investigations? The medical and social agendas in arranging back X-rays. Fam Pract 1998; 15: 264–265.
[Abstract/Free Full Text] - McDonald IG, Daly J, Jelinek VM et al. Opening Pandora’s box: the unpredictability of reassurance by a normal test result. Br Med J 1996; 313: 329–332.
[Abstract/Free Full Text] - Winkens RA, Grol RP, Beusmans GH et al. Does a reduction in general practitioners’ use of diagnostic tests lead to more hospital referrals? Br J Gen Pract 1995; 45: 289–292.[Web of Science][Medline]
- Buxton MJ, Drummond MF, Van Hout BA et al. Modelling in economic evaluation: an unavoidable fact of life. Health Econ 1997; 6: 217–227.[CrossRef][Web of Science][Medline]
- Tausch BD, Harter MC. Perceived effectiveness of diagnostic and therapeutic guidelines in primary care quality circles. Int J Qual Health Care 2001; 13: 239–246.
[Abstract/Free Full Text] - Spooner A, Chapple A, Roland M. What makes British general practitioners take part in a quality improvement scheme? J Health Serv Res Pol 2001; 6: 145–150.
[Abstract/Free Full Text] - Brown CA, Belfield CR, Field SJ. Cost effectiveness of continuing professional development in health care: a critical review of the evidence. Br Med J 2002; 324: 652–655.
[Free Full Text] - Mason J, Freemantle N, Nazareth I et al. When is it cost-effective to change the behavior of health professionals? JAMA 2001; 286: 2988–2292.
[Abstract/Free Full Text]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  C. A. McNulty, M. Thomas, J. Bowen, C. Buckley, A. Charlett, D. Gelb, C. Foy, J. Sloss, and S. Smellie Improving the appropriateness of laboratory submissions for urinalysis from general practice Fam. Pract., August 1, 2008; 25(4): 272 - 278. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. M Woods, E. J Thomas, J. L Holl, K. B Weiss, and T. A Brennan Ambulatory care adverse events and preventable adverse events leading to a hospital admission Qual. Saf. Health Care, April 1, 2007; 16(2): 127 - 131. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. J. Poley, K. I. Edelenbos, M. Mosseveld, M. A.M. van Wijk, D. H. de Bakker, J. van der Lei, and M. P.M.H. Rutten-van Molken Cost Consequences of Implementing an Electronic Decision Support System for Ordering Laboratory Tests in Primary Care: Evidence from a Controlled Prospective Study in The Netherlands Clin. Chem., February 1, 2007; 53(2): 213 - 219. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 T. Perneger Ten reasons to conduct a randomized study in quality improvement Int. J. Qual. Health Care, December 1, 2006; 18(6): 395 - 396. [Full Text] [PDF]
|
|
|
|
 |
|
 W. H. J. M. Verstappen, T. van der Weijden, W. I. Dubois, I. Smeele, J. Hermsen, F. E. S. Tan, and R. P. T. M. Grol Improving Test Ordering in Primary Care: The Added Value of a Small-Group Quality Improvement Strategy Compared With Classic Feedback Only Ann. Fam. Med, November 1, 2004; 2(6): 569 - 575. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||