OUP user menu

Impact of end user involvement in implementing guidelines on routine pre-operative tests

FABRIZIO BARAZZONI, ROBERTO GRILLI, ANNA MARIA VINCENZA AMICOSANTE, SONIA BRESCIANINI, MICHELE A. MARCA, MARCO BAGGI, PAUL BIEGGER, REZIO RENELLA
DOI: http://dx.doi.org/10.1093/intqhc/14.4.321 321-327 First published online: 1 August 2002

Abstract

Objectives. To assess the impact of health professionals’ involvement in the implementation of practice guidelines aimed at reducing the use of pre-operative tests in patients at low anaesthetic risk undergoing elective surgery.

Intervention. A 6 month (September 1997 to February 1998) strategy based upon organization of local meetings involving health professionals from six hospitals of Canton Ticino (Switzerland).

Design. Observational study (pre/post) of pre-operative test utilization between March 1996 and December 1998.

Subjects and methods. A total of 17 978 patients admitted for elective surgery over the study period. The latter was modelled in six intervals, three before (baseline), one during, and two after (adoption) guidelines implementation, respectively. For each time interval the proportion of patients undergoing pre-operative tests was estimated. Multilevel logistic regression analysis was used to assess patient likelihood [expressed as the odds ratio (OR)] of undergoing a diagnostic test in each period, using the implementation interval as the reference category.

Main outcome measure. Change in patient probability of undergoing pre-operative tests in the adoption interval.

Results. Adoption of the recommendations was associated with 81% [OR = 0.19; 95% confidence interval (CI) 0.15–0.23] reduction of patient probability of undergoing coagulation test, 73% (OR = 0.27; 95% CI 0.23–0.33) for glycaemia, 62% (OR = 0.38; 95% CI 0.33–0.44) for azotaemia, 57% (OR = 0.43; 95% CI 0.36–0.51) for chest X-ray, 49% (OR = 0.51; 95% CI 0.44–0.60) for creatinaemia, and 43% (OR = 0.57; 95% CI 0.48–0.69) for ECG. Overall, these findings corresponded to a cost saving of 67 890 Swiss francs (US$42 000) for the last quarter under study.

Conclusions. This study indicates that an implementation strategy based upon direct involvement of end users in the identification of possible barriers to change can be successful in promoting the use of practice guidelines.

  • implementation
  • practice guidelines
  • pre-operative tests

A great deal of research has been devoted to studying the effectiveness of different strategies aimed at changing health professionals’ performance, and their findings have been summarized by a number of systematic reviews [1].

Overall, the conclusions indicate that there is no single effective intervention, rather a plurality of approaches can be effective as long as they are tailored to the specific characteristics of the clinical environment representing a barrier to the desired change.

In this paper we report the results of the implementation of recommendations aimed at reducing the use of pre-operative tests in the hospitals of Canton Ticino, Switzerland. This issue has been the object of systematic reviews and practice guidelines developed over the last 15 years by specialist societies and technology assessment agencies [26], in an attempt to reduce a practice that seems to be as much widespread as it is unsupported by empirical evidence [7, 8].

Current research on practice guidelines indicates the relevance of the identification of barriers to the adoption of the recommended behaviours [9]. In principle, end-users (that is, those by whom practice guidelines are intended to be used) are in the best position to identify contextual factors that may hinder or favour their adherence to guidelines in current clinical practice. In our study we used an implementation strategy, relying as much as possible on the involvement of guideline end-users and on the identification of the obstacles to the adoption of the recommendations.

Materials and methods

Study design

This study is based upon the analysis of the patterns of use of pre-operative tests before, during, and after the implementation of guidelines aimed at reducing the use of unnecessary routine tests.

Study setting

Six (later reduced to five, as one was closed during the study period) acute care urban community hospitals with surgery facilities exist in Canton Ticino, serving approximately 310 000 residents, and accounting for 1049 acute beds in the public sector and 316 doctors. Facilities for general surgery, orthopaedics, neurosurgery, gynaecology, urology, and ophthalmology are available, and all of the hospitals host training activities.

Development of the recommendations

A number of guidelines on the use of pre-operative testing have been developed, including those issued by an Italian national consensus conference on this topic [10], held under the auspices of several specialist societies. In the framework of that conference, a multidisciplinary panel of experts, also including patient and consumer representatives, developed recommendations for adult patients undergoing elective surgery on the basis of a systematic review on the performance of different diagnostic tests in the pre-operative setting, and on the critical appraisal of pre-existing guidelines. The final recommendations were adopted by the medical chief of the Canton Ticino public hospital organization.

The recommendations were not intended to be mandatory, rather they were presented to health professionals as ‘advised clinical behaviours’, and were not linked to any kind of financial incentive.

The implementation strategy

The implementation of the recommendations was based upon the organization of local meetings. Multidisciplinary working groups were convened by the medical chief of the Canton Ticino public hospital organization in each participating hospital, inviting representatives from the different categories of health professionals involved (clinicians, nurses, managers). The main task of the working groups was to examine the clinical content of the recommendations and anticipate their organizational implications, taking into account the settings in which they had to be applied. Participants in the working groups were volunteers, although every effort was made to assure the attendance of department/ward leaders. The discussion focused on the identification of possible barriers to the adoption of the guidelines by clinicians, as well as on the development of feasible strategies to overcome them.

In these meetings, two issues consistently emerged: (i) concerns about the legal implications of a policy that would restrict the use of pre-operative tests; and (ii) the need for organizational changes in the anaesthesiology services. As far as the first is concerned, clinicians claimed that routine use of pre-operative tests had a ‘protective’ effect on health care providers in case of adverse events that could lead to legal action by patients or their relatives.

As for the organizational barriers, the content of the guidelines called for the design of a pre-operative process of care structured in such a way as to allow an anaesthetic work-up based upon a rigorous clinical examination. The results of the tests ordered had then to be seen by the anaesthesiologists before surgery. This policy, requiring a selective ordering of specific tests according to patient clinical history and findings at the clinical examination, appeared to be more complex than the usual one of simply ordering unselectively ‘everything for everybody’ when the decision of referring the patient for elective surgery was made.

The legal issue was addressed through the involvement of the Swiss Institute of Health Rights, who were asked to produce a report on the legal implications of practice guidelines [11]. The content of the report made explicit that guidelines were not to be considered per se either a protection or a danger for the practising clinician, the key issue being the ability of the latter to justify his decision through clear and complete clinical documentation.

The organizational aspects were addressed by changing the procedures at the time of hospital admission or when a surgical intervention was scheduled. In particular, a manual paper reminder was designed in the form of a patient profile checklist that had to be used by a physician or a nurse in patients admitted for elective surgery to order the required pre-operative tests. According to the presence of specific patients’ characteristics (recorded ticking a box), the reminder suggested the type of diagnostic tests recommended by the guidelines. The development of such a tool was suggested by some clinicians in the discussions held at the local meetings.

Adoption of the guidelines

The clinical recommendations were officially adopted on March 1998, and after that were considered as the suggested clinical policy that surgical departments were encouraged to apply.

Evaluation of the impact of the guidelines

The evaluation of the impact of the guidelines was based on the analysis of the patterns of use of pre-operative tests from March 1996 to December 1998. For each patient, administrative data on clinical characteristics [including the severity of the anaesthetic risk, expressed through the American Society of Anesthesiology (ASA) scale] and the type of pre-operative diagnostic tests undertaken were available.

Changes in the number of patients undergoing each individual pre-operative test were assessed, dividing the overall period of observation up into six time intervals: three semesters (baseline, from March 1996 to August 1997) before the inception of the implementation, a fourth representing the implementation period (September 1997 to February 1998), and two following intervals representing the adoption phase (March to August 1998 and September to December 1998). For each interval the proportion of patients receiving pre-operative tests was then estimated.

A logistic regression model was employed in order to assess individual patients’ probability of receiving a test within a specific time interval [12]. In order to account for lack of independence among observations (i.e. individual patients cared for at the same hospital), a multilevel hierarchical model was adopted [13, 14]. With this approach, the violation of the independence of observations assumption is compensated by analysing the effect at interest simultaneously within different hierarchical levels of aggregation (in this case represented by hospitals and patients, respectively). In this study we analysed the variation within the hospital level (second level).

Variables included in the best fitting model were age (categorized in classes), anaesthetic risk according to the ASA classification (dummy variables, with ASA class III as the reference category), and time (dummy variables, with the semester of implementation, September 1997 to February 1998, representing the reference category).

The effect of the implementation strategy is expressed in terms of patient probability of undergoing a pre-operative test represented by an odds ratio (OR) with 95% confidence interval (CI). Therefore, if the intervention is effective, the OR is <1 for the periods March to August 1998 and September to December 1998.

To gain some insight into the impact of the guidelines on resources, fees used were a proxy of the individual test cost multiplied by its frequency of use, before and after the implementation.

Results

Overall, 19 965 administrative records of patients admitted at the six public hospitals of Canton Ticino over the whole study period were available. After exclusion of records from patients aged between 0 and 15 years (as paediatric patients were not considered in the guidelines) and those admitted for surgical interventions in emergency, 17 978 remained. As 705 patients were admitted at the study hospitals more than once (thus accounting for more than one administrative record), only their first episode of care was considered for the analysis. Overall, at the end of this selection process, 17 273 patients were included in the analysis.

Utilization rates of pre-operative tests

The patient distribution by age and ASA class is shown in Table 1. Over the study period there was a decrease of patients aged 16–35 years and an increase in the number of patients aged 36–65 years. Furthermore, there has been a decrease in patients with low anaesthetic risk (ASA I) and an increase for the other two categories. The implementation phase showed the highest proportion of elders (40% of patients were >65 years old) of the whole study period, whereas the other two classes had the lowest proportions (20% for age 16–35 years and 40% for 36–65 years). As for the anaesthetic risk, in that semester, about half the patients were ASA class II, and a third were ASA class I, while the remaining (about 18%) were ASA class III.

View this table:
Table 1

Patient age and ASA class distribution over time

Age class (years) (%)ASA class (%)
Period n 15–3536–65≥65IIIIII
Baseline
Mar. 1996–Aug. 1996 435225.443.131.549.433.617.0
Sept. 1996–Feb. 1997 436324.943.831.341.838.220.0
Mar. 1997–Aug. 1997 283822.942.135.034.041.724.4
Implementation
Sept. 1997–Feb. 1998 119320.040.539.633.148.918.0
Adoption
Mar. 1998–Aug. 1998 233722.845.132.131.244.024.8
Sept. 1998–Dec. 1998 219023.744.731.636.639.623.8
Total
Mar. 1996–Dec. 199817 27323.943.532.739.739.321.0
  • ASA, American Society of Anesthesiologists.

The proportion of patients undergoing each pre-operative test is shown in Table 2. Compared with the baseline period, there has been a decrease in the adoption phase of ∼20% for all the tests (22% for creatinaemia and glycaemia, 21% for chest X-rays, and 20% for coagulation test) apart from azotaemia and ECG, for which the reduction was, respectively, 17% and 11%.

View this table:
Table 2

Proportion of patients undergoing pre-operative tests by study period

Proportion of patients undergoing pre-operative tests (%)
Period n AzotaemiaCoagulation testCreatinaemiaGlycaemiaECGChest X-ray
Mar. 1996–Aug. 1996 435248.193.082.486.761.757.4
Sept. 1996–Feb. 1997 436350.591.280.084.362.856.6
Mar. 1997–Aug. 1997 283855.492.078.484.063.458.9
Sept. 1997–Feb. 1998 119349.090.867.983.362.051.1
Mar. 1998–Aug. 1998 233738.379.362.167.755.141.5
Sept. 1998–Dec. 1998 219029.364.854.558.948.332.0
Baseline11 55350.892.180.585.162.657.5
Implementation 119349.090.867.983.362.051.1
Adoption 452734.072.358.463.451.836.9
  • ECG, electrocardiogram.

These findings were confirmed when data were analysed at the individual patient level through the multilevel logistic regression model. Table 3 shows the parameter estimates for what we believe to be the best model for our data (statistically significant G test, P < 0.0001, when compared with the model with fixed intercept). In this model we allow the intercept to vary randomly among hospitals. Only the systematic component coefficients are reported in Table 3. For all the tests the OR shows a significant decrease in the probability of undergoing such tests after the guideline adoption phase. For some of these, like coagulation test and glycaemia, the effect was evident from the beginning of the adoption phase, while for the remaining it appeared later in the study period. At the end of the period under study there was a reduction in the probability of undergoing each pre-operative test: 81% (OR = 0.19; 95% CI 0.15–0.23) reduction of patient probability of undergoing coagulation test; 73% (OR = 0.27; 95% CI 0.23–0.33) for glycaemia; 62% (OR = 0.38; 95% CI 0.33–0.44) for azotaemia, 57% (OR = 0.43; 95% CI 0.36–0.51) for chest X-ray; 49% (OR = 0.51; 95% CI 0.44–0.60) for creatinaemia; and 43% (OR=0.57; 95% CI 0.48–0.69) for ECG.

View this table:
Table 3

Impact of guidelines introduction on utilization of pre-operative routine tests: results from the multilevel logistic regression

OR (95% CI)
Independent variablesAzotaemiaCreatinaemiaECGCoagulation testGlycaemiaChest X-ray
Intercept1.48 (0.63–3.50)2.98 (1.71–5.18) 0.19 (0.08–0.46)7.08 (4.16–12.05)3.03 (1.61–5.70) 0.25 (0.12–0.54)
ASA I0.48 (0.44–0.53)0.36 (0.32–0.40) 0.73 (0.65–0.82)0.45 (0.39–0.52)0.50 (0.44–0.56) 0.35 (0.31–0.39)
ASA II0.64 (0.58–0.70)0.47 (0.42–0.53) 0.98 (0.87–1.10)0.76 (0.66–0.89)0.75 (0.67–0.85) 0.52 (0.47–0.58)
ASA III11 111 1
15–25 years11 111 1
25–35 years0.95 (0.84–0.10)0.95 (0.83–1.09) 1.61 (1.36–1.92)1.19 (1.02–1.40)0.94 (0.82–1.08) 1.25 (1.04–1.51)
35–45 years1.13 (0.99–1.30)1.27 (1.10–1.47) 6.01 (5.09–7.10)1.58 (1.33–1.87)1.30 (1.13–1.50) 3.78 (3.17–4.50)
45–55 years1.41 (1.23–1.61)1.97 (1.69–2.29)23.48 (19.67–28.03)2.06 (1.73–2.45)2.20 (1.90–2.55)12.68 (10.63–15.12)
55–65 years1.74 (1.53–1.99)2.56 (2.19–2.98)33.25 (27.69–39.92)3.17 (2.62–3.83)3.91 (3.33–4.58)23.59 (19.69–28.28)
65–75 years2.04 (1.78–2.34)2.33 (2.00–2.71)38.24 (31.72–46.12)4.15 (3.38–5.09)5.92 (4.98–7.05)26.52 (22.07–31.88)
75–85 years1.95 (1.69–2.24)1.46 (1.25–1.71)46.76 (38.17–57.29)4.39 (3.51–5.50)6.88 (5.66–8.37)25.48 (21.06–30.84)
Over 85 years1.94 (1.61–2.33)1.43 (1.17–1.76)46.67 (35.55–61.26)5.33 (3.71–7.64)8.12 (5.96–11.06)28.45 (22.23–36.39)
Mar. 1996–Aug. 19961.14 (1.00–1.31)2.36 (2.03–2.75) 1.15 (0.97–1.37)1.60 (1.27–2.03)1.30 (1.08–1.56) 1.80 (1.53–2.11)
Sept. 1996–Feb. 19971.14 (1.00–1.30)1.97 (1.70–2.28) 1.24 (1.05–1.48)1.16 (0.93–1.46)1.10 (0.92–1.32) 1.65 (1.41–1.93)
Mar. 1997–Aug. 19971.22 (1.06–1.41)1.82 (1.56–2.14) 1.27 (1.06–1.52)1.27 (0.99–1.62)1.25 (1.03–1.52) 1.73 (1.46–2.05)
Sept. 1997–Feb. 199811 111 1
Mar. 1998–Aug. 19980.57 (0.49–0.65)0.70 (0.60–0.82) 0.78 (0.65–0.94)0.40 (0.32–0.50)0.41 (0.34–0.50) 0.67 (0.57–0.80)
Sept. 1998–Dec. 19980.38 (0.33–0.44)0.51 (0.44–0.60) 0.57 (0.48–0.69)0.19 (0.15–0.23)0.27 (0.23–0.33) 0.43 (0.36–0.51)
  • ASA, American Society of Anesthesiologists; ECG, electrocardiogram; OR, odds ratios.

As expected, the patient’s age and ASA class emerged as being associated with the use of pre-operative tests, which were used less frequently in patients younger and with lower anaesthetic risk.

In addition, the effect of the implementation was more evident in small hospitals (i.e. with <200 beds), as shown in Figure 1. The difference was statistically significant for creatinaemia (OR = 0.72 and 0.31 for large and small hospitals, respectively) and azotaemia (OR = 0.58 and 0.31 for large and small hospitals, respectively). For all the other tests the difference was smaller, although large hospitals always showed higher ORs.

Figure 1

Effect of guidelines introduction by hospital size. ORs (post-implementation period September 1998 to December 1998 versus implementation period September 1997 to February 1998) and 95% CIs.

Discussion

The growing body of literature on practice guidelines consistently indicates that their effect is dependent upon the adoption of the appropriate implementation strategy, according to the cultural, administrative, and organizational characteristics of the clinical environment [1, 9, 1517]. The message was similar when test ordering was addressed specifically [18].

Routine indiscriminate use of pre-operative tests in patients at low anaesthetic risk undergoing elective surgery is indeed a typical case of professional behaviour that is unsupported on scientific grounds [7], but not easily amenable to change because of the weight of contextual factors. Fear of malpractice accusation by patients or their relatives is often quoted among the reasons for ordering these tests, despite their low predictive value [19]. The assumption is that if many tests are ordered before performing a surgical intervention, the clinician will be less likely to be sued in the case of a negative outcome.

Organizational aspects [20] are also included among the issues at stake, with patient pre-operative process of care organized in order to allow a thorough clinical examination as well as timely communication of information. Routine prescription of an array of predefined diagnostic tests is easier than adopting the more critical attitude of ordering a test only when it is considered to be justified on the grounds of a careful clinical examination.

While all the above can explain the failure of attempts aimed at modifying such an apparently simple behaviour, our findings confirm that local strategies explicitly taking those factors into account can be successful.

In the context of our study, the guidelines were introduced by local meetings and aimed at examining clinicians concerns and how they could be addressed. The official statement of the Swiss Institute of Health Rights was instrumental at pointing out that mere performance of a diagnostic pre-operative test was not sufficient to avoid accusation of negligence, if a thorough clinical examination—highlighted as the key issue—was not performed. On the other hand, uncritical adherence to a guideline recommending not to use pre-operative tests routinely was considered equally insufficient to prevent legal actions if the decision to follow the recommendations could not be justified on clinical grounds.

The organizational issues were also considered in the discussions held at the local meetings, leading to the adoption of a number of practical proposals, in particular the development of a test ordering form that explicitly outlines the type of diagnostic tests to be ordered according to patient characteristics.

Overall, the observed reduction in use was quite homogeneous across the different pre-operative tests, always approximately 20%, the only exceptions being ECG and azotaemia, the reductions of which were relatively smaller (11 and 17%, respectively).

It is worth noting that the effect of the implementation was not homogeneous across hospitals, being more evident in the four smaller (i.e. <200 beds) centres than in the two of larger ones. This finding, which was independent of differences in case mix, indicates how the promotion of behavioural changes can be relatively more difficult in centres with a higher degree of organizational complexity.

Lastly, one has to be certain that the adoption of the recommended behaviour is not accompanied by unintended effects on patient outcomes and health care resources.

Consistent with other studies [20, 21], the decrease in routine pre-operative tests did not increase the rates of any adverse event, and over the whole study period no change in pre- and post-operative mortality rates was observed. In addition, the reduction in the number of tests ordered appeared to be largely confined within patients at low anaesthetic risk (i.e. ASA I–II; see Table 3, where these categories consistently show a lower probability of receiving pre-operative tests when compared with patients at a higher risk).

As far as resources are concerned, considering only the cost to the hospitals of each individual test, we estimated that the economic impact of the guidelines implementation is about 31 Swiss francs (US$19.5) saved per admission, accounting for an overall saving of 67 890 Swiss francs (US$42 000) in the last quarter of the study period.

One has to be aware that the design of this study is observational, and therefore the results should be interpreted with caution as they could be a result of factors other than the intervention at stake, i.e. to a pre-existing trend in the use of pre-operative tests. However, the data from the months prior to the implementation of the guidelines show that the patterns of the diagnostic tests considered were stable, and consistently higher than those observed thereafter. Furthermore, experimental designs are often hard to apply in the evaluation of the impact of clinical policies, and this is particularly true in the context of studies like this one, where one has to deal with hospitals very close to each other in geographical as well as in organizational terms, and therefore exposed to a risk of contamination that is not avoidable through the use of a random allocation. We adopted a time series design, which, although not as robust as an experimental one, through the reliance on several observations before and after the intervention of interest is nevertheless more reliable than a simple before–after comparison [22].

Conclusions

Although of observational design, and therefore with the usual caveats implied in interpreting the findings, this study supports the view that local implementation of guidelines can be successful when accomplished through a strategy based upon the active involvement of health professionals. Explicit consideration of their concerns as well as the sense of ownership, possibly stimulated by their active participation, could explain the positive results of this study.

The implementation strategy adopted was indeed simple, based upon the organization of local meetings and the adoption of a manual paper reminder. While we cannot disentangle the effect of these two components and understand their relative weight in determining the observed changes, this overall effect proved nevertheless to be effective in our analysis.

In conclusion, while this study supports the view that implementation strategies have to take local circumstances into account in order to be effective, it also shows that improving professional practice does not necessarily need to be complex and expensive, and that substantial changes can be achieved with simple tools.

Acknowledgements

We thank the anonymous referees for their useful comments on a previous version of the manuscript.

Footnotes

  • Address reprint requests to Fabrizio Barazzoni, Ente Ospedaliero Cantonale, Viale Officina 3, 6501 Bellinzona, Switzerland. E-mail:fabrizio.barazzoni{at}eoc.ch

REFERENCES

View Abstract