Skip Navigation


International Journal for Quality in Health Care Advance Access originally published online on March 2, 2005
International Journal for Quality in Health Care 2005 17(3):235-242; doi:10.1093/intqhc/mzi027
This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
17/3/235    most recent
mzi027v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrow Search for citing articles in:
ISI Web of Science (12)
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Vlayen, J.
Right arrow Articles by Ramaekers, D.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Vlayen, J.
Right arrow Articles by Ramaekers, D.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?

International Journal for Quality in Health Care vol. 17 no. 3 © The Author 2005. Published by Oxford University Press on behalf of International Society for Quality in Health Care; all rights reserved

A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit

Joan Vlayen1,2, Bert Aertgeerts1,3, Karin Hannes1, Walter Sermeus2 and Dirk Ramaekers1,4

1 Belgian Centre for Evidence-Based Medicine, Leuven, Belgium 2 Catholic University Leuven, Centre for Health Services and Nursing Research, Leuven, Belgium 3 Catholic University Leuven, Academic Centre for General Practice, Leuven, Belgium, and 4 Belgian Federal Health Care Knowledge Centre, Brussels, Belgium

Objective. To identify a critical appraisal tool for clinical practice guidelines that could serve as a basis for the development of an appraisal tool for clinical pathways.

Design. Systematic review of the literature and personal contacts. Databases searched were: Medline, Embase, and Cinahl. Search terms were: practice guidelines, appraisal, and evaluation. The items of the identified appraisal tools were examined and thematically grouped into 10 guideline dimensions. Content analysis and scoring of these domains by the appraisal tools was evaluated.

Results. Twenty-four different appraisal tools of practice guidelines were identified. None scored the evidence base of the clinical content of guidelines. Four tools scored all the guideline dimensions. The Cluzeau instrument is the only one of these four that has been validated. Of the three instruments based on the Cluzeau instrument, the AGREE instrument is the only validated instrument that uses a numerical scale.

Conclusions. Being a simplified version of the Cluzeau instrument, the AGREE instrument has the most potential to serve as a basis for the development of an appraisal tool for clinical pathways. However, important limitations will have to be dealt with when developing such a tool.

Keywords: appraisal tool, clinical pathways, content analysis, practice guidelines

Address reprint requests to Joan Vlayen, Centre for Health Services and Nursing Research, Kapucijnenvoer 35, B-3000 Leuven, Belgium. E-mail: joan.vlayen{at}med.kuleuven.ac.be

Accepted for publication January 5, 2005.

In 1990, the Institute of Medicine (IOM) defined clinical practice guidelines as systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances [1]. Great variability exists in the quality of clinical practice guidelines [2,3], and numerous appraisal instruments were therefore developed in an attempt to discriminate high-quality from lower-quality guidelines. A recent review of the literature identified 13 appraisal instruments of practice guidelines, but concluded that evidence at that time was insufficient to support the exclusive use of just one appraisal instrument [4].

Selection of a decent guideline is one thing, but the selected guideline still has to be implemented. One way to implement clinical practice guidelines into daily practice is through clinical pathways. These are sequences of standardized multidisciplinary processes or critical interventions that must occur for a specific population towards the desired outcomes within a defined time period [5]. Originally, nursing care, organization, and cost-effectiveness were the most important aspects to be covered by clinical pathways. However, they also provide a means of improving systematic collection and abstraction of clinical data for audit and of promoting change in practice [6]. Increasingly, questions on the ability of clinical pathways to improve clinical quality, patient satisfaction, and the satisfaction of care providers are being researched.

Furthermore, it is unclear whether a relationship exists between the methodological and content quality of clinical pathways on the one hand and clinical quality, defined by the judicious and explicit use of the evidence from clinical trials, on the other. The instauration of several diagnostic and therapeutic interventions in a local clinical pathway based purely on the opinion of local specialists and without systematically taking the available up-to-date evidence for these interventions into consideration, can threaten the appropriateness and quality of care.

Clinical pathways show important similarities to clinical practice guidelines. They are both intended to provide appropriate and effective health care for specific clinical circumstances and to reduce variation in practice [1,6]. However, clinical pathways commonly integrate key aspects from several clinical practice guidelines, because they outline the care to be given for the patient’s entire clinical path rather than for one specific clinical situation [7]. Clinical practice guidelines are usually developed by government agencies, institutions, or expert panels, whereas clinical pathways are more local initiatives.

Unlike clinical practice guidelines, no (validated) instrument exists to assess the methodological quality of clinical pathways. A possible strategy to develop a critical appraisal tool for clinical pathways is to base it on an appraisal tool for clinical practice guidelines [7]. In the present article, the first phase of the development of such an appraisal tool for clinical pathways is described. In this first phase, we performed an update of the above-mentioned study of Graham et al. [4]. A systematic review of the literature was carried out to identify and compare existing critical appraisal tools for clinical practice guidelines. Possible scientific validation and international dissemination of the appraisal instruments were of particular concern in our review. In a later phase, one or more instruments will be selected and applied to clinical pathways, and serve as a basis for the development of an appraisal tool for clinical pathways.


   Methods
Top
 Methods
Results
 Discussion
 References
 
Search strategy
A literature search of the English and non-English literature indexed in the Ovid–Medline database (1966–October 2003), Embase database (1990–October 2003), and Cinahl database (1982–October 2003) was conducted, using the following MeSH and text terms in combination: practice guidelines, appraisal, evaluation. Methodology filters were not used, in order to conduct a sensitive literature search and because the usefulness of such filters is unclear for this particular subject. A manual search of the references of relevant articles was conducted. We also contacted the developers of the instruments identified to determine whether they were aware of any other instruments to include in our review.

All articles that described the evaluation of clinical practice guidelines or the development of a guideline appraisal tool were included. Articles describing a practice guideline or the development process of a practice guideline were excluded. No restriction was placed on abstracts, conference proceedings, or language. One investigator ( J.V.) assessed the selected papers and retrieved all the different tools that appraised the quality of clinical practice guidelines. Tools based completely on, or copied from, an existing tool were excluded.

Tools evaluation
The tools were compared for their general characteristics (source, items, scoring system). We also analysed whether items of specific importance in the development of clinical practice guidelines and clinical pathways were evaluated. One investigator (J.V.) therefore examined the questions/statements from all the instruments and thematically grouped them into separate guideline dimensions. Two other investigators (D.R., B.A.) independently performed a content analysis of the different instruments to assess whether or not they covered one or more items of each dimension. The scoring of the evidence base of the clinical content was also assessed. In cases of disagreement, the instruments were analysed and discussed in a small group (J.V., D.R., B.A.).


   Results
Top
 Methods
 Results
Discussion
 References
 
The Embase search yielded 11 523 articles of which 42 were selected. Of the Medline search another 41 of 2863 retrieved articles were selected. The Cinahl database contained 1187 articles, four being selected. A manual search of the bibliographies of the articles retrieved yielded another 11 relevant articles.

In the 98 articles retrieved, we identified 24 possible critical appraisal tools of guidelines. One tool was excluded from the review because it concerned an automated version [8] of another tool [9]. An additional instrument, not found through the systematic review of the literature or our personal contacts, was handled by one of the reviewers of this article [10].

General characteristics of the appraisal tools
Table 1 provides an overview of the characteristics of the 24 tools. The first appraisal tool was published in 1992 by Lohr and Field [11]. Before 1995, tools were exclusively published in North America [11–16]. Since 1995, appraisal tools have gained interest all over the world. Twenty-two tools were developed in eight different countries: six tools in the USA [11,12,15–18], five in Canada [13,14,19–21], four in the UK [9,22–24], two in Australia [25,26] and Italy [27,28], one in France [29], Germany [30], and Spain [31]. Two tools have been developed internationally [10,32].


View this table:
[in this window]
[in a new window]
  		Table 1 Characteristics of critical appraisal tools of guidelines

 

Eleven instruments [9,10,13,18,21,22,24,26,27,29,32] were based on the instrument developed by the IOM [11], three instruments [9,13,19] referred to Hayward et al. [14], another three instruments [23,30,32] referred to Cluzeau et al. [22].

The number of questions ranged from 3 to 52. Some questions were subdivided into two or more smaller questions. Nine tools used no specified scoring system, 10 tools used a yes/no score with or without the possibility of answering ‘not sure’ or ‘not applicable’ (Table 1). Five instruments used some kind of scaling system [17,18,25,26,32]. The instrument developed by Sanders [18] and the AGREE instrument [32] use a numerical scale.

All but three instruments were published in peer-reviewed literature. Only four instruments have been subject to a validation study: the instruments developed by Shaneyfelt [9], Cluzeau [22], and Grilli [28], and the AGREE instrument [32].

Content analysis
In total, a list of 469 questions or statements was generated from the 24 instruments. The common questions/statements were grouped into 50 different items (Table 2). These 50 items were then grouped into 10 guideline dimensions based on the work of the IOM [33] and the study of Graham et al. [4] (Table 2): validity, reliability/reproducibility, clinical applicability, clinical flexibility, multidisciplinary process, clarity, scheduled review, dissemination, implementation, and evaluation. All 50 items could be fit into the 10 dimensions.


View this table:
[in this window]
[in a new window]
  		Table 2 Items of the appraisal instruments grouped into 10 guideline dimensions

 

The two independent reviewers of the different instruments agreed completely on four instruments [9,22,23,26]. Disagreement existed on 0–5 dimensions per instrument (mean 2.2). The dimension clinical flexibility most frequently caused disagreement (10 instruments).

All the instruments evaluate the validity of guidelines with at least one item, almost all evaluate the clinical applicability (Table 3). Approximately 75% of the instruments score the dimensions reliability/reproducibility, clinical flexibility, scheduled review, and multidisciplinary process. Fourteen tools address dimension clarity. Only a minority scores the dimensions dissemination, implementation, and evaluation. Three appraisal tools (Table 3) score all the dimensions mentioned above by using at least one question [22,24,30]. None of the instruments score the evidence base of the clinical content of guidelines.


View this table:
[in this window]
[in a new window]
  		Table 3 Dimensions covered by the critical appraisal tools

 


   Discussion
Top
 Methods
 Results
 Discussion
References
 
In addition to the 13 instruments found by Graham et al. [4], we identified another 11 different tools for the critical appraisal of clinical practice guidelines. Comparison of these 24 instruments showed a wide variation in source, number of items, ways of scoring, and specific aspects that are scored. The questions used in the appraisal tools were grouped into 50 different items, which is slightly more than in the study of Graham et al. [4]. However, these 50 items could easily be fitted into the 10 guideline dimensions used by Graham et al.

Three appraisal tools were found to address all the guideline dimensions [22,24,30]. Of these, the Cluzeau instrument [22] is the only instrument that has been subject to a thorough validation study. It was originally based on the instrument developed by the IOM [11] and contains 37 items divided into three dimensions: rigour of development (questions 1–20), context and content (questions 21–32), and application (questions 33–37). A yes/no score is used to respond to each question.

Three additional instruments are based on the Cluzeau instrument [23,30,32]. Of these, the AGREE instrument [32] is the only instrument that has been validated. It uses a numerical scoring scale, making it easier to compare scores. It is more compact than the Cluzeau instrument, containing only 23 items divided into six domains: scope and purpose, stakeholder involvement, rigour of development, clarity and presentation, applicability, and editorial independence. Unlike the Cluzeau instrument, the dimension dissemination is not scored in AGREE. Because the AGREE instrument is a validated, easy-to-use, and transparent instrument, which was internationally developed and widely accepted, it can possibly serve as a basis for an instrument to evaluate the methodological quality of clinical pathways. English investigators have already reported on an appraisal tool, the Integrated Care Pathway Appraisal Tool (ICPAT) [7], which is based on the AGREE instrument, but is yet to be validated.

There are some important limitations in the use of the AGREE instrument. Firstly, the domain scores are useful for comparing clinical practice guidelines, but it is not possible to set thresholds for the scores to classify a clinical practice guideline as ‘good’ or ‘bad’. Secondly, the AGREE instrument does not assess the clinical content of the clinical practice guideline nor the quality of evidence supporting the recommendations, which is a common deficit in all the existing appraisal tools. The use of a systematic methodology in the retrieval of evidence supporting guideline development is frequently scored by appraisal tools [9,11,13,14,16,17,19–22,24–26,30–32]. However, even by using a recent and advanced instrument such as the Cluzeau instrument or the AGREE instrument, the results of the search for evidence, the correct use of inclusion and exclusion criteria, and the critical appraisal of the retrieved evidence are not validated. Therefore, a major conclusion of this review is that in order to evaluate the quality of the clinical content and more specifically the evidence base of a clinical practice guideline, verification of the completeness and the quality of the literature search and its analysis has to be added to the process of validation by an appraisal instrument. Experience with the methodologies of evidence-based medicine such as literature search and critical appraisal is therefore essential for guideline validators to assure the quality of the appraisal process. Blind application of an appraisal instrument, even when validated and widely implemented, without particular attention to the evidence supporting the guideline, can threaten the credibility of these instruments and the current evolution in the international community to further elaborate the quality of guideline development [34].

Because of the differences between clinical pathways and clinical practice guidelines, the AGREE instrument cannot be applied to clinical pathways using the present version. Some items will have to be reformulated or removed, new items will have to be included. For example, the language used in the AGREE instrument is clearly ‘guideline language’ and will have to be translated into ‘pathway language’: for example, clinical pathways contain concrete interventions rather than recommendations, clinical pathways are implemented rather than published. At present, our team is composing a development group consisting of experts in clinical pathway development and experts in clinical practice guideline development and validation to create a version of the AGREE instrument applicable to clinical pathways. The development of this instrument will be described in a subsequent publication.

In conclusion, 24 different appraisal tools of clinical practice guidelines were identified. Of these tools, the Cluzeau instrument seems to be the most complete. Being a more compact version of the Cluzeau instrument and using a numerical scale, the AGREE instrument has the potential to serve as a basis for a critical appraisal tool for clinical pathways. However, some important limitations of the AGREE instrument will have to be dealt with when developing such a tool.


   References
Top
 Methods
 Results
 Discussion
 References
 

  1. Institute of Medicine Committee to Advise the Public Health Service on Clinical Practice Guidelines. Clinical Practice Guidelines: Directions For A New Program. Washington DC: National Academy Press, 1990.

  2. Saturno PJ, Medina F, Valera F, Montilla J, Escolar P, Gascon JJ. Validity and reliability of guidelines for neck pain treatment in primary health care. A nationwide empirical analysis in Spain. Int J Qual Health Care 2003; 15: 487–493.[Abstract/Free Full Text]

  3. Christiaens T, De Backer D, Burgers J, Baerheim A. Guidelines, evidence, and cultural factors. Scand J Prim Health Care 2004; 22: 141–145.[CrossRef][ISI][Medline]

  4. Graham ID, Calder LA, Hebert PC, Carter AO, Tetroe JM. A comparison of clinical practice guideline appraisal instruments. Int J Technol Assess Health Care 2000; 16: 1024–1038.[CrossRef][ISI][Medline]

  5. Nyberg D, Marschke P. Critical pathways: tools for continuous quality improvement. Nurs Adm Q 1993; 17: 62–69.

  6. Campbell H, Hotchkiss R, Bradshaw N, Porteous M. Integrated care pathways. Br Med J 1998; 316: 133–137.[Free Full Text]

  7. de Luc K, Whittle C. An integrated care pathway appraisal tool: a ‘badge of quality’. Journal of Integrated Care Pathways 2002; 6: 13–17.

  8. Agrawal A, Shiffman RN. Evaluation of guideline quality using GEM-Q. Medinfo 2001; 10: 2–101.

  9. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 1999; 281: 1900–1905.[Abstract/Free Full Text]

  10. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J, Deshpande AM. Standardized reporting of clinical practice guidelines: a proposal from the conference on guideline standardization. Ann Intern Med 2003; 139: 493–498.[Abstract/Free Full Text]

  11. Lohr K, Field M. eds. A provisional instrument for assessing clinical practice guidelines. In Guidelines for Clinical Practice: From Development to Use. Washington DC: National Academy Press, 1992.

  12. Selker HP. Criteria for adoption in practice of medical practice guidelines. Am J Cardiol 1993; 71: 339–341.[CrossRef][ISI][Medline]

  13. Hayward RS, Wilson MC, Tunis SR, Bass EB, Guyatt G. Users’ guides to the medical literature. VIII. How to use clinical practice guidelines. A. Are the recommendations valid? The Evidence-Based Medicine Working Group. JAMA 1995; 274: 570–574.[CrossRef][ISI][Medline]

  14. Hayward RS, Wilson MC, Tunis SR, Bass EB, Rubin HR, Haynes RB. More informative abstracts of articles describing clinical practice guidelines. Ann Intern Med 1993; 118: 731–737.[Abstract/Free Full Text]

  15. Mendelson EB. The development and meaning of appropriateness guidelines. Radiol Clin North Am 1995; 33: 1081–1084.[ISI][Medline]

  16. Woolf SH. Practice guidelines: what the family physician should know. Am Fam Physician 1995; 51: 1455–1463.[ISI][Medline]

  17. Reed GM, McLaughlin CJ, Newman R, American Psychological Association. American Psychological Association policy in context. The development and evaluation of guidelines for professional practice. Am Psychol 2002; 57: 1041–1047.[CrossRef][Medline]

  18. Sanders GD, Nease RF, Jr, Owens DK. Design and pilot evaluation of a system to develop computer-based site-specific practice guidelines from decision models. Med Decis Making 2000; 20: 145–159.[Abstract/Free Full Text]

  19. Calder L, Hébert P, Carter A, Graham I. Review of published recommendations and guidelines for the transfusion of allogeneic red blood cells and plasma. CMAJ 1997; 156: S1–S8.

  20. Marshall JK. A critical approach to clinical practice guidelines. Can J Gastroenterol 2000; 14: 505–509.[ISI][Medline]

  21. Savoie I, Green C, Bassett K et al. Critical Appraisal Criteria for Clinical Practice Guidelines. Vancouver: BC Office of Technology Assessment, 1996.

  22. Cluzeau FA, Littlejohns P, Grimshaw JM, Feder G, Moran SE. Development and application of a generic methodology to assess the quality of clinical guidelines. Int J Qual Health Care 1999; 11: 21–28.[Abstract/Free Full Text]

  23. Hutchinson A, McIntosh A, Anderson J, Gilbert C, Field R. Developing primary care review criteria from evidence-based guidelines: coronary heart disease as a model. Br J Gen Pract 2003; 53: 690–696.[ISI][Medline]

  24. Scottish Intercollegiate Guidelines Network (SIGN). Clinical Guidelines: Criteria for Appraisal for National Use. Edinburgh: Royal College of Physicians, 1995.

  25. Liddle J, Williamson M, Irwig L. Method for Evaluating Research Guidelines Evidence (MERGE). Sydney: New South Wales Department of Health, 1996.

  26. Ward JE, Grieco V. Why we need guidelines for guidelines: a study of the quality of clinical practice guidelines in Australia. Med J Aust 1996; 165: 574–576.[ISI][Medline]

  27. Apolone G, Bamfi F. A model for an evaluation of clinical guidelines (in Italian). Ric Pract 1999; 15: 116–124.

  28. Grilli R, Magrini N, Penna A, Mura G, Liberati A. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet 2000; 355: 103–106.[CrossRef][ISI][Medline]

  29. Mottur-Pilson C. Internists’ evaluation of guidelines: the IMCARE Practice Guidelines Network. Int J Qual Health Care 1995; 7: 31–37.[Abstract/Free Full Text]

  30. Helou A, Ollenschlager G. Goals, possibilities and limits of quality evaluation of guidelines. A background report on the user manual of the ‘Methodological Quality of Guidelines’ check list (in German). Z Arztl Fortbild Qualitatssich 1998; 92: 361–365.[Medline]

  31. Casi CA, Aizpuru BF, Ibanez PF, Grupo de Trabajo sobre Calidad de las Guias de Practica Clinica en Atencion Primaria de la Sociedad Vasca de Medicina de Familia. Quality analysis of clinical practice guidelines on cardiovascular risks in the Basque country and a proposal for their evaluation (in Spanish). Aten Primaria 2000; 26: 585–589.[Medline]

  32. AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003; 12: 18–23.[Abstract/Free Full Text]

  33. Institute of Medicine. Guidelines for Clinical Practice: From Development to Use. Washington, DC: National Academy Press, 1992.

  34. Guidelines International Network: http://www.g-i-n.net/ Accessed 2 October 2004.


Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us    What's this?


This article has been cited by other articles:


Home page
 Clin. Chem.Home page
J. S. Burgers
Guideline Quality and Guideline Content: Are They Related?
Clin. Chem., January 1, 2006; 52(1): 3 - 4.
[Full Text] [PDF]

Home page
 Clin. Chem.Home page
J. Watine, B. Friedberg, E. Nagy, R. Onody, W. Oosterhuis, P. S. Bunting, J.-C. Charet, and A. R. Horvath
Conflict between Guideline Methodologic Quality and Recommendation Validity: A Potential Problem for Practitioners
Clin. Chem., January 1, 2006; 52(1): 65 - 72.
[Abstract] [Full Text] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
17/3/235    most recent
mzi027v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrow Search for citing articles in:
ISI Web of Science (12)
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Vlayen, J.
Right arrow Articles by Ramaekers, D.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Vlayen, J.
Right arrow Articles by Ramaekers, D.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?