OUP user menu

Predictors of high quality clinical practice guidelines: examples in oncology

Beatrice Fervers, Jako S. Burgers, Margaret C. Haugh, Melissa Brouwers, George Browman, Francoise Cluzeau, Thierry Philip
DOI: http://dx.doi.org/10.1093/intqhc/mzi011 123-132 First published online: 21 January 2005

Abstract

Objective. Clinical practice guidelines are widely used as effective tools for improving the management of patients with cancer. However, there is increasing concern about variation in guideline quality. In this study we identified predictors for high-quality guidelines in oncology.

Design. The quality scores for 32 oncology guidelines from 13 countries were determined by four independent appraisers using the Appraisal of Guidelines for Research and Evaluation (AGREE) instrument.

Main measures. The contribution to the quality score of six characteristics of guidelines and three of guideline developing organizations was then assessed using analysis of variance and stepwise linear regression analysis.

Results. Some guideline and organizational characteristics were shown to be responsible for a large part of the variations in quality scores. The availability of background information was the strongest predictor of quality with an explained variance ranging from 17% (‘Applicability’) to 67% (‘Rigour of development’). High-quality guidelines were more often produced by government-supported organizations and/or within a structured, coordinated programme. The other characteristics (publication year, type of guideline, format, level of care, and scope) were not independent predictors of quality.

Conclusions. Guidelines should provide more explicit information about the context of their development and methods used in order to improve their quality and thus encourage their use in clinical practice.

  • clinical practice guidelines
  • international
  • network
  • oncology quality assessment

Introduction

Evidence-based clinical practice guidelines are widely used to promote effective and efficient health care [1]. Numerous guideline development programmes have been set up at local, regional, national and international levels. In the field of oncology the results from several studies show that guidelines can improve the care process, as well as improve patient outcomes [24]. Moreover, there is increasing concern about the variation of guideline quality and recommendations [5,6]. If guidelines are to improve the quality of health care they must be credible, in order to inspire confidence in prospective users; guideline quality is an important contribution to their credibility. The present study aims to identify predictors of high quality for clinical practice guidelines in oncology. These will be important to guideline users to help them identify credible guidelines and to help promote the development of high-quality guidelines in oncology.

Methods

Instrument development

The Appraisal of Guidelines Research and Evaluation, (AGREE) Collaboration developed ‘the AGREE Instrument’ using a multi-step process that included item generation, item selection, scaling, field-testing, and refinement. The first draft of the instrument contained 24 items scored using a four-point Likert response scale ranging from ‘strongly agree’ to ‘strongly disagree’. The 24 items were grouped into six theoretical domains of quality: Scope and purpose, Stakeholder involvement, Rigour of development, Clarity and presentation, Applicability, and Editorial independence. As part of the validation of the instrument a study was conducted to assess the quality of a sample of 100 guidelines. Four independent appraisers assessed each guideline, and each appraiser assessed two guidelines where possible. There were 264 appraisers including medical practitioners, clinical experts, clinical researchers, and methodologists. Members of the guideline development group, members of the secretariat that produced the guidelines, and external referees were not eligible to be appraisers. The reliability of the final instrument was acceptable for most domains (Cronbach’s alpha ranged from 0.64 to 0.88). After validation, two of the original items were removed, and a new item was added, resulting in a final scale composed of 23 items. More details about the development and validation of the instrument are reported elsewhere [7].

Selection of guidelines

The definition that we used for a guideline was ‘a set of systematically developed statements to assist practitioner and patient decisions about appropriate health care for one specific clinical condition or disease area’ [8]. Documents that did not contain recommendations for clinical practice (e.g. systematic reviews, service documents) were excluded. The country coordinators were asked to select 7–10 guidelines, published between 1992 and 1999. They were instructed to provide both high- and low-quality guidelines, in order to test the discriminative value of the instrument. Amongst the 100 guidelines selected, 32 addressed cancer-related issues. The guidelines were produced by 25 organizations in 13 countries. The cancer-related guidelines (Table 1) were analysed further to identify predictors of a high quality score using the AGREE Instrument [9].

View this table:
Table 1

Cancer guidelines included in AGREE project

CountryOrganizationTitleYear of Publication
AustraliaNational Health and Medical Research Council (NHMRC)The Management of Early Breast Cancer: Systemic Adjuvant Therapy1995
CanadaCanadian Steering Committee on CPGs for the Care and Treatment of Breast Cancer (CSC)1The Palpable Breast Lump: Information and Recommendations to Assist Decision-making when a Breast Lump is Detected1998
The Management of Ductal Carcinoma in Situ (DCIS)1998
Adjuvant Systemic Therapy for Node Negative Breast Cancer1998
Cancer Care Ontario Practice Guidelines Initiative (CCOPGI)1Adjuvant Systemic Therapy for Node-negative Breast Cancer1999
The Management of Ductal Carcinoma in Situ (DCIS)1996
DenmarkNational Board of HealthLung Cancer: Examination and Treatment.1998
England and WalesRoyal College of Radiologists and Clinical Oncology Information NetworkGuidelines on the Non-surgical Management of Breast Cancer1999
Guidelines on the Management of Prostate Cancer1999
NHS Breast Screening Programme1Guidelines for Referral of Patients with Breast Problems1999
FinlandDuodecim—Current CareTreatment of Breast Cancer1998
Duodecim—EBM guidelinesTreatment of Breast Cancer1998
Turku University Hospital Region (TYKS)Treatment of Breast Cancer1994
FranceFrench Federation of Comprehensive Cancer Centres (FNCLCC)1Non-metastatic Breast Cancer1996
Hormone Therapy in Non-metastatic Breast Cancer1999
Management of Patients with Oropharynx Squamous Cell Carcinoma1999
National Agency for Accreditation and Evaluation in Healthcare (ANAES)Breast Cancer: Diagnostic Procedure in Women who have a Mammogram showing Subclinical Abnormalities1998
French Society of GastroenterologyPrevention, Screening, Diagnosis, and Treatment of Colon Cancer1998
GermanyGerman Society of Gynecologists and Obstetricians (DGGG) and German Society of OncologyDiagnostic and Therapeutic Standards for Cervixcarcinoma1998
DGGG, Association for Gynecological Endoscopy and Association for Gynecological OncologyGuideline for Laparoscopic Surgery of Ovarian Tumors1998
German Society of Oncology1Rehabilitation with Colorectal Carcinoma1997
ItalyItalian National Research CouncilBreast Cancer Follow-up1995
Hospital Trust PerugiaManagement of Breast Cancer1999
NetherlandsComprehensive Cancer Centre South1Management of Breast Carcinoma1997
Comprehensive Cancer Centre North1Management of Breast Carcinoma1999
ScotlandScottish Intercollegiate Guideline Network (SIGN)Breast Cancer in Women1998
SpainHTA Office Osteba and Basque Health DepartmentEarly Detection of Breast Cancer in the Basque Country1994
SwitzerlandSwiss Society of Internal Oncology1Guidelines for Adjuvant Treatment and Follow-up of Primary Breast Cancer1999
Swiss Society of Medical RadiologyRecommendations for MammographyUnknown
European Panel on Appropriateness of Gastrointestinal GastroscopyAppropriateness of Colonoscopy: Screening for Colorectal Cancer in Asymptomatic Individuals1999
USAAmerican Society of Clinical Oncology (ASCO)1Recommended Breast Cancer Surveillance Guideline1997
National Comprehensive Cancer Network (NCCN)1Treatment of Breast Cancer1997
  • 1 Organizations exclusively developing cancer guidelines.

Selection of variables

The guidelines were classified using the following six variables related to the guideline itself: publication date (1992–1994, 1995–1997, 1998–1999); type of guideline (initial update/revision); format of the clinical practice guideline (paper, paper and/or electronic); the level of care addressed (primary care, secondary care, both levels); scope (narrow, i.e. only screening, diagnosis or treatment; broad, i.e. combination of diagnosis and treatment); availability of background information (no information, some information or only references, detailed and structured documentation); and three variables related to the organization that produced the guideline: type of organization (professional/specialist societies, government-supported organizations, other), development in the setting of a guidelines programme or not (i.e. developed within a structured and coordinated programme): and exclusive focus on oncology or not (developing oncology guidelines exclusively, not developing oncology guidelines exclusively). The selection of these variables was based on a study reported in 2000 and published questionnaires in this area [10,11]. The information for the variables was obtained for each guideline using a standard form.

Analysis

In this paper we present the quality scores obtained with the first version of the AGREE instrument, using only data for the items that were retained for the final version. For each guideline, mean domain scores were derived by summing up the scores across the four appraisers and standardizing them as a percentage of the maximum possible score a guideline could achieve for that domain, i.e. Math Analysis of variance was used to compare mean domain scores as a function of each of the six guideline characteristics and each of the three organizational characteristics. Where significant differences were observed for a variable with more than two categories, post hoc tests were conducted using the Bonferroni approach for the t test to identify which categories were significantly different. To evaluate which variables had the most influence on the mean domain scores (i.e. which explained the variance) a linear stepwise regression analysis was performed. A variable was entered into the model if the significance level of its F value was <0.05, and was removed if the significance level was >0.10. Analyses were performed using SPSS (V.10.1.0) software (SPSS Inc, Chicago, IL, USA).

Results

The mean domain scores for the 32 oncology guidelines are shown in Table 2.

View this table:
Table 2

Descriptive statistics of domain scores for the 32 oncology guidelines

DomainMeanSDMinimumMaximum
Scope and purpose63.222.722.294.4
Stakeholder involvement34.317.56.379.2
Rigour of development42.226.04.878.6
Clarity and presentation57.219.413.987.5
Applicability25.818.00.075.0
Editorial independence47.333.20.0100.0

Analyses of guideline characteristics

The guideline characteristics found to be associated with significant differences in quality scores were ‘availability of background information’, ‘scope’, and ‘format’. ‘Availability of background information’ was the most influential characteristic (Table 3). For example, compared with guidelines with little or no background information, those with detailed background information had significantly higher mean domain scores (P < 0.05) for all domains except ‘Applicability’. These scores ranged from 43% ± 16% for ‘Stakeholder involvement’ to 74% ± 16% for ‘Scope and purpose’ for those with detailed background information, and from 4% ± 8% for ‘Editorial independence’ to 55% ± 23% for ‘Scope and purpose’ for those with little or no background information. The widest range of mean domain scores was observed for ‘availability of background information’ for the domain ‘Rigour of development’ (60% ± 17% for those with detailed background information versus 18% ± 9% and 12% ± 6% for guidelines with some or no background information, respectively, P < 0.01).

View this table:
Table 3

Domain scores for the oncology guidelines according to the characteristics of guidelines

Guideline characteristicDomains
Scope and purposeStakeholder involvementRigour of developmentClarity and presentationApplica-bilityEditorial independence
Publication date1
    1992–1994 (n = 2)62.529.223.864.640.337.5
    1995–1997 (n = 8)57.938.742.352.329.645.8
    1998–1999 (n = 21)67.233.345.059.523.950.3
P value20.600.710.560.600.400.86
Type of guideline1
    Initial (n = 23)62.937.045.158.226.148.2
    Updated/revised (n = 8)62.825.934.656.123.044.4
P value20.990.130.340.790.680.79
Format
    Paper (n = 15)60.432.131.953.329.148.9
    Paper and/or electronic (n = 17)65.636.351.360.523.045.9
P value20.520.510.0330.300.350.80
Level of care
    Primary care (n = 2)66.728.118.562.527.841.7
    Secondary care (n = 21)58.936.744.658.621.247.1
    Both levels (n = 9)72.430.042.052.636.349.1
P value20.330.570.410.700.100.96
Scope
    Narrow (n = 16)66.438.952.765.224.551.6
    Broad (n = 16)59.829.731.749.027.143.1
P value20.420.140.0190.0160.690.48
Availability of background information
    No information (n = 4)31.3 (0.047)321.9 (0.003)311.6 (0.000)347.4 (0.12)312.54.2 (0.002)3
    Some information/only references (n = 9)55.1 (0.001)321.8 (0.040)318.3 (0.000)341.9 (0.002)320.235.8 (0.001)3
    Detailed and structured documentation (n = 19)73.742.960.066.431.361.8
P value20.0000.0020.0000.0020.080.001
  • 1 Total number is 31 because the year of publication and type of one guideline was unknown for one guideline in each analysis.

  • 2 Result of the analysis of variance.

  • 3 Result of the Bonferroni t test (P value) with ‘detailed and structured documentation’ as reference category.

Guidelines with a narrow scope had significantly higher mean domain scores (P < 0.05) for the domains ‘Rigour of development’ (53% ± 23%) and ‘Clarity and presentation’ (65% ± 12%) compared with those with a broad scope (32% ± 25% and 49% ± 22%, respectively). Guidelines using multiple formats also had significantly higher mean domain scores (P < 0.05) for the domain ‘Rigour of development’ (52% ± 26%) compared with ‘paper-only’ guidelines (32% ± 22%). ‘Publication date’, ‘type of guideline’ (initial or updated) and ‘level of care’ (primary care, secondary care, both) did not explain the variance in mean scores for any of the domains.

Analyses of organizational characteristics

The most influential organizational characteristic was ‘type of organization’ (Table 4). Guidelines developed by government-supported organizations had significantly higher mean domain scores (P < 0.05) than guidelines developed by professional organizations or specialist societies on all domains, except ‘Applicability’. The mean domain scores ranged from 43% ± 17% for ‘Stakeholder involvement’ to 72% ± 21% for ‘Scope and purpose’ versus 21% ± 20% for ‘Rigour of development’ to 46% ± 16% for ‘Scope and purpose’. Guidelines developed by organizations with structured and coordinated programmes had significantly higher mean domain scores for the domains ‘Rigour of development’ and ‘Clarity and presentation’ than guidelines developed by organizations without such programmes (48% ± 27% versus 27% ± 15%, P <0.05 and 63% ± 18% versus 42% ± 16%, P < 0.01). The differences for the mean domain scores were not statistically significant between organizations developing oncology guidelines exclusively and those also developing guidelines in other clinical disciplines. In fact, for the domain ‘Applicability’ the mean domain scores were significantly lower for organizations developing oncology guidelines exclusively (19% ± 15% versus 32% ± 19%, P < 0.05).

View this table:
Table 4

Domain scores for the oncology guidelines according to characteristics of organizations responsible for guideline development

Organization characteristicDomains
Scope and purposeStakeholder involvementRigour of developmentClarity and presentationApplicabilityEditorial independence
Type of organization
Professional/specialist societies (n = 9)45.6 (0.012)121.8 (0.008)121.3 (0.011)140.7 (0.006)117.121.9 (0.025)1
Government supported organizations (n = 16)72.042.851.664.630.057.3
Other (n = 7)65.531.047.761.327.557.1
P value20.0140.0090.0110.0060.220.020
Guideline programme3
Part of guideline programme (n = 23)65.136.448.162.925.848.1
Not part of guideline programme (n = 9)58.328.927.242.526.045.4
P value20.460.280.0390.0050.970.84
Focus on oncology
Exclusively developing oncology guidelines (n = 15)62.537.248.860.019.152.8
Not exclusively developing oncology guidelines (n = 17)63.731.836.454.731.842.5
P value20.880.390.180.440.0450.39
  • 1 Result of the Bonferroni t test (P value) with ‘government supported organizations’ as reference category.

  • 2 Result of the analysis of variance.

  • 3 Total number is 31 because this variable was unknown for one guideline.

Characteristics of high-quality guidelines

Stepwise regression analysis showed that, together, the guideline and organizational characteristics explained between 31% (Applicability) and 67% (Rigour of development) of the variance in the mean domain scores (Table 5). The characteristic ‘availability of background information’ accounted for most of the variance in the mean domain scores (range from 17% for ‘Applicability’ to 67% for ‘Rigour of development’). The contribution of the other variables was relatively small or absent. The type of organization and guideline programme explained 11% and 12%, respectively, of the variance for the domain ‘Clarity and presentation’ and the absence of an exclusive focus on oncology explained 14% of the variance for the domain ‘Applicability’.

View this table:
Table 5

Explained variance (%) for predictors of domain scores1

Domains
Scope and purposeStakeholder involvementRigour of developmentClarity and presentationApplicabilityEditorial independence
Availability of background information35.632.067.124.216.636.0
Type of organizationNDNDND10.9NDND
Guideline programmeNDNDND11.7NDND
Focus on oncologyNDNDNDND14.5ND
All predictors35.632.067.146.831.136.0
  • 1 If the explained variance is large, the independent variable is a good predictor of the dependent variable (i.e. domain scores). The higher the percentage, the higher the contribution to the variance of the scores.

  • ND = not determined.

Guidelines with higher mean domain scores were generally found to provide more specific information on aspects of the guideline development process than the lower-scoring guidelines. Examples of good reporting from oncology guidelines and other high-quality aspects of guidelines for each item in the AGREE instrument are availabe on the Journal’s website (http://intqhc.oupjournals.org). Some organizations producing guidelines that obtained high mean domain scores in this analysis are presented in Appendix 1.

Discussion

In this study some predictors for high-quality oncology clinical practice guidelines were identified using the AGREE instrument. This instrument, which has been validated and endorsed by the World Health Organization, is considered by many guidelines organizations as the standard in guideline assessment [6,12,13]. We found that some guideline and organization characteristics explained the variation of the quality scores for the guidelines analysed here. The ‘availability of background information’ had the greatest impact on the guideline quality score, in particular for the domain ‘Rigour of development’. This finding is consistent with other studies [7,14]. Explicit and detailed information about the objectives and context of the guideline development, including the methods used, and the people and organizations involved in the development process are very important elements to include. Clinical practice guideline users will have more confidence in guidelines with these elements, since they facilitate acceptance and adherence to the guidelines [15,16]. However, although access to this background information is important, providing it can make the guideline document unwieldy and therefore more difficult to use. This could be detrimental for its conviviality and its uptake in clinical practice. Using an electronic format might overcome this difficulty since hyperlinks to technical documents and additional information sources can provide access to more detailed information on the various phases of the development process, without affecting the conviviality of the clinical practice guideline. In our analysis, we observed that clinical practice guidelines published in electronic or multiple formats (including electronic) had significantly higher scores for the domain ‘Rigour of development’.

The results of the stepwise regression analysis showed that the type of organization and the setting of a guideline programme had independent positive influences for the domain ‘Clarity of presentation’. Government-supported organizations probably have more resources and larger budgets, which will facilitate professionalism [11]. In the same way, structured guideline programmes provide an environment in which experienced support staff are available to help the guidelines panels during the development process, and for the formulation and drafting of guidelines [17,18]. In the cancer field there are several, non-government-funded organizations that produce high-quality guidelines, e.g. the American Society of Clinical Oncology (ASCO), the Cancer Care Ontario Practice Guideline Initiative (CCOPGI) in Canada [19], and the Standards Options Recommendations (SOR) programme of the FNCLCC in France [20]. Professional organizations may encourage buy-in from potential guideline users, and therefore result in better uptake of the recommendations in practice. It has been observed that these professional organizations can provide the leadership and organizational commitment that seem to be necessary ingredients for successful implementation [24].

Surprisingly, the characteristic ‘focus on oncology’ was associated with significantly lower quality scores in the domain of ‘Applicability’. The guidelines generally failed to address issues such as barriers to implementation and cost implications, and did not include criteria for monitoring. This may be explained in settings where links with regional cancer networks exist, since the networks adapt clinical practice guidelines into protocols for local implementation and use [4]. The low scores for the domain ‘Applicability’ emphasizes the need to take into consideration implementation during the development process to ensure that guidelines have an influence on clinical practice [21,22]. Thus, anticipating the potential organizational and attitudinal barriers and identifying the means of overcoming these will certainly contribute to improved acceptance and use of guidelines.

The methodology for the development of clinical practice guidelines is evolving with the aim of improving the quality of guidelines [21]. The number of publications reporting research into guideline development methodology has increased each year since the early 1990s. We would therefore expect that the quality of guidelines would increase over time, and that the updated guidelines would have higher quality scores. In this analysis, only a slight trend for improvement over time was observed; however, it should be noted that our sample was small, with only 32 guidelines, all developed between 1992 and 1999. A larger sample of oncology guidelines, with both initial and updated versions, would need to be assessed to determine whether there is improvement over time.

The guidelines analysed here were not a random sample and the number per organization was small, therefore they may not be representative of all existing guidelines. Some countries or organizations may have selected their best guidelines, while others may have included a range of guidelines with varying quality. In our analyses we used characteristics of guidelines as the units of analysis not the guidelines themselves, therefore the non-random selection probably does not have a major influence on the results presented here.

The examples presented on the Journal’s website (http://intqhc.oupjournals.org) were selected to help guideline users to understand what type of information they should look for when they are assessing clinical practice guidelines before deciding to use them. Also guideline developers can use these examples as an aid to good reporting to improve the quality of their guidelines. The main advantage of a well-reported guideline is that flaws in the methodology are more easily detected, so that inherent biases can be considered more explicitly and scrutinized by the potential users [23]. It is possible that a well-reported guideline contains flawed recommendations, and conversely, a guideline that is not systematically developed or well reported, may provide sound recommendations that are consistent with the evidence [24]. Recently, experts from the USA and Canada convened a conference on guideline standardization that resulted in a checklist for reporting clinical practice guidelines [5]. This is an important step towards international consensus on guideline reporting, in line with that developed for clinical trials in the CONSORT statement [25].

Organizations wishing to develop high-quality guidelines, including those with small budgets or ad hoc initiatives not able to set up a structured programme, should consider adapting existing high-quality guidelines for producing relevant guidelines for their own context [26,27]. An excellent source for identifying existing guidelines is the US National Guideline Clearinghouse, which includes more than 200 oncology guidelines (http://www.guideline.gov/). In addition, the recently established Guidelines International Network website is an increasingly important source for background documentation and access to existing guidelines (http://www.g-i-n.net). Organizations deciding to adapt existing guidelines could avoid duplication of effort and this approach will enable access to high-quality ingredients for producing their guidelines. An example of this approach in Europe is the European Society of Medical Oncology’s initiative ‘Minimum Clinical Recommendations’, which was set up in 1998 [28]. The guidelines are intended to provide the user with a set of statements for a basic standard of care felt to be necessary in all European countries, based on existing clinical practice guidelines. Thus this initiative provides the basis for the development of guidelines while avoiding duplication of effort.

Clinical practice guidelines are important tools for encouraging a comprehensive approach to cancer care, and contribute to bridging the gap between research results and clinical practice to improve the management and outcomes of patients with cancer. They have also contributed to the decrease in cancer mortality that has been observed over the past 15 years [29]. Nonetheless, there is considerable room for improvement. In particular, guideline developers should pay more attention to the applicability and implementation of guidelines to ensure that they will influence clinical practice effectively [17,20]. Including the patients’ perspectives and expectations in the clinical practice guideline development process would also improve the relevance of the guidelines to clinical decisions in oncology. Future international collaboration could be a means to improve a professional approach to guideline development and research. This would facilitate effective and efficient guideline production and enhance their appropriate use by maximizing the use of limited resources and avoiding unnecessary duplication of effort [12,30].

View this table:
Appendix 1

Organizations currently producing high-quality oncology guidelines

OrganizationYear of first guidelineWebsite
American Society of Clinical Oncology (ASCO)1994 www.asco.org
Cancer Care Ontario Practice Guidelines Initiative (CCOPGI, Canada)1994 www.cancercare.on.ca
French Federation of Comprehensive Cancer Centres (FNCLCC) Standards, Options and Recommendations project1993 www.fnclcc.fr
National Health and Medical Research Council (NHMRC, Australia)1995 www.health.gov.au/nhmrc
National Institute of Excellence (NICE, England and Wales)2000 www.nice.org.uk
Scottish Intercollegiate Guidelines Network (SIGN)1995 www.sign.ac.uk
  • More oncology guideline programmes can be found on www.g-i-n.net.

Acknowledgements

The authors would like to thank Serge Theobald for providing statistical advice. The AGREE Collaboration is an international group of researchers from 13 countries, who designed and field-tested the AGREE Instrument. Individuals participating in the AGREE Collaboration are listed on the AGREE website (www.agreecollaboration.org). The AGREE project was supported by a grant from the European Union BIOMED2 Programme (BMH4-98-3669). This study was financed by the French Rhône-Alpes Region and the French Cancer League.

References

View Abstract