International Journal for Quality in Health Care Advance Access originally published online on January 21, 2005
International Journal for Quality in Health Care 2005 17(2):123-132; doi:10.1093/intqhc/mzi011
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Predictors of high quality clinical practice guidelines: examples in oncology
1 Federation Nationale des Centres de Lutte Contre le Cancer, SOR, Paris, France, 2 Centre Léon Berard, GRESAC UMR-CNRS 5823, Lyon, France, 3 University Medical Centre Nijmegen, Centre for Quality of Care Research, Nijmegen, The Netherlands, 4 Program in Evidence-based Cancer Care Ontario, Hamilton, Ontario, Canada, 5 McMaster University, Department of Clinical Epidemiology and Biostatistics, Hamilton, Ontario, Canada, 6 Tom Baker Cancer Centre, Calgary, Alberta, Canada, 7 St George’s Hospital Medical School, Department of Community Health Sciences, London, UK
Objective. Clinical practice guidelines are widely used as effective tools for improving the management of patients with cancer. However, there is increasing concern about variation in guideline quality. In this study we identified predictors for high-quality guidelines in oncology.
Design. The quality scores for 32 oncology guidelines from 13 countries were determined by four independent appraisers using the Appraisal of Guidelines for Research and Evaluation (AGREE) instrument.
Main measures. The contribution to the quality score of six characteristics of guidelines and three of guideline developing organizations was then assessed using analysis of variance and stepwise linear regression analysis.
Results. Some guideline and organizational characteristics were shown to be responsible for a large part of the variations in quality scores. The availability of background information was the strongest predictor of quality with an explained variance ranging from 17% (‘Applicability’) to 67% (‘Rigour of development’). High-quality guidelines were more often produced by government-supported organizations and/or within a structured, coordinated programme. The other characteristics (publication year, type of guideline, format, level of care, and scope) were not independent predictors of quality.
Conclusions. Guidelines should provide more explicit information about the context of their development and methods used in order to improve their quality and thus encourage their use in clinical practice.
Keywords: clinical practice guidelines, international, network, oncology quality assessment
Address reprint requests to Margaret C. Haugh or Beatrice Fervers, E-mail: m.haugh{at}lyon.fnclcc.fr or fervers{at}lyon.fnclcc.fr
Accepted for publication October 18, 2004.
 |
Introduction |
|---|
 Top Introduction MethodsResultsDiscussionAcknowledgementsReferences |
|---|
Evidence-based clinical practice guidelines are widely used to promote effective and efficient health care [1]. Numerous guideline development programmes have been set up at local, regional, national and international levels. In the field of oncology the results from several studies show that guidelines can improve the care process, as well as improve patient outcomes [2–4]. Moreover, there is increasing concern about the variation of guideline quality and recommendations [5,6]. If guidelines are to improve the quality of health care they must be credible, in order to inspire confidence in prospective users; guideline quality is an important contribution to their credibility. The present study aims to identify predictors of high quality for clinical practice guidelines in oncology. These will be important to guideline users to help them identify credible guidelines and to help promote the development of high-quality guidelines in oncology.
|
Methods |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Instrument development
The Appraisal of Guidelines Research and Evaluation, (AGREE) Collaboration developed ‘the AGREE Instrument’ using a multi-step process that included item generation, item selection, scaling, field-testing, and refinement. The first draft of the instrument contained 24 items scored using a four-point Likert response scale ranging from ‘strongly agree’ to ‘strongly disagree’. The 24 items were grouped into six theoretical domains of quality: Scope and purpose, Stakeholder involvement, Rigour of development, Clarity and presentation, Applicability, and Editorial independence. As part of the validation of the instrument a study was conducted to assess the quality of a sample of 100 guidelines. Four independent appraisers assessed each guideline, and each appraiser assessed two guidelines where possible. There were 264 appraisers including medical practitioners, clinical experts, clinical researchers, and methodologists. Members of the guideline development group, members of the secretariat that produced the guidelines, and external referees were not eligible to be appraisers. The reliability of the final instrument was acceptable for most domains (Cronbach’s alpha ranged from 0.64 to 0.88). After validation, two of the original items were removed, and a new item was added, resulting in a final scale composed of 23 items. More details about the development and validation of the instrument are reported elsewhere [7].
Selection of guidelines
The definition that we used for a guideline was ‘a set of systematically developed statements to assist practitioner and patient decisions about appropriate health care for one specific clinical condition or disease area’ [8]. Documents that did not contain recommendations for clinical practice (e.g. systematic reviews, service documents) were excluded. The country coordinators were asked to select 7–10 guidelines, published between 1992 and 1999. They were instructed to provide both high- and low-quality guidelines, in order to test the discriminative value of the instrument. Amongst the 100 guidelines selected, 32 addressed cancer-related issues. The guidelines were produced by 25 organizations in 13 countries. The cancer-related guidelines (Table 1) were analysed further to identify predictors of a high quality score using the AGREE Instrument [9].
|
Selection of variables
The guidelines were classified using the following six variables related to the guideline itself: publication date (1992–1994, 1995–1997, 1998–1999); type of guideline (initial update/revision); format of the clinical practice guideline (paper, paper and/or electronic); the level of care addressed (primary care, secondary care, both levels); scope (narrow, i.e. only screening, diagnosis or treatment; broad, i.e. combination of diagnosis and treatment); availability of background information (no information, some information or only references, detailed and structured documentation); and three variables related to the organization that produced the guideline: type of organization (professional/specialist societies, government-supported organizations, other), development in the setting of a guidelines programme or not (i.e. developed within a structured and coordinated programme): and exclusive focus on oncology or not (developing oncology guidelines exclusively, not developing oncology guidelines exclusively). The selection of these variables was based on a study reported in 2000 and published questionnaires in this area [10,11]. The information for the variables was obtained for each guideline using a standard form.
Analysis
In this paper we present the quality scores obtained with the first version of the AGREE instrument, using only data for the items that were retained for the final version. For each guideline, mean domain scores were derived by summing up the scores across the four appraisers and standardizing them as a percentage of the maximum possible score a guideline could achieve for that domain, i.e.
 |
|
Results |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
The mean domain scores for the 32 oncology guidelines are shown in Table 2.
|
Analyses of guideline characteristics
The guideline characteristics found to be associated with significant differences in quality scores were ‘availability of background information’, ‘scope’, and ‘format’. ‘Availability of background information’ was the most influential characteristic (Table 3). For example, compared with guidelines with little or no background information, those with detailed background information had significantly higher mean domain scores (P < 0.05) for all domains except ‘Applicability’. These scores ranged from 43% ± 16% for ‘Stakeholder involvement’ to 74% ± 16% for ‘Scope and purpose’ for those with detailed background information, and from 4% ± 8% for ‘Editorial independence’ to 55% ± 23% for ‘Scope and purpose’ for those with little or no background information. The widest range of mean domain scores was observed for ‘availability of background information’ for the domain ‘Rigour of development’ (60% ± 17% for those with detailed background information versus 18% ± 9% and 12% ± 6% for guidelines with some or no background information, respectively, P < 0.01).
|
Guidelines with a narrow scope had significantly higher mean domain scores (P < 0.05) for the domains ‘Rigour of development’ (53% ± 23%) and ‘Clarity and presentation’ (65% ± 12%) compared with those with a broad scope (32% ± 25% and 49% ± 22%, respectively). Guidelines using multiple formats also had significantly higher mean domain scores (P < 0.05) for the domain ‘Rigour of development’ (52% ± 26%) compared with ‘paper-only’ guidelines (32% ± 22%). ‘Publication date’, ‘type of guideline’ (initial or updated) and ‘level of care’ (primary care, secondary care, both) did not explain the variance in mean scores for any of the domains.
Analyses of organizational characteristics
The most influential organizational characteristic was ‘type of organization’ (Table 4). Guidelines developed by government-supported organizations had significantly higher mean domain scores (P < 0.05) than guidelines developed by professional organizations or specialist societies on all domains, except ‘Applicability’. The mean domain scores ranged from 43% ± 17% for ‘Stakeholder involvement’ to 72% ± 21% for ‘Scope and purpose’ versus 21% ± 20% for ‘Rigour of development’ to 46% ± 16% for ‘Scope and purpose’. Guidelines developed by organizations with structured and coordinated programmes had significantly higher mean domain scores for the domains ‘Rigour of development’ and ‘Clarity and presentation’ than guidelines developed by organizations without such programmes (48% ± 27% versus 27% ± 15%, P <0.05 and 63% ± 18% versus 42% ± 16%, P < 0.01). The differences for the mean domain scores were not statistically significant between organizations developing oncology guidelines exclusively and those also developing guidelines in other clinical disciplines. In fact, for the domain ‘Applicability’ the mean domain scores were significantly lower for organizations developing oncology guidelines exclusively (19% ± 15% versus 32% ± 19%, P < 0.05).
|
Characteristics of high-quality guidelines
Stepwise regression analysis showed that, together, the guideline and organizational characteristics explained between 31% (Applicability) and 67% (Rigour of development) of the variance in the mean domain scores (Table 5). The characteristic ‘availability of background information’ accounted for most of the variance in the mean domain scores (range from 17% for ‘Applicability’ to 67% for ‘Rigour of development’). The contribution of the other variables was relatively small or absent. The type of organization and guideline programme explained 11% and 12%, respectively, of the variance for the domain ‘Clarity and presentation’ and the absence of an exclusive focus on oncology explained 14% of the variance for the domain ‘Applicability’.
|
Guidelines with higher mean domain scores were generally found to provide more specific information on aspects of the guideline development process than the lower-scoring guidelines. Examples of good reporting from oncology guidelines and other high-quality aspects of guidelines for each item in the AGREE instrument are availabe on the Journal’s website (http://intqhc.oupjournals.org). Some organizations producing guidelines that obtained high mean domain scores in this analysis are presented in Appendix 1.
|
Discussion |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
In this study some predictors for high-quality oncology clinical practice guidelines were identified using the AGREE instrument. This instrument, which has been validated and endorsed by the World Health Organization, is considered by many guidelines organizations as the standard in guideline assessment [6,12,13]. We found that some guideline and organization characteristics explained the variation of the quality scores for the guidelines analysed here. The ‘availability of background information’ had the greatest impact on the guideline quality score, in particular for the domain ‘Rigour of development’. This finding is consistent with other studies [7,14]. Explicit and detailed information about the objectives and context of the guideline development, including the methods used, and the people and organizations involved in the development process are very important elements to include. Clinical practice guideline users will have more confidence in guidelines with these elements, since they facilitate acceptance and adherence to the guidelines [15,16]. However, although access to this background information is important, providing it can make the guideline document unwieldy and therefore more difficult to use. This could be detrimental for its conviviality and its uptake in clinical practice. Using an electronic format might overcome this difficulty since hyperlinks to technical documents and additional information sources can provide access to more detailed information on the various phases of the development process, without affecting the conviviality of the clinical practice guideline. In our analysis, we observed that clinical practice guidelines published in electronic or multiple formats (including electronic) had significantly higher scores for the domain ‘Rigour of development’.
The results of the stepwise regression analysis showed that the type of organization and the setting of a guideline programme had independent positive influences for the domain ‘Clarity of presentation’. Government-supported organizations probably have more resources and larger budgets, which will facilitate professionalism [11]. In the same way, structured guideline programmes provide an environment in which experienced support staff are available to help the guidelines panels during the development process, and for the formulation and drafting of guidelines [17,18]. In the cancer field there are several, non-government-funded organizations that produce high-quality guidelines, e.g. the American Society of Clinical Oncology (ASCO), the Cancer Care Ontario Practice Guideline Initiative (CCOPGI) in Canada [19], and the Standards Options Recommendations (SOR) programme of the FNCLCC in France [20]. Professional organizations may encourage buy-in from potential guideline users, and therefore result in better uptake of the recommendations in practice. It has been observed that these professional organizations can provide the leadership and organizational commitment that seem to be necessary ingredients for successful implementation [2–4].
Surprisingly, the characteristic ‘focus on oncology’ was associated with significantly lower quality scores in the domain of ‘Applicability’. The guidelines generally failed to address issues such as barriers to implementation and cost implications, and did not include criteria for monitoring. This may be explained in settings where links with regional cancer networks exist, since the networks adapt clinical practice guidelines into protocols for local implementation and use [4]. The low scores for the domain ‘Applicability’ emphasizes the need to take into consideration implementation during the development process to ensure that guidelines have an influence on clinical practice [21,22]. Thus, anticipating the potential organizational and attitudinal barriers and identifying the means of overcoming these will certainly contribute to improved acceptance and use of guidelines.
The methodology for the development of clinical practice guidelines is evolving with the aim of improving the quality of guidelines [21]. The number of publications reporting research into guideline development methodology has increased each year since the early 1990s. We would therefore expect that the quality of guidelines would increase over time, and that the updated guidelines would have higher quality scores. In this analysis, only a slight trend for improvement over time was observed; however, it should be noted that our sample was small, with only 32 guidelines, all developed between 1992 and 1999. A larger sample of oncology guidelines, with both initial and updated versions, would need to be assessed to determine whether there is improvement over time.
The guidelines analysed here were not a random sample and the number per organization was small, therefore they may not be representative of all existing guidelines. Some countries or organizations may have selected their best guidelines, while others may have included a range of guidelines with varying quality. In our analyses we used characteristics of guidelines as the units of analysis not the guidelines themselves, therefore the non-random selection probably does not have a major influence on the results presented here.
The examples presented on the Journal’s website (http://intqhc.oupjournals.org) were selected to help guideline users to understand what type of information they should look for when they are assessing clinical practice guidelines before deciding to use them. Also guideline developers can use these examples as an aid to good reporting to improve the quality of their guidelines. The main advantage of a well-reported guideline is that flaws in the methodology are more easily detected, so that inherent biases can be considered more explicitly and scrutinized by the potential users [23]. It is possible that a well-reported guideline contains flawed recommendations, and conversely, a guideline that is not systematically developed or well reported, may provide sound recommendations that are consistent with the evidence [24]. Recently, experts from the USA and Canada convened a conference on guideline standardization that resulted in a checklist for reporting clinical practice guidelines [5]. This is an important step towards international consensus on guideline reporting, in line with that developed for clinical trials in the CONSORT statement [25].
Organizations wishing to develop high-quality guidelines, including those with small budgets or ad hoc initiatives not able to set up a structured programme, should consider adapting existing high-quality guidelines for producing relevant guidelines for their own context [26,27]. An excellent source for identifying existing guidelines is the US National Guideline Clearinghouse, which includes more than 200 oncology guidelines (http://www.guideline.gov/). In addition, the recently established Guidelines International Network website is an increasingly important source for background documentation and access to existing guidelines (http://www.g-i-n.net). Organizations deciding to adapt existing guidelines could avoid duplication of effort and this approach will enable access to high-quality ingredients for producing their guidelines. An example of this approach in Europe is the European Society of Medical Oncology’s initiative ‘Minimum Clinical Recommendations’, which was set up in 1998 [28]. The guidelines are intended to provide the user with a set of statements for a basic standard of care felt to be necessary in all European countries, based on existing clinical practice guidelines. Thus this initiative provides the basis for the development of guidelines while avoiding duplication of effort.
Clinical practice guidelines are important tools for encouraging a comprehensive approach to cancer care, and contribute to bridging the gap between research results and clinical practice to improve the management and outcomes of patients with cancer. They have also contributed to the decrease in cancer mortality that has been observed over the past 15 years [29]. Nonetheless, there is considerable room for improvement. In particular, guideline developers should pay more attention to the applicability and implementation of guidelines to ensure that they will influence clinical practice effectively [17,20]. Including the patients’ perspectives and expectations in the clinical practice guideline development process would also improve the relevance of the guidelines to clinical decisions in oncology. Future international collaboration could be a means to improve a professional approach to guideline development and research. This would facilitate effective and efficient guideline production and enhance their appropriate use by maximizing the use of limited resources and avoiding unnecessary duplication of effort [12,30].
|
|
Acknowledgements |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
The authors would like to thank Serge Theobald for providing statistical advice. The AGREE Collaboration is an international group of researchers from 13 countries, who designed and field-tested the AGREE Instrument. Individuals participating in the AGREE Collaboration are listed on the AGREE website (www.agreecollaboration.org). The AGREE project was supported by a grant from the European Union BIOMED2 Programme (BMH4-98-3669). This study was financed by the French Rhône-Alpes Region and the French Cancer League.
|
References |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
- Woolf SH, Grol R, Hutchinson A et al. Potential benefits, limitations, and harms of clinical guidelines. Br Med J 1999; 318: 527–530.
[Free Full Text] - Smith TJ, Hilnner BE. Ensuring quality cancer care by the use of clinical practice guidelines and critical pathways. J Clin Oncol 2001; 19: 2886–2897.
[Abstract/Free Full Text] - Ray-Coquard I, Philip T, Lehmann M et al. Impact of a clinical guidelines program for breast and colon cancer in a French cancer center. J Am Med Assoc 1997; 278: 1591–1595.
[Abstract/Free Full Text] - Ray-Coquard I, Philip T, de Laroche G et al. A controlled ‘before–after’ study: impact of a clinical guidelines programme and regional cancer network organization on medical practice. Br J Cancer 2002; 86: 313–321.[CrossRef][Web of Science][Medline]
- Shiffman RN, Shekelle P, Overhage M et al. Standardizing reporting of clinical practice guidelines: a proposal from the conference on guideline standardization. Ann Intern Med 2003; 139: 493–498.
[Abstract/Free Full Text] - Harpole LH, Kelley MJ, Schreiber G et al. Assessment of the scope and quality of clinical practice guidelines in lung cancer. Chest 2003; 123 (Suppl. 1): 7S–20S.
- AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003; 12: 18–23.
[Abstract/Free Full Text] - Field MJ, Lohr KN. Clinical Practice Guidelines: Directions for a New Program. Institute of Medicine, Washington DC: National Academy Press, 1990.
- Burgers JS, Fervers B, Haugh M et al. International assessment of the quality of clinical practice guidelines in oncology using the AGREE Instrument. J Clin Oncol 2004; 22: 2000–2007.
[Abstract/Free Full Text] - AGREE (Appraisal of Guidelines, Research, and Evaluation in Europe) Collaborative Group. Guideline development in Europe: an international comparison. Int J Technol Assess Health Care 2000; 16: 1029–1049.
- Burgers JS, Grol R, Klazinga NS et al. Towards evidence-based clinical practice: an international survey of 18 clinical guideline programs. Int J Qual Health Care 2003; 15: 31–45.
[Abstract/Free Full Text] - Grol R, Cluzeau FA, Burgers JS. Clinical practice guidelines: towards better quality guidelines and increased international collaboration. Br J Cancer 2003; 89 (Suppl. 1): S4–S8.[Medline]
- Graham ID, Calder LA, Hebert PC et al. A comparison of clinical practice guideline appraisal instruments. Int J Technol Assess Health Care 2000; 16: 1024–1038.[CrossRef][Web of Science][Medline]
- Cluzeau F, Littlejohns P, Grimshaw J et al. Development and application of a generic methodology to assess the quality of clinical guidelines. Int J Qual Health Care 1999; 11: 21–28.
[Abstract/Free Full Text] - Grol R, Dalhuijsen J, Thomas S, in ‘tVeld C, Rutten G, Mokkink H. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. Br Med J 1998; 317: 858–861.
[Abstract/Free Full Text] - Burgers JS, Grol RPTM, Zaat JOM, Spies TH, Van der Bij AK, Mokkink HGA. Characteristics of effective clinical guidelines for general practice. Br J Gen Pract 2003; 53: 15–19.[Web of Science][Medline]
- Browman GP. Background to clinical guidelines in cancer: SOR, a programmatic approach to guideline development and aftercare. Br J Cancer 2001; 84 (Suppl. 2): 1–3.[Medline]
- Woolf SH, DiGuiseppi CG, Atkins D, Kamerow DB. Developing evidence-based clinical practice guidelines: lessons learned by the US Preventive Services Task Force. Annu Rev Public Health 1996; 17: 511–538.[Web of Science][Medline]
- Browman GP, Levine MN, Mohide EA et al. The practice guidelines development cycle: a conceptual tool for practice guidelines development and implementation. J Clin Oncol 1995; 13: 502–512
[Abstract/Free Full Text] - Fervers B, Hardy J, Blanc-Vincent MP et al. SOR: project methodology. Br J Cancer 2001; 84 (Suppl. 2): 8–16.[Medline]
- Browman GP. Improving clinical practice guidelines for the 21st century. Attitudinal barriers and not technology are the main challenges. Int J Technol Assess Health Care 2000; 16: 959–968.[CrossRef][Web of Science][Medline]
- Bennett CL, Somerfield MR, Pfister DG et al. Perspectives on the value of American Society of Clinical Oncology clinical guidelines as reported by oncologists and Health Maintenance Organizations. J Clin Oncol 2003; 21: 937–941.
[Abstract/Free Full Text] - Austoker J. The potential and pitfalls for developing guidelines in oncology. Ann Oncol 2001; 12: 1189–1190.
[Free Full Text] - Soares HP, Daniels S, Kumar A et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. Br Med J 2004; 328: 22–25.
[Abstract/Free Full Text] - Begg C, Cho M, Eastwood S et al. Improving the quality of reporting of randomized controlled trials: the CONSORT statement. J Am Med Assoc 1996; 276: 637–639.
[Abstract/Free Full Text] - Graham ID, Harrison MB, Brouwers M et al. Facilitating the use of evidence in practice: evaluating and adapting clinical practice guidelines for local use by health care organizations. J Obstet Gynecol Neonatal Nurs 2002; 31: 599–611.[CrossRef][Web of Science][Medline]
- Voellinger R, Berney A, Baumann P et al. Major depressive disorder in the general hospital: adaptation of clinical practice guidelines. Gen Hosp Psychiatry 2003; 25: 185–193.[CrossRef][Web of Science][Medline]
- ESMO Guidelines Task Force. ESMO minimum clinical recommendations—the beginning of a process. Ann Oncol 2001; 12: 1037–1038.
[Free Full Text] - Boyle P, d’Onofrio A, Maisonneuve P et al. Measuring progress against cancer in Europe: has the 15% decline targeted for 2000 come about? Ann Oncol 2003; 14: 1312–1325.
[Abstract/Free Full Text] - Fervers B, Philip T, Haugh M et al. Clinical practice guidelines in Europe: time for European co-operation for cancer guidelines. Lancet Oncol 2003; 4: 139–140.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  P. Debourdeau, D. Kassab Chahmi, G. Le Gal, I. Kriegel, E. Desruennes, M.-C. Douard, I. Elalamy, G. Meyer, P. Mismetti, M. Pavic, et al. 2008 SOR guidelines for the prevention and treatment of thrombosis associated with central venous catheters in patients with cancer: report from the working group Ann. Onc., June 12, 2009; (2009) mdp052v1. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
G. Pentheroudakis, R. Stahel, H. Hansen, and N. Pavlidis Heterogeneity in cancer guidelines: should we eradicate or tolerate? Ann. Onc., December 1, 2008; 19(12): 2067 - 2078. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 E. Nagy, J. Watine, P. S. Bunting, R. Onody, W. P. Oosterhuis, D. Rogic, S. Sandberg, K. Boda, and A. R. Horvath Do Guidelines for the Diagnosis and Monitoring of Diabetes Mellitus Fulfill the Criteria of Evidence-Based Guideline Development? Clin. Chem., November 1, 2008; 54(11): 1872 - 1882. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
N. Pavlidis, H. Hansen, and R. Stahel ESMO clinical recommendations: a practical guide for medical oncologists Ann. Onc., November 1, 2007; 18(11): 1759 - 1763. [Full Text] [PDF]
|
|
|
|
|
|
F. Vigna-Taglianti, P. Vineis, A. Liberati, and F. Faggiano Quality of systematic reviews used in guidelines for oncology practice Ann. Onc., April 1, 2006; 17(4): 691 - 701. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. S. Burgers Guideline Quality and Guideline Content: Are They Related? Clin. Chem., January 1, 2006; 52(1): 3 - 4. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||