International Journal for Quality in Health Care 15:487-493 (2003)
© 2003 International Society for Quality in Health Care
Validity and reliability of guidelines for neck pain treatment in primary health care. A nationwide empirical analysis in Spain
1 Department of Public Health and Preventive Medicine and
2 Department of Physiotherapy, University of Murcia, Murcia, Spain
Objectives. To assess the reliability and validity of existing clinical guidelines on neck-pain physiotherapy treatment and follow-up in Spain.
Design. We identified existing guidelines through a nationwide census and listed their recommendations, grouped according to the main steps of the process flow-chart. To assess reliability we analysed the variability of statements. To analyse validity we assessed the type of scientific evidence supporting the recommendations, and we compared them with a list of evidence-based recommendations that was elaborated for this study.
Setting and participants. Primary health care centres (n = 24) with guidelines for neck-pain treatment and follow-up.
Main outcome measures. We quantified the number of recommendations, the proportion of valid statements, the frequencies of non-evidence-based recommendations, and the absence of the evidence-based recommendations we had identified.
Results. The 34 identified guidelines contained 325 recommendations, with great variation between guidelines with respect to the number, type (for up to 26 different clinical decisions), and content of the recommendations they provided. Direct assessment of the scientific evidence was not possible because no specific reference was given to support any recommendation. When compared with our list, only 20.9% of the recommendations could be considered evidence-based. No guideline contained all the eight evidence-based recommendations we identified.
Conclusions. The results question the guidelines’ reliability and validity, and their usefulness in ensuring quality. We conclude that guidelines should be reviewed and re-designed with greater scientific rigour.
Keywords: evidence-based medicine, neck pain, practice guidelines
Accepted for publication August 5, 2003.
Neck pain is a high-prevalence health problem [1]. Approximately half of the active population has suffered from neck pain on some occasion. In 14% of individuals, it lasts for 6 months or longer [1,2], causing not only difficulties at work, but also in their everyday lives at home and in their leisure activities [2]. Neck pain generates considerable health care costs, the most significant being those caused by compensation for sick leave [3]. It has even been demonstrated that neck pain causes as much sick leave from work as back pain [4].
In the Spanish health care system, physical therapy has progressively been used as a simultaneous or alternative therapy to the traditional pharmacotherapy, and neck pain has become one of the health care problems most governed by protocols [5].
Protocols or guidelines for clinical practice are useful tools for decision making and the achievement of quality [6–8], as they should reduce unnecessary variability. According to the Institute of Medicine (IOM) of the United States [7,8], to help ensure quality and to avoid unnecessary variability, guidelines must be valid and reliable. Guidelines are reliable if different groups of experts make the same recommendations, and those recommendations are interpreted in a uniform way by those who intend to follow them. Validity implies the achievement of the expected results by following the recommendations. Validity can be indirectly assessed according to the type of scientific evidence that justifies those recommendations [7,8]. Thus, it is advisable to consider systematically the available evidence when elaborating guidelines.
The purpose of this study was to assess the variability and scientific evidence behind the recommendations for neck-pain physiotherapy treatment, as stated in the clinical guidelines compiled in Spain. At the same time, we also wished to list evidence-based recommendations in order to determine how many, and which of them, are not included in the documents analysed. The final objective was to ascertain whether it is necessary to critically re-develop the existing documents and progress towards the designing of better quality clinical practice guidelines.
 |
Methods |
|---|
 Top Methods ResultsDiscussionReferences |
|---|
The existing guidelines were identified through a national census of locally adopted guidelines, which was developed previously for a project focusing on assessing and improving quality in all types of physiotherapy clinical guidelines in primary health care [9]. Guidelines were defined as ‘written documents, including text and algorithm formats, which advise or recommend specific actions for the health care of specific health problems’. We have also included, substituting the initial documents, those guidelines that were re-elaborated as a consequence of that project, so that all the documents analysed could be considered current.
For the present study, we selected all documents focusing on health care for patients with non-traumatic neck pain, with or without irradiation, and of a subacute or chronic nature. The target population of some documents included different subgroups that fulfilled the inclusion requirements (e.g. patients with neck pain associated with cervicoarthrosis, cervical radiculopathy, or chronic neck pain), and as a result a group of specific recommendations was established. Such documents were considered to represent several guidelines in the analysis (the number of guidelines corresponding to the number of target groups addressed). Guidelines for traumatic neck pain and protocols limited to the description of procedures for applying specific techniques or devices were excluded.
To guide data gathering and to group the clinical recommendations for the analysis, we flow-charted the basic steps of the clinical process and we identified the specific clinical decisions for which explicit recommendations could be made. The process was divided into five components: two related to treatment and three related to follow-up. Treatment components included: (i) patient education; and (ii) first-choice therapies. Therapy recommendations were classified into four groups: exercises, manual therapies, physical agents, and electrotherapy methods. Follow-up components included: (i) number and periodicity of the initial treatment visits; (ii) periodicity of the clinical evolution assessment sessions; and (iii) number, contents, and periodicity of the follow-up visits after initial treatment.
We harmonized the terminology using the categories established by the Physical Therapy Interventions Classification, which was developed by the American Physical Therapy Association (APTA) [10].
We quantified the recommendations’ variability by counting the number of documents in which each of them appeared.
The scientific evidence of the recommendations was assessed as follows. First, we searched the guidelines for statements related to type of evidence and references that specifically justified the recommendations, so that we could access the referenced articles and assess the type of study. For this assessment we used the classification provided by the US Preventive Services Task Force [11]. Secondly, we made a list of evidence-based recommendations, to compare them with the recommendations made in the documents. To elaborate the list we identified and assessed the evidence available through a systematic review of the Medline (1966–1999), Rehabdata, IME (1971–1999), Cochrane (1998 version) and Pedros databases (1999 update), as well as the lists of references quoted in the articles reviewed. The key words for the search were ‘mechanical neck pain’, ‘chronic neck pain’, ‘cervical radiculopathy’, and ‘neck disorder’, all of them related to the target health problem, and ‘physical therapy’, ‘physiotherapy’, ‘therapy’, ‘conservative treatment’, ‘intervention’, ‘management’, and ‘health education’, to identify studies related to physical therapy interventions. We considered as acceptable evidence the results from randomized controlled trials, well designed non-randomized controlled trials, and cohort and case-control studies (levels I, II-1 and II-2 of the reference classification [11]). We then assessed the presence of these evidence-based recommendations in the evaluated guidelines.
The validity assessment was summarized, calculating: (i) the proportion of evidence-based recommendations (levels I and II in the reference classification) over the total number of stated recommendations within all guidelines; and (ii) the absolute frequencies of absence in the guidelines for each evidence-based recommendation from our list. Additionally, even though all guidelines were considered current by the health care centres, we grouped them into categories according to the date of completion (1990–1992, 1993–1995, and post-1996) and the number of recommendations they contained (1–7, 8–14, and 15–22). We then analysed, using the Mantel-Haenszel’s 
2 for trends, if there was any trend over time with respect to the proportion of evidence-based recommendations. Calculations were performed using SPSS 9.0.
|
Results |
|---|
|
TopMethodsResultsDiscussionReferences |
|---|
Guidelines identified
We identified 24 documents. Seven of them included guidelines for more than one target group: two types in four of the guidelines, and three in the remaining three. In total, 34 guidelines were considered for the analysis. Ten of them were elaborated between 1990 and 1992, seven between 1993 and 1995, and 19 (56%) after 1996.
In nine guidelines, the main criterion to define the target population was the time during which neck-pain was present, and they included chronic neck pain exclusively. In six guidelines, the target population was identified according to underlying conditions (cervicoarthrosis). Twelve (35.2%) included cervicobrachial neck pain (neck pain associated with cervical radiculopathy) as the target population, and 15 (44.1%) were developed for any type of mechanical neck pain.
Twenty-three guidelines (68%) had been developed exclusively by physical therapists, and the rest by physical therapists and other professionals, mainly physicians.
Variability in relation to patients’ education and choice of therapy
The guidelines provided 31 different treatment recommendations; five related to patient education and 26 related to therapies.
Except for one document that provided just one recommendation, all guidelines gave several, averaging nine recommendations per document (range 3–22). The average number of therapy recommendations per document (7.5) was higher than that related to education (2.1).
We found 26 therapeutic recommendations (Table 1), none of which appeared in all documents. Posture re-education was the most frequent recommendation (present in 82.4% of documents), followed by superficial irradiation thermotherapy (64.7%) and joint mobility exercises (58.8%). In general, the use of physical agents presented the greatest variability in terms of number of recommendations per document, and manual therapies were the least frequently recommended, as nearly 50% of guidelines did not include any recommendation of this type.
|
Guidelines provided five recommendations on patient education (Table 2). The most common one, in 31 guidelines, was advising patients on their everyday life and work activities. The two recommendations related to functional training were the least common ones. Only two guidelines contained the five recommendations.
|
Variability in the recommendations for follow-up
The recommended number of visits for initial treatment ranged between three and 20 (most frequent recommendation between 15 and 20; Table 3). Two guidelines recommended less than eight visits, and they were also the only ones recommending a combination of patient education and exercises (without any manual, electro-, or physical-agent therapies). In eight of the 34 guidelines, the number of visits was not stated, granting the professional the opportunity to choose the appropriate number.
|
Twelve guidelines (35.3%) recommended a certain, but not uniform, periodicity for treatment visits. Seven guidelines recommended three visits per week, one guideline recommended one visit per week, three guidelines offered a choice between daily or alternative-days treatment, and one suggested choosing between two or three visits per week.
Details on the periodicity for clinical evolution assessments were included in 20 of the 34 guidelines (60.8%). Thirteen guidelines recommended a single assessment at the end of the treatment and prior to discharge, one suggested assessments after every visit, three recommended assessment every 15 days, and three recommended assessment every 30 days. In the other guidelines there were no recommendations for clinical evolution assessment. Twenty-four (70.6%) guidelines did not include any statement relating to monitoring after initial treatment, and there was important variability, including contradictory recommendations, in the other 10: eight guidelines recommended monitoring visits, whereas two did not. When recommended, there was no agreement on the number and periodicity of visits.
Scientific evidence for statements
No recommendation was qualified with its type of scientific evidence. Additionally, although the guideline documents usually include a bibliography section, none of the 325 recommendations identified within the 34 guidelines was specifically linked to any reference. Consequently, it was not possible to assess, with the information provided in the documents, the scientific evidence supporting the statements. In our literature review, however, we found a total of eight recommendations with sufficient evidence. Except for one, all recommendations in our list were of evidence type I (Table 4), and related to what we have called treatment. No sufficient evidence was found to support follow-up statements.
|
All the evidence-based recommendations identified in our review (Table 4) should be included in all guidelines, except the one on applying traction, which is only appropriate for cervicobrachial pain. Nevertheless, none of the guidelines included all of them. The recommendations more frequently stated were the ‘advice on repetitive movements or maintained postures’ and ‘at-home exercises’, in 76.5% and 64.7% of the guidelines, respectively. The other evidence-based recommendations appeared in < 36% of the guidelines. Three of them—those related to oculocervickinetic re-education, transcutaneous electro-stimulation, and mechanical traction—were not given in any document.
Table 5 shows the frequency with which evidence-based recommendations were not stated in the guidelines. A recommendation was defined as absent when: (i) it was not being included; (ii) the recommendation was incomplete and could not be considered specific enough for a particular dysfunction (in 14 documents); or (iii) not enough parameters were indicated or were different to those that justified the recommendation (in 18 documents). For example, recommendations linked to the identification of a particular disorder (radiculopathy, trigger points, intervertebral disorder) were frequently incomplete because the authors tended to suggest generic recommendations for neck pain, which were not specific for the associated disorders. In other cases the stated specifications differed from those expected according to the scientific evidence.
|
The total number of evidence-based recommendations in the 34 documents was 68, 27.2% of the total possible evidence-based recommendations we could find [(34 guideline documents x seven evidence-based recommendations) + (12 guidelines documents including cervicobrachialgia x 1 recommendation) = 250] and 20.9% of the 325 recommendations identified. On the other hand, the proportion of evidence-based recommendations was not significantly associated with the number of stated recommendations (2MH = 1.16; P = 0.28), nor with the year in which the guidelines were designed (2MH = 0.27; P = 0.6). However, when we analysed both variables simultaneously, the proportion of evidence-based recommendations showed a significant trend (2MH = 18.6; P < 0.005) as it decreased over time, changing from 0.23 in guidelines designed within the 1990–1992 period to 0.20 in guidelines designed after 1996.
|
Discussion |
|---|
|
TopMethodsResultsDiscussionReferences |
|---|
We have found a considerable variability in the recommendations provided by the analysed guidelines, and we have even identified clear contradictions among them. This variability makes evident the guidelines’ lack of reliability, and questions the scientific rigour applied when designing or adapting them. If the guidelines were to be followed in the health centres they come from, the same person would be treated in a very different way; consequently the outcomes and the resources needed are likely to differ also. Both under- and over-use of resources could be expected when following the guidelines, granting equally inappropriate treatments. This consequence has also been suggested in previous studies of the content variation of guidelines for very diverse conditions such as high blood pressure [18], anticoagulant treatment in atrial fibrillation [19], and screening for gestational diabetes [20].
We found no studies assessing reliability and validity in practice guidelines for neck pain, so we could not compare our results. Different studies have assessed validity and reliability in a group of guidelines using generic tools [21–23], but they do not give comparative results for guidelines on the same health condition. However, variability seems to be the rule also when comparing the contents of guidelines for the same health condition, both in comparisons within [18,19,24,25] and between countries [26–28], particularly when the assessed guidelines are locally adapted or developed [18,19], and regardless of the type of guideline developers (e.g. local groups [18,19] or national scientific societies [18,24]).
Variability in the recommendations for the same clinical decision questions reliability, but it also casts doubt on whether design and/or local adaptation of the guidelines have considered the degree of scientific evidence [29]. There was actually a low proportion of valid recommendations in the assessed guidelines (ranging between 0 and 0.5), as expected from their variability. The low proportion of evidence-based recommendations, along with the absence of recommendations linked to specific bibliography references, makes evident that when designing them, the scientific literature was not systematically reviewed. This situation has not improved with time: the proportion of valid recommendations tended even to decrease over time, suggesting that this problem will not improve unless it is explicitly addressed.
Moreover, apart from the fact that not all the recommendations contained in the analysed guidelines were valid, we could not find in them those evidence-based recommendations that resulted from our review. The first four recommendations shown in Table 4 can be applied to all patients with neck pain, regardless of how long patients have been suffering from such problems (both subacute or chronic neck pain) or of the different underlying diagnoses (e.g. cervical radiculopathy or cervicoarthrosis). Thus, even though it can be argued that recommendations 6 and 7 in Table 4 apply only in the event of detecting trigger points or intervertebral disorders through physical examination, both disorders can appear in all types of neck pain [30]. Along the same lines, recommendation 5 (Table 4), related to oculocervical re-education, is specific to patients with chronic neck pain, but we believe it should only be absent in guidelines that are exclusively designed for subacute neck pain.
Two of the evidence-based recommendations (oculocervical re-education and myointensive manipulations) were not included in any document. The fact that, among all evidence-based recommendations, these two are the ones that could demand more time in their application suggests that when designing guidelines, priority was given to adaptation to the context in which they were expected to be applied. Thus, a critical review to distinguish between what can be scientifically proven and what could be considered as only an opinion [29] might not be the main priority. The studies we found in our search are the same as those the guideline developers could have found if they had developed the guidelines systematically.
According to the results of our study, clinical guidelines should be reviewed and re-designed with greater scientific rigour. It seems reasonable to make an effort and to develop continuous training programs that would spread the principles stated by the IOM [7,8], and to adopt a model that would help design, in a systematic and planned way, the new guidelines and/or re-design the existing ones [31].
|
Footnotes |
|---|
Address reprint requests to Pedro J. Saturno, Facultad de Medicina, 30100 Campus de Espinardo, Murcia, Spain. E-mail: psaturno{at}um.es
|
References |
|---|
|
TopMethodsResultsDiscussionReferences |
|---|
- Bovim G, Schraeder H, Sand T. Neck pain in the general population. Spine 1994; 15: 1307–1309.
- Makela M, Heliovaara M, Sievers K, Impivaara O, Knekt P, Aromaa A. Prevalence, determinants and consequences of chronic neck pain in Finland. Am J Epidemiol 1991; 34: 1356–1367.
- Borghouts JAJ, Koes BW, Vondeling H, Bouter LM. Cost-of-illness of neck pain in The Netherlands in 1996. Pain 1999; 80: 629–636.[CrossRef][Web of Science][Medline]
- Kvarnstrom S. Occurence of musculoskeletal disorders in a manufacturing industry with special attention to occupational shoulder disorders. Scand J Rehabil Med 1983; 8 (suppl.): 1–114.
- Medina i Mirapeix F, Torres Vaquero AI, Bonilla Solís R, Montilla Herrador J, Escolar Reina P, Valera Garrido JF. Characteristics of physiotherapy protocols developed in primary health care in the period 1990–1996 [in Spanish]. Centro Salud 1999; 7: 697–702.
- Schoenbaum SC, Gottlieb LK. Algorithm based improvement of clinical quality. Br Med J 1990; 301: 1374–1376.
[Free Full Text] - Field MJ, Lohr KN (eds). Clinical Practice Guidelines. Directions for a New Program. Washington, DC: National Academy Press, 1990.
- Field MJ, Lohr KN (eds). Guidelines for clinical practice. From development to use: Washington: National Academy Press, 1992.
- Medina F, Torres AI, Saura J, Saturno P et al. Assessing the structural quality of primary health care physiotherapy clinical protocols (in Spanish). Aten Primaria 1999; 23: 520–525.[Medline]
- Guide to Physical Therapist Practice. Part 1: A description of patient/client management. Chapter 3: What types of interventions do physical therapists provide? Phys Ther 1997; 77: 1213–1226.
- US Preventive Services Task Force. Guide to Clinical Preventive Services: an Assessment of the Effectiveness of 169 Interventions. Baltimore, MD: Williams and Wilkins, 1989.
- Jonsson BG, Persson J, Kilbom A. Disorders of the cervicobrachial region among female workers in the electronics industry: a two-years follow-up. Int J Ind Ergo 1988; 3: 1–12.
- Levoska S, Keinänen-Kiukaanniemi S. Active or Passive Physiotherapy for occupational Cervicobrachial disorders? A comparison of two treatment methods with a 1-year follow-up. Arch Phys Med Rehabil 1993; 74: 425–430.[Web of Science][Medline]
- Revel M, Minguet M, Gregoy P, Vaillant J, Manuel JL. Changes in cervicocephalic kinesthesia after a proprioceptive rehabilitation program in patients with neck pain: a randomized controlled study. Arch Phys Med Rehabil 1994; 75: 895–899.[CrossRef][Web of Science][Medline]
- Hsueh TC, Cheng PT, Kuan TS, Hong CZ. The immediate effectiveness of electrical nerve stimulation and electrical muscle stimulation on myofascial trigger points. Am J Phys Med Rehabil 1997; 76: 471–476.[CrossRef][Medline]
- Cassidy JD, Lopes AA, Yong-Hing K. The immediate effect of manipulation versus mobilization on Pain and Range of Motion in the Cervical Spine: A Randomized Controlled Trial. J Manipulative Physiol Ther 1992; 15: 570–575.[Web of Science][Medline]
- Zylbergold RS, Piper M. Cervical spine disorders. A comparison of three types of traction. Spine 1985; 10: 867–871.[CrossRef][Web of Science][Medline]
- Saturno Hernández PJ, Antón Botella JJ, Murcia Payá JF. Variability in the guidelines for screening, diagnosis and initial assessment of hypertensive patients in 40 health centres [in Spanish]. Med Clin (Barc) 2000; 114 (suppl. 2): 14–18.
- Thomson R, McElroy H, Sudlow M. Guidelines on anticoagulant treatment in atrial fibrillation in Great Britain: variation in content and implications for tratment. Br Med J 1998; 316: 509–513.
[Abstract/Free Full Text] - Vogel N, Burnand B, Vial Y, Ruiz J, Paccaud F, Hohlfeld P. Screening for gestational diabetes: variation in guidelines. Eur J Obstet Gynecol Reprod Biol 2000; 91: 29–36.[CrossRef][Web of Science][Medline]
- Cluzeau FA, Littlejohns P, Grimshaw JM, Feder G, Moran SE. Development and application of a generic methodology to assess the quality of clinical guidelines. Int J Qual Health Care 1999; 2: 21–28.
- Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. J Am Med Assoc 1999; 281: 1900–1905.
[Abstract/Free Full Text] - Ward JE, Grieco V. Why we need guidelines for guidelines: a study of the quality of clinical practice guidelines in Australia. Med J Aust 1996; 165: 574–576.[Web of Science][Medline]
- Ravago TS, Mosniam J, Alem F. Evaluation of community acquired pneumonia guidelines. J Med Syst 2000; 24: 289–296.[Medline]
- Beck C, Cody M, Souder E, Zhang M, Small GW. Dementia diagnostic guidelines: methodologies, results and implementation costs. J Am Geriatr Soc 2000; 48: 1195–1203.[Medline]
- Burgers JS, Bailey JV, Klazinga NS, Van Der Bij AK, Grol R, Feder G. Inside guidelines: comparative analysis of recommendations and evidence in diabetes guidelines from 13 countries. Diabetes Care 2002; 25: 1933–1939.
[Abstract/Free Full Text] - Eisinger F, Geller G, Burke W, Holtzman NA. Cultural basis for differences between US and French clinical recommendations for women at increased risk of breast and ovarian cancer. Lancet 1999; 353: 91920.
- Hart RG, Bailey RD. An assessment of guidelines for prevention of ischemic stroke. Neurology 2002; 59: 977–982.
[Abstract/Free Full Text] - Eddy DM. Clinical decision making: from theory to practice. Designing a practice policy. Standards, guidelines, and options. J Am Med Assoc 1990; 263: 3077–3084.
[Abstract/Free Full Text] - Travell JG, Simons DG. Myofascial Pain and Dysfunction. The Trigger Point Manual. Baltimore, MD: Williams and Wilkins, 1998.
- Saura Llamas J, Saturno Hernández PJ, Clinical Protocols Evaluation and Improvement Group. Clinical protocols: How to develop them? Proposal of a model for their design and classification [in Spanish]. Aten Primaria 1996; 18: 91–96.[Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  J. Vlayen, B. Aertgeerts, K. Hannes, W. Sermeus, and D. Ramaekers A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit Int. J. Qual. Health Care, June 1, 2005; 17(3): 235 - 242. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||