International Journal for Quality in Health Care Advance Access originally published online on February 16, 2006
International Journal for Quality in Health Care 2006 18(3):195-202; doi:10.1093/intqhc/mzi104
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Lack of validity of a French adaptation of a scale measuring attitudes towards clinical practice guidelines
1 Université Paris-Descartes, Service de 6 Neurologie, Hôpital Sainte-Anne Paris, 2 Université Paris-Descartes, AP-HP, Hôpital Cochin Paris, 3 Comité de Coordination de l’Evaluation Clinique et de la Qualité en Aquitaine (CCECQA), Hôpital Xavier Arnozan, Pessac, 4 Santé Publique et Informatique Médicale, Hôpital Européen Georges Pompidou Paris, 5 General Practitioners are from Paris, and 6 Medexact Company, Boulogne-Billancourt, France
Background. Valid instruments to measure practitioners’ attitudes towards clinical practice guidelines need to be developed. However, few of the available instruments have been thoroughly validated.
Objective. To adapt into French and to test the reliability and validity of a scale for measurement of attitudes towards guidelines developed by Elovainio et al.
Methods. A 27-item scale (divided into six dimensions) measuring attitudes towards guidelines was translated into French by two English native translators, reviewed and finalized by expert committee and administered to 314 practitioners who agreed to participate. Main practitioners’ characteristics were collected. Item and dimension reproducibility were assessed for 62 practitioners by calculation of intraclass correlation coefficients. Internal construct validity was assessed by principal components analyses. Convergent and discriminant validity were analysed.
Results. Item response rates ranged from 82 to 100%. In the test–retest procedure, intraclass correlation coefficients for separate items ranged from 0.1 to 0.7 and those for dimensions were 0.7 [95% confidence interval (CI): 0.5–0.8] for usefulness, 0.5 (0.3–0.6) for reliability, 0.4 (0.2–0.5) for individual competence, 0.5 (0.3–0.6) for organizational competence, 0.7 (0.5–0.8) for impracticality and 0.4 (0.3–0.6) for availability. The factorial structure after Varimax rotation showed that none of the different solutions obtained had a strictly comparable structure to that of the original scale. External construct validity was satisfactory.
Conclusion. This scale does not have satisfactory psychometric properties and therefore cannot confidently be used in future research assessing whether attitudes towards guidelines are a determining factor in physicians’ compliance with guidelines. More research is needed to develop valid scales in a more rigorous procedure, involving qualitative and quantitative steps.
Keywords: data validation, physicians’ opinions, practice guidelines, questionnaire, translation
Address reprint requests to Emmanuel Touzé, Université Paris-Descartes, Service de Neurologie, Hôpital Sainte-Anne, 1 rue Cabanis, 75674 Paris Cedex 14, France. E-mail: e.touze{at}ch-sainte-anne.fr
Accepted for publication January 17, 2006.
Clinical practice guidelines are systematically developed statements to assist practitioner and patient decisions concerning appropriate health care for specific clinical circumstances [1]. They are seen as tools for making care more consistent and efficient and for closing the gap between what clinicians do and what scientific evidence supports. Nevertheless, despite wide promulgation, guidelines have limited effect in changing physicians’ behaviours [2–4]. Some qualitative research has suggested that general attitudes on the part of health care providers towards guidelines and implementation strategies may determine physicians’ compliance with guidelines [2,5–7]. These findings have led to the development of scales for measurement of attitudes towards guidelines, which could be helpful in promulgating guidelines. However, although several scales have been proposed [8–13], few have been thoroughly validated [8,11–13], and their reliability has never been assessed. Moreover, it is uncertain whether these scales can be used in different countries with differing cultures. Using various English-language published scales and those developed in RAND and Stakes (National Research and Development Centre for Welfare and Health), Elovainio et al. compiled a 27-item instrument for measuring attitudes towards guidelines [8,10]. Although not published in a peer-review journal, this instrument seemed to have several advantages from the point of view of the present research: it includes many items which have been used in the most prominent published scales, it has been subjected to partial content and construct validation and it has been developed in a European Health System. In the present study, this scale for measurement of attitudes towards guidelines developed by Elovainio et al. [8] was adapted into French and its reliability and validity were tested.
 |
Methods |
|---|
 Top Methods ResultsDiscussionAppendixAcknowledgementsReferences |
|---|
Instrument
The scale, comprising 27 items listed in Table 1, was developed in English before being used in Finnish (personal communication from M Elovainio) [8]. According to results of exploratory factor analysis, the scale is divided into six subscales in relation to guidelines: usefulness, practicability, reliability, availability and individual and organizational competence in the use of guidelines. Responses to each item (in the form of proposed statements) are to be supplied on a 7-point Likert scale ranging from ‘strongly disagree’ to ‘strongly agree’. The authors tested the questionnaire in two population samples and assessed the validity of the scale using exploratory and confirmatory factor analyses. They also proposed a shorter version (14 items) of the scale, divided into seven subscales.
|
Translation process and cultural adaptation
According to standard recommendations [14], two initial translations were made independently by two English native translators (AV, SL) who had experience in health questionnaire translation. Then, in a consensus meeting, all the options were reviewed and the translation choices and cultural adaptations were made. The translation panel for this meeting consisted of researchers, two general practitioners and two translators. The few adaptations made concerned the conceptual content with respect to the French clinical practice environment. For example, in order to use this questionnaire in hospital as well as in private practice, the term ‘structure’ was translated as ‘team’ which was explained to respondents as meaning all colleagues with whom they usually work at the hospital or in a private network. Similarly, ‘team-oriented health-care delivery’ (item 6) was translated by ‘multidisciplinary health-care delivery’ which seemed more appropriate for both kinds of practice in the French context. The translation panel considered that item 17 was difficult to translate because this question does not distinguish practitioners who do not apply guidelines because of their high cost from those who apply guidelines in spite of their cost. Therefore, this item was worded in the conditional tense (‘would be too expensive’) to identify those who do not apply guidelines because of their high cost. ‘Autonomy’ (item 19) was translated as ‘freedom of prescription’ which fitted French practice better. The meeting panel thought that it was obvious that long guidelines with too many pages are inconvenient in real situations (item 22). However, it does not mean that they cannot be implemented. We thought that this item should identify practitioners who do not apply time-consuming long guidelines, rather than merely point out negative opinions on guideline design. Therefore, the item was translated as ‘Long guidelines with too many pages cannot be implemented’. In France, private practitioners are not administratively linked to a central hospital (item 25). However, they are used to interacting with different types of private structures potentially able to provide them with guidelines. Therefore, these structures were proposed as an alternative to central hospital.
Concerning the translation of response choices, the translators produced a comparable 7-point Likert scale, which was discussed and adopted during the meeting. The final translated version is given in the annex. Finally, this first version was administered to five volunteer practitioners who did not suggest any alterations.
Validation sample
From 10 to 25 June 2004, the questionnaire was sent by post to 289 general practitioners and 219 hospital and/or private specialists randomly selected in two French administrative regions. There was one postal mailing with an enclosed reply-paid envelope. In addition, the questionnaire was put on-line on a medical website (http://www.medexact.fr) during the same period. Medexact is a publishing company dedicated to medical practitioners. Its website is consulted by 400 practitioners per day on average. The survey was announced in one newsletter during the study period. Three months later, the questionnaire was sent by post or by e-mail (according to which address was available) a second time to 80 practitioners who were randomly selected among those who had responded in the first phase. Secondarily follow-up calls were made to the practitioners who had not responded after two solicitations.
Data collected
In order to test the convergent and discriminant validity of the scale, practitioners were asked about their main socio-demographic characteristics, if they used guidelines in daily practice (published by the French National Agency for Accreditation and Evaluation in Health or by learned societies) and how frequently they used them, if they had participated in expert-committees establishing guidelines, and if they had any educational activity. Practitioners were also questioned about the impact of guidelines on day-to-day medical decisions using a 6-point ordinal scale from no impact to very strong impact.
Statistics
All statistical analyses were carried out using SAS (8.1). After having reversed items 11–22, 24 and 26, a score was calculated for each dimension by summing the Likert-scale response obtained for each item. Scores were calculated only if a response was obtained for all items of the dimension. The higher the score was, the more positive attitudes towards guidelines the practitioner had. Reproducibility of each item and that of each dimension was assessed by calculation of intraclass correlation coefficients and their 95% CI [15]. For each dimension, Bland and Altman plots of the difference between the two responses in the reproducibility study against the mean of the two responses were also derived [16].
Internal construct validity (or factorial validity) was assessed by principal component analyses (SAS® PROC FACTOR). To determine the number of components to retain before Varimax rotation, Velicer’s minimum average partial method and Horn’s parallel analysis were applied, these being generally the most exact methods [17]. Then a Varimax rotation was applied and the loading values of the variables were analysed. For interpretation, variables with loadings of at least 0.40 on one or more rotated factors were considered to be heavily loaded on that factor. Thus, different structures corresponding to the number of factors retained were determined. Each structure obtained in these analyses was compared with factorial structure of the original scale. A confirmatory factor analysis using the six dimensions of the original scale was also performed (SAS® PROC CALIS). The fit was assessed by calculating the Bentler & Bonett’s index (a good fit is defined by a NFI > 0.90) [17]. Finally, the internal consistency was also assessed by calculating Cronbach’s alpha coefficient for the whole questionnaire and for each domain in the scale. Convergent and discriminant validity were evaluated by the relationships between subscale scores and several external criteria. For these analyses, Student t-tests were used.
|
Results |
|---|
|
TopMethodsResultsDiscussionAppendixAcknowledgementsReferences |
|---|
Practitioner sample characteristics
One hundred and seven out of the 508 practitioners contacted by post and 207 out of the 711 who consulted the questionnaire on the web site completed the questionnaire, providing a total of 314 questionnaires that could be exploited. The main characteristics of the two groups were similar, except that Internet respondents were more often male (78 versus 63%), more often worked in a hospital (48 versus 33%), and more often had some educational activity (39 versus 29%) than postal respondents. Item response rates ranged from 82 to 100% (Table 1).
Attitudes towards guidelines
The proportion of respondent practitioners considering that guidelines are useful and reliable was high, around 85%. Only a minority stated that they had problems of individual or organizational competence. Nevertheless, between 17 and 24% were undecided about these questions and 27% underlined the lack of staff as a potential barrier to guidelines application. It was observed that 33% of the respondents thought that guidelines oversimplified medical practice and 57% thought that long guidelines with too many pages were not applicable. Only 33% of practitioners thought that guidelines were easy to find although they stated they had them available in their work place.
Reliability
Sixty-two out of the 80 who were contacted for the reproducibility analysis returned the questionnaire within a median period of 7.0 (range 3–8) months after the first solicitation. Intraclass correlation coefficients of separate items were overall low ranging from 0.1 to 0.7 (Table 1) and those of dimensions were 0.7 (95% CI: 0.5–0.8) for usefulness, 0.5 (0.3–0.6) for reliability, 0.4 (0.2–0.5) for individual competence, 0.5 (0.3–0.6) for organizational competence, 0.7 (0.5–0.8) for impracticality and 0.4 (0.3–0.6) for availability. Bland and Altman plots for dimensions showed that the limits of agreement, within which 95% of the differences between the two responses are expected to lie, were very similar for all dimensions, most often ranging from 7 to 9 around the median difference. Otherwise, plots did not show systematic positive or negative bias (data not shown; available on request from the authors). We observed that disagreement, defined by an absolute difference 
2 on the Likert scale between the two responses in at least nine items (one-third), was not associated to the main practitioner characteristics: specialty, age, gender, number of consultations per year and participation in expert-committees establishing guidelines.
Internal construct validity
Velicer’s minimum average partial criteria as well as Horn’s partial analysis gave a number of four components to retain in the principal component analysis. Factorial structure after Varimax rotation for this solution, including variables which substantially load (>0.40) on a component, is given in Table 2. This structure did not include items 20 and 22. We also examined structures with five and six components which enabled inclusion of items 20 and 22 but neither was strictly comparable to the structure of the original scale (data not shown). Although the first two domains can be easily differentiated, the others were more mixed. Item 11 (I do not have enough information about guidelines to implement them) was correlated with items 23, 24 and 26 which are related to availability of guidelines. All items relating to individual or organizational problems for implementing guidelines were strongly correlated with one another. The dimension named ‘impracticality’ did not include item 21 (Guidelines are impractical) but included items 25 (Our central hospital provides local treatment programs based on guidelines) and 27 (Guidelines are used in my unit for quality review). Cronbach’s alpha coefficient was 0.6 for the full questionnaire and ranged from 0.6 to 0.9 for the different domains in the questionnaire (Table 1). Confirmatory factor analysis showed a poor fit between our data and the original scale structure (Bentler & Bonett’s index fit = 0.67).
|
External construct validity
Correlations between subscale scores and the different external variables are presented in Table 3. A positive correlation was found between positive attitudes towards guidelines and the impact on day-to-day medical decisions of guidelines published by the French National Agency for Accreditation and Evaluation in Health or by learned societies. Hospital practitioners were more positive towards guidelines than private practitioners, and specialists felt more competent to use guidelines than general practitioners. Those who use guidelines at least once a month had a positive attitude towards practicality and availability of guidelines. Conversely, attitudes towards guidelines were not correlated with age, gender, number of patients seen, educational activity and participation in expert-committees establishing guidelines (data not shown). Results did not differ between postal and Internet practitioner samples.
|
The reliability, internal construct validity and external construct validity of the 14-item scale were very similar to those of the 27-item scale (data not shown, available on request to the authors).
|
Discussion |
|---|
|
TopMethodsResultsDiscussionAppendixAcknowledgementsReferences |
|---|
Understanding why guidelines are not applied in practice is a major challenge [2,6]. As practitioners’ attitude towards guidelines is potentially an important determinant in the process of guidelines application, valid instruments to measure that determinant need to be developed. However this study failed to document satisfactory psychometric properties for the scale used to measure attitudes towards guidelines, although the translation and adaptation procedure was conducted in a thorough manner.
The first finding is that, in our study sample of French practitioners, the reliability of this scale is poor. To our knowledge, the present study is the first that has assessed the reproducibility of such a scale. As it seems unlikely that the translation procedure per se could have altered the reproducibility characteristics, our results could reflect that the original instrument is not reproducible. More interestingly, our findings could also suggest that attitudes of practitioners in general towards guidelines change over time. Because practitioners were slow to respond a second time, the period of time between the two surveys was quite long (median: 7 months). Although most clinicians expressed welcoming attitudes overall towards guidelines, their attitudes may have fluctuated as a result, for instance, of recent positive or negative experiences with use of a particular guideline. If this hypothesis is true, it could partly explain discrepancies between what doctors state they do or think about guidelines and what they actually do [13]. However, to our knowledge, alterations in doctors’ attitudes towards guidelines over time and reasons for such changes have never been specifically assessed. Here, practitioners for whom the two responses disagreed did not differ from those for whom responses were in agreement, but the study was not designed to assess this question and the sample was too small to identify the factors associated with these changes over time.
The second finding is that the internal structure of the French-translated scale was different from that of the original scale, both for the 27-item scale and the shorter 14-item scale. For example, items related to ‘Lack of individual competence’ and ‘lack of organizational competence’ could not be differentiated in factor analysis. Item 11, which is related to information about guidelines, was clustered with items 23, 24 and 26 which are related to availability of guidelines, but not to the ‘individual competence’ dimension. Item 11 (‘I do not have enough information...’) is indeed probably more related to the notion of ‘availability’ than to that of ‘individual competence’.
Conversely, acceptability and construct validity of the scale were relatively satisfactory. In participants, item response rates ranged from 82 to 99%. As expected, positive attitudes were more frequent among practitioners who stated that they used guidelines and that they had some impact on their practice, and specialists were more familiar with guidelines, because they probably use the guidelines specific to their own speciality [11]. However, no relationship between positive attitudes and participation in expert committees establishing guidelines was found.
Although we consider our results to be valid, the present study has several limitations. Firstly, as the response rate was low, the results cannot readily be generalized. The study found that attitudes of practitioners about guidelines were generally positive, as in almost all comparable previously published studies, where the response rates were close to the present results [9–12,18]. Therefore, it is not known how this scale would behave in a population of practitioners with less positive attitudes towards guidelines, although there is no reason to believe that it would be very different. Secondly, this scale assessed attitudes towards guidelines in general. It has been previously shown that doctors’ confidence in guidelines may depend on the topic of the guidelines as well as on the guidelines provider [5,7,11]. However, even in the few studies devoted to specific guidelines, the reliability of the scales used has never been assessed [7,19–21]. In addition, because there is a growing number of guidelines published by government agencies and learned societies all over the world [3], it would be very difficult to develop and validate a scale for each guideline. Thirdly, as the period of time between the two surveys in the present study was quite long, practitioners’ attitudes may have changed in relation to health care policy changes. However, Bland and Altman plots did not show any positive or negative bias that could have suggested such systematic changes in attitude.
In conclusion, the scale adapted here cannot confidently be used in future research assessing whether attitudes towards guidelines are a determinant in physicians’ compliance with guidelines. This may be also the case for the other previously published scales which may not have been thoroughly validated. For example, there is very little information on the way these questionnaires were generated (e.g. qualitative steps are often lacking). Therefore, more research is needed to develop valid scales in a more rigorous and thorough manner, using qualitative and quantitative steps as it has been successfully done in other health fields such as that of quality of life or satisfaction with care [22,23]. It would be also important to determine why attitudes about whether guidelines are good or bad vary so much and whether these variations are due to real changes in attitude towards guidelines or to weaknesses in the instruments used [1].
|
Appendix |
|---|
|
TopMethodsResultsDiscussionAppendixAcknowledgementsReferences |
|---|
|
|
Acknowledgements |
|---|
|
TopMethodsResultsDiscussionAppendixAcknowledgementsReferences |
|---|
This study was funded through a grant from Servier Medical which had no role in the collection, analysis, or interpretation of the data or in the decision to publish the manuscript. The authors acknowledge the help of S. Helfen and the staff of ‘Unité de Recherche Clinique Cochin-Port Royal’ in conducting the study.
|
References |
|---|
|
TopMethodsResultsDiscussionAppendixAcknowledgementsReferences |
|---|
- Woolf SH, Grol R, Hutchinson A et al. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. Br Med J 1999; 318: 527–530.
[Free Full Text] - Cabana MD, Rand CS, Powe NR et al. Why don’t physicians follow clinical practice guidelines. JAMA 1999; 282: 1458–1465.
[Abstract/Free Full Text] - Grol R. Successes and failures in the implementation of evidence-based guidelines for clinical practice. Med Care 2001; 39: II46–II54.
- Lenfant C. Clinical research to clinical practice – lost in translation? N Engl J Med 2003; 349: 868–874.
[Free Full Text] - Grol R, Dalhuijsen J, Thomas S et al. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. Br Med J 1998; 317: 858–861.
[Abstract/Free Full Text] - Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet 2003; 362: 1225–1230.[CrossRef][Web of Science][Medline]
- Saillour-Glenisson F, Michel P. Facteurs individuels et collectifs associés à l’application des recommandations de pratique clinique par le corps médical. Revue de la littérature. Rev Epidemiol Santé Publique 2003; 51: 65–80.[Web of Science][Medline]
- Elovainio M, Eccles M, Mäkelä M. Attitudes towards guidelines and a scale for measuring them. In Thorsen T, Mäkelä M, eds. Changing Professional Practice. Theory and Practice of Clinical Guidelines Implementation. Copenhagen: Danish Institute for Health Services Research and Development, 1999: 153–167.
- Hayward RS, Guyatt GH, Moore KA et al. Canadian physicians’ attitudes about and preferences regarding clinical practice guidelines. CMAJ 1997; 156: 1715–1723.[Abstract]
- Mansfield CD. Attitudes and behaviors towards clinical guidelines: the clinicians’ perspective. Qual Health Care 1995; 4: 250–255.
[Abstract/Free Full Text] - Tunis SR, Hayward RS, Wilson MC et al. Internists’ attitudes about clinical practice guidelines. Ann Intern Med 1994; 120: 956–963.
[Abstract/Free Full Text] - Watkins C, Harvey I, Langley C et al. General practitioners’ use of guidelines in the consultation and their attitudes to them. Br J Gen Pract 1999; 49: 11–15.[Web of Science][Medline]
- Grol R. National standard setting for quality of care in general practice: attitudes of general practitioners and response to a set of standards. Br J Gen Pract 1990; 40: 361–364.[Web of Science][Medline]
- Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: literature review and proposed guidelines. J Clin Epidemiol 1993; 46: 1417–1431.[CrossRef][Web of Science][Medline]
- Shrout PE, Fleiss J. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979; 86: 420–428.[CrossRef][Web of Science][Medline]
- Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1: 307–310.[CrossRef][Web of Science][Medline]
- Coste J, Bouée S, Ecosse E et al. Methodological issues in determining the dimensionality of composite health measures using principal component analysis: case illustration and suggestions for practice. Qual Life Res 2005; 14: 641–654.[CrossRef][Web of Science][Medline]
- Newton J, Knight D, Woolhead G. General practitioners and clinical guidelines: a survey of knowledge, use and beliefs. Br J Gen Pract 1996; 46: 513–517.[Web of Science][Medline]
- Beaulieu MD, Brophy J, Jacques A et al. Physicians’ attitudes to the pharmacological treatment of patients with stable angina pectoris. Q J M 2005; 98: 41–51.
- Graham ID, Evans WK, Logan D et al. Canadian oncologists and clinical practice guidelines: a national survey of attitudes and reported use. Provincial lung disease site group of cancer care ontario. Oncology 2000; 59: 283–290.[CrossRef][Web of Science][Medline]
- Hobbs FDR, Erhardt L. Acceptance of guideline recommendations and perceived implementation of coronary heart disease prevention among primary care physicians in five European countries: the Reassessing European Attitudes about Cardiovascular Treatment (REACT) survey. Fam Pract 2002; 19: 496–504.
[Abstract/Free Full Text] - Leplège A, Ecosse E, Verdier A et al. The French SF-36 health survey: translation, cultural adaptation and preliminary psychometric evaluation. J Clin Epidemiol 1998; 51: 1013–1023.[CrossRef][Web of Science][Medline]
- Ware JE, Gandek B for the IQOLA project group. Methods for testing data quality, scaling assumptions, and reliability: the IQOLA project approach. J Clin Epidemiol 1998; 11: 945–952.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  E. Touze, J. Coste, M. Voicu, J. Kansao, R. Masmoudi, B. Doumenc, P. Durieux, and J.-L. Mas Importance of In-Hospital Initiation of Therapies and Therapeutic Inertia in Secondary Stroke Prevention: IMplementation of Prevention After a Cerebrovascular evenT (IMPACT) Study Stroke, June 1, 2008; 39(6): 1834 - 1843. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||