Skip Navigation

This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrow Search for citing articles in:
ISI Web of Science (2)
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Bindels, R.
Right arrow Articles by Winkens, R. A. G.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Bindels, R.
Right arrow Articles by Winkens, R. A. G.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?

International Journal for Quality in Health Care 15:501-508 (2003)
© 2003 International Society for Quality in Health Care

User satisfaction with a real-time automated feedback system for general practitioners: a quantitative and qualitative study

Rianne Bindels1, Arie Hasman1, Mieke Derickx1, Jan W. J. van Wersch2 and Ron A. G. Winkens3,4

1 Department of Medical Informatics and
4 Department of General Practice, Maastricht University, Maastricht,
2 Atrium Medical Center Heerlen, Heerlen,
3 Transmural Care Unit, University Hospital Maastricht, Maastricht, The Netherlands

Objective. The GRIF automated feedback system produces real-time comments on the appropriateness of diagnostic tests ordered by general practitioners (GPs) based on recommendations from accepted national and regional practice guidelines. We investigated the experiences of GPs with this system and, more specifically, with the recommendations produced by the system as well as their views on using this system in daily practice.

Setting. We tested the GRIF system in an experiment in a laboratory setting and in a daily practice trial.

Study participants. General practitioners.

Intervention. In the laboratory experiment, GPs used the GRIF system to assess the appropriateness of 30 request forms. Each of the GPs was confronted with requests they had submitted to the diagnostic unit of the hospital in the past. In the field trial, the GRIF system was applied during patient consultations for 1 year.

Main outcome measures. We measured GPs’ satisfaction with the system using a questionnaire, and also conducted group discussions (in the laboratory experiment) and in-depth interviews (in the field trial) to elicit GPs’ opinions of and experiences with the system. In addition, we explored GPs’ reasons for not accepting the comments offered by the GRIF system.

Results. The results show that the GPs in the laboratory experiment had more positive attitudes towards the system compared with participants in the field trial. All discussion groups and most of the GPs in the field trial regarded receiving the immediate feedback during the test ordering process as an important advantage. The most frequently mentioned reason to reject the recommendation was disagreement with the content and/or the recommendations in the practice guidelines.

Conclusion. Apart from securing agreement on guideline content, a prerequisite for using GRIF in daily practice on a large scale is that more attention is paid to promotion of the guidelines and their adoption, and stimulation of a positive attitude towards the practice guidelines among the users.

Keywords: clinical decision support systems, guideline adherence, primary health care, user-satisfaction

Accepted for publication August 5, 2003.

Practice guidelines are becoming more and more common in the medical profession [1]. However, implementation of these guidelines and adherence to them in daily practice are still problematic [2]. Some of the reasons for physicians not using or following practice guidelines are lack of awareness of the content of a guideline and lack of familiarity or agreement with these guidelines [3]. Different strategies have been developed to improve the adherence to practice guidelines in daily practice [4]. In recent years, automated feedback systems, akind of decision support system, are used to remind physicians of the content of practice guidelines during patient consultation. Automated feedback systems are being developed to advise physicians on diagnostic test ordering [5–7], prescription of drugs [8–11], and preventive actions in an ambulatory setting [12].

We developed and validated the so called GRIF real-time automated feedback system [13,14]. GRIF is the acronym for the Dutch phrase ‘Geautomatiseerde Reminders als Interactieve Feedback’ (Automated Reminders as Interactive Feedback). The GRIF system is intended to stimulate adherence to practice guidelines in diagnostic test ordering. This system was developed to support and eventually replace the written feedback that has been given by the Transmural Care Unit (TCU) of the Maastricht University Hospital since 1985 [15,16]. Twice ayear, general practitioners (GPs) in the Maastricht region receive personal feedback (based on accepted and evidence-based regional practice guidelines) on the adequacy of their test ordering behavior. Although this method proved to be effective and was appreciated by the GPs [16,17], a more direct and less laborious method would be preferred.

The GRIF system contains an electronic request form to be filled in during patient consultation. It produces comments about the appropriateness of the requested diagnostic tests ordered before the request form is submitted to the TCU [13]. The present study was carried out to investigate experiences of GPs with the GRIF system itself and, more specifically, with the recommendations produced by the system, as well as their views on the use of such a system in daily practice. We were also interested in the reasons for non-acceptance of the recommendations presented by the GRIF system. These reasons may be helpful in tracing factors that influence the implementation process and may assist in finding better strategies toimplement guidelines and ways to provide indications on how to adjust such practice guidelines.


   Materials and methods
Top
 Materials and methods
Results
 Discussion
 Conclusion
 Appendix 1
 References
 
The GRIF system
The GRIF system consists of five parts: a knowledge base, an order entry module, a module that provides reactive support (i.e. the recommendations), a module that provides passive support, and a database. The order entry system is not yet fully integrated with the Electronic Patient Record (EPR) of the GPs, but has an online connection with the EPR via an intermediate database. Relevant patient-related data (name, address, date of birth, gender, medication, and existing problems) are being transferred from the EPR to the GRIF system to prevent unnecessary double registration of patient data.

Additional medical patient data (such as working hypotheses, signs and symptoms, and the reason for the test order) and the desired diagnostic tests have to be entered into an electronic order entry form. The medical terms have to be entered using International Classification of Primary Care (ICPC) codes [18]. A search program assists the user in the selection of the appropriate terms that are associated with an ICPC code.

The knowledge base in which the recommendations are stored contains 150 rules (recommendations) derived from regional and national practice guidelines for treating various medical problems [13]. The input for the conditions of the rules may contain a requested test, medical patient information, personal information (age and gender), or a combination of these types of information. The knowledge base of the GRIF system contains roughly five types of recommendations:

  1. Advice not to request a test for a specified working hypothesis and/or complaint.
  2. Advice that checks for the presence of a particular working hypothesis and/or complaint when a certain test is requested and critiques the request when that (coded) working hypothesis and/or complaint is not present.
  3. Advice to omit a test because another more appropriate or more efficient test was also requested.
  4. Advice not to request a test because of the characteristics of the patient such as age and/or gender.
  5. Advice to order a more appropriate or efficient test instead of the planned test.

A recommendation or point of advice is presented to the GP whenever a test order is not in accordance with the guidelines (reactive support module). The GP can indicate whether he or she wants to follow or ignore the recommendation. If the GP accepts a recommendation the corresponding request form will be changed automatically according to the advice. When needed, the system is able to provide additional context-specific background information about the guidelines when a recommendation is presented (passive support module).

Data collection
We collected data during an experiment in a laboratory setting (laboratory experiment) and during a trial in daily practice (field trial). The laboratory experiment was conducted at our university. During the field trial the participating GPs used the system in their own practices. In the laboratory experiment we confronted 24 GPs with the recommendations generated by our system. We randomly selected for each GP a set of 30 request forms that he or she had submitted earlier to the Transmural Care Unit. Patient information and the requested tests were entered into the automated feedback system by one of the authors (R.B.). The request forms were made anonymous to prevent knowledge of the test result influencing the GPs’ judgements.

Each GP had to go through the cases one-by-one using the GRIF system within a limited period of time to simulate the time pressure during patient consultation.

We added a pop-up window to the system that appeared when the GP rejected a recommendation. In this window the GP had to describe, in free text, the reason for the decision to reject the recommendation. The laboratory experiment was conducted in four sessions, with four to eight GPs. After each session, we measured user-satisfaction with the GRIF system using a questionnaire based on the IBM computer usability satisfaction questionnaire [19]. The questionnaire consisted of 15 items in total: six about usability, four about user satisfaction, three about the expected change in test-ordering behavior, and two about the GRIF system compared with the written feedback method. The changes in the IBM scale and the questions that are added to the questions in the IBM scale are listed in Appendix 1. All items were categorized on a five-point Likert scale, where 1 = ‘strongly agree’ and 5 = ‘strongly disagree’. Finally, a semi-structured group discussion led by one of the authors (R.W.) was conducted to obtain the GPs’ opinions about the GRIF system, its recommendations, the obstacles for its use in daily practice, and suggestions for improvement. The discussions lasted 30–45 min.

In the field trial, the GRIF automated feedback system was introduced and used in 21 practices (32% of those eligible to use the system). Of these practices 10 practices with 11 GPs participated for >=1 year. Fourteen GPs from 11 practices dropped out of the field trial; 10 due to technical problems (mainly caused by an unexpected revision of the GPs’ information system by the vendor), three because they could not (or did not want to) integrate the routine use of the system into their busy daily practice, and one because he suffered from a protracted illness during the trial.

A questionnaire about user satisfaction with the automated feedback system was sent to all initial participants of the field trial. We used the same questionnaire as for the laboratory experiment. In-depth interviews were held by one of the authors (M.D.) in the GPs’ practices. The themes covered in the interviews included opinions about the automated feedback system, problems during the intervention period, and experiences with the recommendations produced. The interviews usually lasted 30–45 min.

Analyses
We calculated the sum scores for usability, user satisfaction, expectation of change in ordering behavior, and opinions about the GRIF system versus written feedback (Maastricht region only). Median scores and 25th and 75th percentiles for each sum score were determined. Pair-wise differences between the three groups (laboratory experiment participants, field trial participants, and field trial drop-outs) were tested using the Mann–Whitney test.

The group discussions and interviews were audio taped, transcribed, and analyzed using QSR Nud*IST®, a software package for analysis of qualitative data. The transcriptions were analyzed by selecting and reorganizing responses according to themes by one of the authors (R.B.) [20].


   Results
Top
 Materials and methods
 Results
Discussion
 Conclusion
 Appendix 1
 References
 
GP characteristics are listed in Table 1. The random selection of request forms for the laboratory experiment contained 4196 tests (mean 5.6 tests per form) and in the field trial 10139 tests were ordered (mean 4.1 tests per form).


View this table:
[in this window]
[in a new window]
  		Table 1 Characteristics of the GPs in the laboratory experiment and field trial

 

User satisfaction questionnaire
All GPs participating in the field and the laboratory experiment, and 86% of those GPs who dropped out of the field trial completed the questionnaire. The results presented in Table 2 show that not all GPs completed the whole questionnaire. All questionnaire aspects were considered positive by the laboratory participants (median scores and 75th percentile < 3.0). Both field trial participants and drop-outs were less positive about measured aspects (user friendliness, user satisfaction, expected change in behavior, and automated versus written feedback). The drop-outs were even more negative about the GRIF system compared with the written feedback (median scores and 75th percentile >=4.0) (Table 2).


View this table:
[in this window]
[in a new window]
  		Table 2 Median, and 25th and 75th percentile scores for the sum scores for four topics in the study questionnaire

 

Compared with those GPs who took part in the field trial (participants and drop-outs), GPs in the laboratory experiment were significantly more positive about user friendliness (P < 0.01 for both drop-outs and participants) and more satisfied with the GRIF system (P < 0.01 for drop-outs and P = 0.02 for participants) (Table 3). Also, opinions on the influence of the GRIF system on the GPs’ test-ordering behavior (P = 0.02 for drop-outs and P < 0.01 for participants) and on substitution of the GRIF system for the written feedback method (P = 0.03 for drop-outs and P < 0.01 for participants) were significantly different compared with those of the laboratory experiment participants. No significant differences were found for the above-mentioned analyses between the GPs who dropped out of the field trial and the GPs who used the GRIF system for >=1 year.


View this table:
[in this window]
[in a new window]
  		Table 3 P-values of Mann–Witney tests for comparisons of different groups

 

Opinions and experiences
The results of the group discussion (laboratory experiment) and the in-depth interviews (field trial) are divided into: (i) views about and experiences with the GRIF system; (ii) experiences with recommendations (accuracy and effectiveness); and (iii) substitution of the GRIF system for the written feedback method.

All discussion groups regarded the feedback during the test-ordering process as a major advantage. The respondents considered the system to be more effective than the written feedback because it is possible to adjust the request form immediately based on the feedback. GPs expected this to have a stimulating effect on their test-ordering behavior. In contrast, feedback and the critical reflection on one’s own test-ordering behavior were only mentioned as a positive issue a few times during the interviews (field trial). One respondent stated ‘The comments of the automated feedback system were helpful to me in explaining to the patient why a particular diagnostic test is not necessary’. GPs in the field trial often mentioned the background information available in the GRIF system as being a positive aspect of the system. However, almost all GPs admitted that they did not read this background information during patient consultation. In the field trial, an appreciated feature of the GRIF system was the readability and improved structure of the printed request form.

The dependence of the recommendations on the amount and quality of the patient’s medical information provided was seen as an important problem by most GPs. A few respondents mentioned that instead of being corrected by recommendations of the GRIF system, they preferred to be guided by the system. A majority of the field trial participants mentioned that the GRIF system took too much time during patient consultation due to the coding of the patient’s medical information, the slowness of their own computer system and searching for difficult to find, infrequently requested tests. Another negative aspect mentioned was the need to enter a specific and concrete working hypothesis on the electronic request form.

Experiences with recommendations. Almost all GPs in both trials found that the recommendations presented were correct in most of the situations. Some GPs in the field trial thought that the use of the system had not changed their test-ordering behavior. An important reason was that GPs grew accustomed to the recommendations and ignored them, sometimes even without reading the recommendation. Only three field-trial GPs said that recommendations given repeatedly sometimes irritated them. Five GPs said that their behavior had indeed changed. They mentioned that they omitted some previously routinely ordered diagnostic tests such as leukocyte counts and liver tests.

Substitution for written feedback. All GPs indicated that they considered the GRIF system a good alternative to the written feedback. Some GPs mentioned that the GRIF system could be an additional method for providing feedback, but could never replace the personal character of the written feedback. Because of the anonymous character of the automated feedback system it was regarded as having less impact on their test-ordering behavior. A few GPs mentioned that they missed the overviews of tests ordered in a particular period of time that were present in the written feedback.

Reasons for non-acceptance of recommendations
Five main themes emerged from the data in the laboratory experiment and the in-depth interviews of the field trial. In the laboratory experiment the GPs had to indicate a reason for non-acceptance each time they ignored a recommendation of the GRIF system (Table 4).


View this table:
[in this window]
[in a new window]
  		Table 4 Reasons for non-acceptance of the comments of the GRIF system and the number (and percentage) of times the explanation was given in the laboratory experiment

 

Practice guidelines. If the GPs in the laboratory experiment ignored a recommendation, they mostly did so because they simply did not agree with the recommendation. A few times they admitted they were not familiar with a particular practice guideline. GPs interviewed in the field trial indicated that they agreed with the majority of the recommendations. Three GPs mentioned that advice about vague complaints such as fatigue was often ignored. One of them stated ‘I want to order more tests than mentioned in the practice guidelines because this might help me to come up with an explanation for this fatigue in a later phase of the diagnostic process’.

Provided information. A number of recommendations indicated that a test was ordered that did not accord with the guidelines because there was no indication of a particular disease or complaint. GPs in the laboratory experiment frequently indicated that they did not mention the particular disease or complaint on the request form, but they did have one in mind. In addition, the amount of information they provided and its quality were sometimes not sufficient to assess the appropriateness of the recommendation.

In a number of situations a recommendation was presented incorrectly according to the GPs in both the laboratory experiment and in the field trial. According to the GPs, the automated feedback system did not take relevant information (such as the duration of the complaint) into account, and this was most frequently the cause of an incorrectly presented recommendation. In the field trial, GPs also mentioned that they did not code all relevant medical information (mostly concerning complaints) using the ICPC-coding module of the GRIF system due to lack of time or because they could not find a satisfying corresponding ICPC description.

Medical considerations. Relatively frequently, GPs in both trials indicated that they wanted more information about the patient for check-up purposes or preventive screening. Arecurring comment of several GPs was: ‘I want to have ageneral impression of the patient’s health status’. In the laboratory experiment, twice a GP could not agree with the recommendations presented because a specialist had advised her to order the test.

GP and practice characteristics. In some instances, GPs wanted to order tests although they knew this was not in accordance with the guidelines. They needed more certainty to exclude a particular working hypothesis; for example ‘I just don’t dare to rely on a rectal examination only, when I suspect a benign prostatic hyperplasia’.

In a few situations, the GPs also indicated that they did not adhere to the advice of the recommendation for logistic reasons. For example, when a non-fasting glucose was ordered instead of a fasting glucose, one of the GPs stated ‘The elderly patient lives in a retirement home. The time the blood sample is taken depends on the geriatric helper’.

Patient. A minor part of the reasons for rejection of the recommendations concerned test orders that were made at the patient’s request. In the field trial the GP’s relationship with the patient was mentioned more often than in the laboratory experiment. One GP stated ‘Yes, I easily go along with the patient. I find it difficult to say no to the patient and explain why I do not order the test. I’m left with a strange feeling about it’. Another GP stated ‘I order a few simple and general diagnostic tests to break out of an impasse’.


   Discussion
Top
 Materials and methods
 Results
 Discussion
Conclusion
 Appendix 1
 References
 
We focused on two aspects of the use of the GRIF automated feedback system: user-satisfaction and the reasons why GPs reject the recommendations.

A relatively low number of GPs participated in the field trial, which may have influenced the reliability of the quantitative results of the questionnaire. With low numbers there may be ahigher chance of selection bias of results due to the selection of GPs included in the study. The chance of such a bias is low in our laboratory experiment, as a large majority of those—randomly selected—GPs immediately agreed to participate. However, due to a relatively high number of drop-outs in the field trial, the chance of selection bias is present. The qualitative results on the experiences with the GRIF system and the reasons for non-acceptance of the system’s recommendations presented in this paper were less influenced by the low number of participating GPs [20].

The questionnaire showed that the GPs in the laboratory experiment were positive about the GRIF system, and certainly more positive compared with the participants and drop-outs in the field trial. The use of the system in the laboratory experiment differed from the use in daily practice in a number of aspects. The GPs in the laboratory experiment did not have to enter and/or code any medical patient data or diagnostic tests, and they did not have to print the request form. Measurements in daily practice showed that it took GPs a median time of 77 seconds (25th and 75th percentiles, 49 and 121 seconds, respectively) to complete the order entry form of the GRIF system [21]. The extra time that the use of the system and the coding of medical information takes, in combination with the relatively short time for patient consultation, is seen as the main obstacle to the use of the GRIF system in daily practice. Coding medical terms using ICPC codes takes time, and an additional disadvantage lies in the fact that the GP cannot translate the entire medical context of the request for diagnostic tests into such codes [22,23].

Although most of the respondents in the laboratory experiment were positive about the written feedback method, they all thought that the GRIF system could provide a good alternative. Some GPs preferred a decision support system that could give them more guidance during the test-ordering process. Our reactive system assesses the GP’s decision afterwards, while other pro-active systems suggest a list of possibilities based on the patient’s medical problem, such as the Bloodlink system [24]. Using a more protocol-based approach, the rules in the GRIF knowledge base and the GRIF system itself can easily be changed into a system that supports both approaches, allowing GPs to choose their preferred approach.

Also, the suggestion to add overviews of the tests requested and recommendations presented could be beneficial to the whole feedback process. This is in line with the results of other researchers who have concluded that a combination of multiple interventions achieves a more adequate use of diagnostic tests [4,25].

Innovation may not only require user friendliness, but also (or mainly) information, user involvement in the design and development process [26,27], and change agents [28]. Change agents promote an innovation and can influence the opinion of others about it; the innovation being the GRIF system in our case. In this study we focus on user friendliness. However, in the design phase of the study other potential users were also involved and the system was introduced to the GPs during a symposium [13].

Apart from medical reasons, other important reasons for non-acceptance of recommendations appeared to be the content of and disagreement with the practice guidelines, and the generation of general recommendations as a result of the poor quality of the patient medical information provided. GPcharacteristics such as habitual test-ordering routines, tolerance of diagnostic uncertainty, and attitude towards the guidelines also play a role in this process. Grol et al. identified that a positive attitude towards practice guidelines and their use in daily practice are important prerequisites for a successful implementation of practice guidelines [27]. External factors such as practice organization, costs of the tests, and time pressure are mentioned relatively less often. The reasons for ignoring recommendations presented in this study are also mentioned in other qualitative studies about the implementation of evidence in general practice [29,30] and about test-ordering behavior specifically [31].

Litzelman and Tierney [32] analyzed reasons for non-acceptance of automated recommendations in a hospital environment in more detail. They found that the recommendations were rejected because they were ‘not applicable’ (41%), ‘reacted upon in the next visit because the physician was too busy’ (41%), and ‘were refused by the patient’ (18%). Our results show that GP characteristics such as tolerance to diagnostic uncertainty and attitude towards the guidelines are also important reasons for non-acceptance of automated feedback messages. Van Wijk et al. [33] concluded that non-compliance with advice from an automated decision support system based on practice guidelines for diagnostic blood test ordering seemed to be partially based on the fact that GPs applied new medical insights before this knowledge was incorporated in a revision of the guidelines. In the present study, only one of the GPs mentioned this reason for non-compliance. The influence of the patient was mentioned relatively frequently in the field trial. The role of the patient, nowadays, is considered ‘the key medical decision maker’ in the decision process, and is becoming more and more important [34].

One of our aims was to identify reasons why GPs rejected recommendations. Although these reasons are related to the content of the recommendations and thus the content of the guidelines, there may be more general reasons why GPs reject advice of automated feedback systems. To overcome resistance to guidelines and to improve adherence to automated advice presented, the reasons why GPs reject these must be clarified. Reassurance of his/her own capacity to come up with the right diagnosis or treatment is a reason that requires an entirely different approach to overcome disagreement with the content of the guideline. During the implementation of guidelines, these different behavioral aspects need to be taken into account.


   Conclusion
Top
 Materials and methods
 Results
 Discussion
 Conclusion
Appendix 1
 References
 
We conclude that it takes more than ‘satisfied customers’ for practice guidelines to be implemented through an automated system that provides feedback directly during the test-ordering process. In general, non-acceptance of the automated recommendations is mainly caused by not taking (or not being able to take) time to provide sufficient information and by disagreement with the guidelines involved. Therefore, we recommend paying much more attention to one of the steps in the implementation cycle, namely the promotion of the guidelines, and the stimulation of a positive attitude towards the guidelines among users.


   Appendix 1
Top
 Materials and methods
 Results
 Discussion
 Conclusion
 Appendix 1
References
 
Changes compared with the IBM scale [19]
We changed the following questions:

Question 10 of the IBM scale was divided into two questions in the GRIF questionnaire: a question about the ease of recovering mistakes and a question about quickness of recovering mistakes.

Question 11 of the IBM scale was divided into two questions in the GRIF questionnaire: a question about the online help and a question about the paper manual.

Question 12 of the IBM scale was divided into two questions in the GRIF questionnaire: a question about the ease of finding information and a question about quickness of finding information.

Question 14 of the IBM scale was divided into two questions in the GRIF questionnaire: a question about the reduction of the number of requested tests as a result of using GRIF and about the change in type of requested tests as a result of using GRIF.

Question 17 of the IBM scale was not used in the GRIF questionnaire, because translation of this question would lead to about the same formulation as question 16 of the IBM scale.

We added questions on the following topics:

The ease of use of the guidelines in the GRIF system.

The overall usefulness of the GRIF system.

The ease to fill in complex test order forms using GRIF.

The appropriateness of the GRIF recommendations.

The presentation of too few recommendations by the GRIF system.

ICPC-module: the satisfaction with the ICPC-module and finding the right terms using the ICPC-module.

The use of a computer makes the GP more aware of the need to request tests.

The satisfaction with the GRIF system compared with the written feedback.

The usefulness of the GRIF system compared with the written feedback.

The usefulness of the GRIF system for the patient.

The preference to have recommendations before ordering tests rather than afterwards.

Being prepared to code more medical information in order to use decision support.

The desire to send the request form electronically.

Registering the requested tests in the GP information system.


   Footnotes
 
Address reprint requests to Rianne Bindels, Maastricht University, Department of Medical Informatics, PO Box 616, 6200 MD Maastricht, The Netherlands. E-mail: r.bindels{at}mi.unimaas.nl Back


   References
Top
 Materials and methods
 Results
 Discussion
 Conclusion
 Appendix 1
 References
 

  1. Woolf SH, Grol R, Hutchinson A et al. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. Br Med J 1999; 318: 527–530.[Free Full Text]

  2. Grol R, Eccles M, Maisonneuve H et al. Developing clinical practice guidelines: the European experience. Dis Manage Health Outcomes 1998; 4: 255–266.[CrossRef]

  3. Cabana MD, Rand CS, Powe NR et al. Why don’t physicians follow clinical practice guidelines? A framework for improvement. JAm Med Assoc 1999; 282: 1458–1465.[Abstract/Free Full Text]

  4. Wensing M, van der Weijden T, Grol R. Implementing guidelines and innovations in general practice: which interventions are effective? Br J Gen Pract 1998; 48: 991–997.[Web of Science][Medline]

  5. Tierney WM, McDonald CJ, Hui SL et al. Computer predictions of abnormal test results: effects on outpatient testing. J Am Med Assoc 1988; 259: 1194–1198.[Abstract/Free Full Text]

  6. Bates DW, Kuperman GJ, Rittenberg E et al. Reminders for redundant tests: results of a randomized controlled trial. Proc Annu Symp Comput Appl Med Care 1995; 19: 935.

  7. Modai I, Sigler M. A computerized laboratory alerting system in a psychogeriatric unit. MD Comput 1998; 15: 95–99.[Medline]

  8. Persson M, Mjorndal T, Carlberg B et al. Evaluation of a computer-based decision support system for treatment of hypertension with drugs: retrospective, nonintervention testing of cost and guideline adherence. J Intern Med 2000; 247: 87–93.[CrossRef][Web of Science][Medline]

  9. Teich JM, Merchia PR, Schmiz JL et al. Effects of computerized physician order entry on prescribing practices. Arch Intern Med 2000; 160: 2741–2747.[Abstract/Free Full Text]

  10. Walton RT, Harvey EL, Dovey S et al. Computerised advice on drug dosage to improve prescribing practice [Cochrane Review]. Cochrane Database Syst Rev 2001; 1.

  11. Evans RS, Pestotnik SL, Classen DC et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med 1998; 338: 232–238.[Abstract/Free Full Text]

  12. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. JAm Med Inform Assoc 1996; 3: 399–409.

  13. Bindels R, de Clercq P, Winkens RAG et al. A test ordering system with automated reminders for primary care based on practice guidelines. Int J Med Inform 2000; 58–59: 219–233.

  14. Bindels R, Winkens RAG, Pop P et al. Validation of a knowledge based reminder system for diagnostic test ordering in general practice. Int J Med Inform 2001; 64: 341–354.[CrossRef][Web of Science][Medline]

  15. Pop P, Winkens RAG. A diagnostic centre for general practitioners: results of individual feedback on diagnostic actions. J R Coll Gen Pract 1989; 39: 507–508.[Web of Science][Medline]

  16. Winkens RAG, Pop P, Bugter-Maessen AMA et al. Randomised controlled trial of routine individual feedback to improve rationality and reduce numbers of test request. Lancet 1995; 345: 498–502.[CrossRef][Web of Science][Medline]

  17. Winkens RAG, Pop P, Grol RPTM et al. Effects of routine individual feedback over nine years on general practitioners’ requests for tests. Br Med J 1996; 312: 490.[Free Full Text]

  18. Lamberts H, Wood M. International Classification of Primary Care, 3rd edition. Oxford: Oxford University Press, 1987.

  19. Lewis JR., IBM computer usability satisfaction questionaires: psychometric evaluation and instructions for use. Int J Human-Comput Interact 1995; 7: 57–78.

  20. Mays N, Pope C. Qualitative research in health care. Assessing quality in qualitative research. Br Med J 2000; 320: 50–52.[Free Full Text]

  21. Bindels R., To test or not to test? Guideline-based automated feedback on test ordering in general practice. Maastricht: Maastricht University, 2003.

  22. Hofman-Okkes IM, Lamberts H. The International Classification of Primary Care (ICPC): new applications in research and computer-based patient records in family practice. Fam Pract 1996; 13: 294–302.[Abstract/Free Full Text]

  23. Gebel RS. De ICPC-1: Stand van zaken, praktische problemen en oplossingen [The ICPC-1: State of the art, practical problems and solutions]. Huisarts Wet 2000; 41: 20–22.

  24. van Wijk MAM, van der Lei J, Mosseveld M et al. Assessment of decision support for blood test ordering in primary care. A randomized trial. Ann Intern Med 2001; 134: 274–281.[Abstract/Free Full Text]

  25. Wensing M, Grol R. Single and combined strategies for implementing changes in primary care: a literature review. Int J Qual Health Care 1994; 6: 115–132.[Abstract/Free Full Text]

  26. van der Meijden MJ, Tange H, Troost J et al. Development and implementation of an EPR: how to encourage the user. Int J Med Inform 2001; 64: 173–185.[CrossRef][Web of Science][Medline]

  27. Grol R, Dalhuijsen J, Thomas S et al. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. Br Med J 1998; 317: 858–861.[Abstract/Free Full Text]

  28. Rogers EM., Diffusion of innovations, 4th edition. New York: The Free Press, 1995.

  29. Freeman AC, Sweeney K. Why general practitioners do not implement evidence: qualitative study. Br Med J 2001; 323: 1100–1102.[Abstract/Free Full Text]

  30. Veldhuis M, Wigersma L, Okkes I. Deliberate departures from good general practice: a study of motives among Dutch general practitioners. Br J Gen Pract 1998; 48: 1833–1836.[Web of Science][Medline]

  31. van der Weijden T, van Bokhoven L, Dinant G-J et al. Understanding laboratory superfluous laboratory testing in diagnostic uncertainty: a qualitative study in general practice. Br J Gen Pract 2002; 52: 974–980.[Web of Science][Medline]

  32. Litzelman DK, Tierney WM. Physicians’ reasons for failing to comply with computerized preventive care guidelines. J Intern Med 1996; 11: 497–499.

  33. van Wijk MAM, van der Lei J, Mosseveld M et al. Compliance of general practitioners with a guideline-based decision support system for ordering blood tests. Clin Chem 2002; 48: 55–60.[Abstract/Free Full Text]

  34. Deyo RA., A key medical decision maker: the patient. Br Med J 2001; 323: 466–467.[Free Full Text]


Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us    What's this?



This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrow Search for citing articles in:
ISI Web of Science (2)
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Bindels, R.
Right arrow Articles by Winkens, R. A. G.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Bindels, R.
Right arrow Articles by Winkens, R. A. G.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?