A cluster-randomized trial of the significance of a reminder procedure in a patient evaluation survey in general practice
1 The Research Unit and Department for General Practice, and 2 The Research Unit for General Practice, University of Aarhus, Aarhus C, Denmark
Objectives. To determine whether adding a reminder procedure to the personal handing out of questionnaires to patients by general practitioners (GPs) in a patient evaluation survey added further information to the study and whether this influenced the results of the evaluations fed back to the GPs.
Design. Patient evaluation survey in general practice.
Study participants. 6822 patients consulting 60 GPs in a Danish county. GPs were voluntarily participating in a national patient evaluation project.
Intervention. We used the EUROPEP instrument for patient evaluation in general practice. It contains 23 items in five dimensions and two additional questions on general satisfaction. GPs were randomized into two groups with and without a reminder procedure.
Main outcome measure. Scores in the six assessment dimensions and patient characteristics were compared for the primary and the reminder respondents and between the two randomization groups. In the analyses, we adjusted for the clustering of patients.
Results. We found that the use of a reminder procedure increased the response rate. Respondents to a reminder were younger than the primary responding patients and were more critical in their GP assessment. Patient evaluations of the individual GPs were statistically significantly more critical if these reminder responses were included. Absolute differences were too small to have any practical implications.
Conclusions. Adding a reminder procedure to the face-to-face handing out of questionnaires to patients by the GPs increased the response rate significantly but produced no clinically significant differences in the assessment of the GPs.
Address reprint requests to Hanne N. Heje, The Research Unit and Department for General Practice, University of Aarhus, Aarhus C, Denmark. E-mail: hh{at}alm.au.dk
Accepted for publication March 6, 2006.
The response rate is often a key issue when assessing the quality of survey results [1–5]. A high response rate is believed to be equivalent to the absence of major selection bias due to non-response and particularly important in surveys where lack of information about the study population makes it impossible to estimate the consequences of such non-response.
Previous studies have shown that appropriate survey design can produce high response rates [2,6–8]. Factors influencing rates vary and depend on the object studied, the population in focus, and the study design [9]. But studies are also seen where the very efforts to maximize response rates produce selection bias [10].
In surveys, non-responders often receive reminders to raise the response rate. This is often costly, and unless such strategies produce enough new data to affect the conclusion of the survey, little may be gained by this extra effort [11]. It would therefore seem appropriate to restrict the use of reminders to studies where they are likely to significantly affect the outcome.
The aim of this study was to determine whether adding a reminder procedure to the personal handing out of questionnaires to the patients by the doctors in a patient evaluation survey in general practice changed the results of the evaluations.
 |
Materials and methods |
|---|
 Top Materials and methods ResultsDiscussionAcknowledgementsReferences |
|---|
Study population
In 2003, all 232 general practitioners (GPs) in the county of Vejle, Denmark, were invited to carry out patient evaluations of their practices. Those who signed in were randomly drawn to form two groups of equal size: one group with and the other group without a reminder procedure. The level of randomization was the individual doctor.
The participants handed out questionnaires to successive patients who attended the doctor in the surgery or were visited at home. The patients were at least 18 years of age, were listed in the practice, and were able to understand Danish. They were informed that their replies were anonymous to the doctor. Patients included by the doctors in the group with the reminder procedure were told that they would receive a reminder. Those in the group without were not told that they would not receive a reminder.
Each questionnaire was identified by a serial number, connecting it with the doctor who handed it out. Doctors in the reminder group handed out 100 questionnaires, whereas those in the group without a reminder procedure handed out 130. Questionnaires not handed out within 2 weeks could be returned to the secretariat. In other counties, we obtained response rates exceeding 80% when using reminders. With an expected response rate without reminders of around 60%, the doctors in the group without reminders would have to hand out 130 questionnaires each. In this way, the succeeding feedback of individual results would be based on approximately the same amount of data irrespective of the randomization group. This consideration for the validity of the feedback also implied that the statistical power of the study was by far secured.
The questionnaire
The questionnaire contained the 23 items forming the EUROPEP instrument [12,13]. These questions covered aspects of general practice and fell into the following five dimensions: doctor–patient relationship, medical care, information and support, organization of care, and accessibility. Answers were marked on a five-point Likert scale ranging from ‘poor’ to ‘excellent’. Alternatively, the patients could choose the category ‘not able to answer/not relevant’. An additional sixth dimension contained two questions on general satisfaction. Finally, there were questions about the patient’s gender, age, educational level, frequency of attendance to a general practice, time listed with the practice, self-rated health, and chronic conditions.
The patients were asked to assess the doctor they considered to be their personal physician based on their contact experience throughout the past 12 months. They were also asked to write the doctor’s name on the questionnaire to avoid confusion about assessments and to enable individual assessment of doctors in partnership practices. The questionnaires were returned by the patients in prepaid envelopes to the project secretariat. The doctors reported the dates of the handing out period to the secretariat. Those in the group with a reminder procedure also sent in labels containing names, addresses, and serial numbers from the questionnaires handed out to these specific patients. After 3 weeks, a reminder and a new questionnaire with the same serial number were sent to non-responding patients.
Analyses
Comparisons.
We compared the responses, respondents, and the response rates in the two randomization groups and the group of primary responses in the group with a reminder procedure to examine the effects of using reminders. We also compared the primary responses and respondents with the reminder responses and respondents in the group with a reminder procedure. A response was categorized as a reminder response if received after the reminders were mailed.
Response rates.
We excluded the following questionnaires from the analysis: questionnaires not handed out but returned by the doctors unused, questionnaires returned by patients younger than 18 years, and reminder questionnaires returned by the mail service because of insufficient addressing. Questionnaires assessing non-participating doctors and questionnaires where it was unclear which doctor was assessed were included in the analysis of the response rates but were withdrawn from the analysis of the assessment data.
Assessments.
The mean assessment score for each response was calculated and expressed as a percentage of the maximum possible score for each of the six assessment dimensions. Answers in the category ‘not able to answer/not relevant’ were excluded. We used linear regression to allow adjustment for patient clustering [14,15] when comparing the mean assessment scores of the different groups. We also calculated the assessments per doctor in the reminder group with and without the reminder responses.
Respondents.
For the responding patients, we calculated the percentage of male patients, patients with a high educational level, patients with a good self-rated health, and patients who reported suffering from a chronic condition. We calculated means for the following variables: age, frequency of attendance, and time listed with the practice. Again, we adjusted for patient clustering when comparing the percentages and means of the patient characteristics between the different groups using logistic and linear regression, respectively [14,15]. Statistically significant differences in characteristics between different patient groups were adjusted for in the analyses of differences in responses between the same groups [16].
We calculated two-sided P-values. Because of repeated testing (multiple comparisons [17,18]), we chose a level for statistical significance at or below 1%. We used STATA 8 for data processing [15].
|
Results |
|---|
|
TopMaterials and methodsResultsDiscussionAcknowledgementsReferences |
|---|
A total of 60 doctors entered the study and were randomized into two groups of 30. Three doctors returned 9, 30 and 39 questionnaires unused. The doctors in the reminder group spent 5–26 (mean = 11.3) weekdays handing out their 100 questionnaires. Those in the group without the reminder procedure spent 7–46 (mean = 14.3) weekdays handing out their 130 questionnaires. After adjusting for the number of questionnaires handed out by the individual doctors, we found no statistically significant difference (t-test, P = 0.755) in the length of the periods in the two groups. In the group with a reminder procedure, 60% of the doctors were males, the mean age was 48.5 years, and mean seniority 12.5 years compared with 67%, 51.2 years, and 14.9 years in the group without. Analysis showed no statistically significant differences.
Response rates. A total of 6822 questionnaires were handed out (Table 1), of which 4683 were returned. We mailed 872 reminders, of which 47 were returned by the mail service because of insufficient addressing. The reminders provided us with another 361 responses, of which 342 were valid. This produced significantly different response rates (65.7% in the group without a reminder procedure; primarily 71.8% and finally 83.4% in the group with a reminder procedure).
|
Assessments. We compared the scores of the primary responses and the reminder responses (Table 2) and found that the latter were statistically significantly more critical in all dimensions. The reminder responses had no impact on the final data, as we found no statistically significant differences between the responses in the randomization groups irrespective of whether responses were excluded [primary responses in the group with reminders compared with the responses in the group without (not shown in the table)] or included (all responses in the group with reminders compared with the responses in the group without).
|
Figure 1 shows that including the reminder responses did not change the assessment of the individual doctors significantly (only assessments regarding the doctor–patient relationship are shown).
|
Respondents. The patients who had responded to a reminder were statistically significantly younger than the primary responding patients but did not differ in the other characteristics ( Table 2). The patients in the two randomization groups did not differ in any of the characteristics either excluding the patients who had responded to a reminder or including them.
Adjusting for differences in respondent characteristics. When adjusting for the difference in age, the differences in assessment between the primary and the reminder responses remained statistically significant.
|
Discussion |
|---|
|
TopMaterials and methodsResultsDiscussionAcknowledgementsReferences |
|---|
Our results showed that two different ways of carrying out a hand-out patient evaluation survey in general practice produced no statistically significant differences in assessment. As expected, we found that the use of a reminder procedure yielded a higher response rate, but that the reminder responses did not significantly change the assessment per dimension. We also found that the patients responding to a reminder were younger than the primary responding patients and were more critical in their assessment.
The doctors handed out questionnaires to successive patients, and frequently attending patients were therefore relatively over-represented. Care should therefore be taken when comparing with the results from studies with a different distribution method. The method for distribution also implied a risk of selection bias if the doctors forgot to hand out questionnaires or more systematically excluded certain patients. We do not know the extent of this problem that would have been eliminated in a postal survey.
The primary response rates diverged statistically significantly between the two randomization groups. The patients included by the doctors in the reminder group knew that they would receive a reminder if they did not respond and the doctors spent some time writing reminder labels. The doctors may therefore have been more successful in motivating the patients, hence producing a higher response rate [7]. We have no reason to believe that there was a differentiated selection bias in the two randomization groups, as we found no difference between the lengths of the periods of randomization.
The reminders produced a final response rate in the group with a reminder procedure of 83.4%, but the study design left us with no opportunity to analyse the non-responding part of the study population, and thus we do not know whether the study was actually biased and in which direction. The number of responses assessing each doctor in the reminder group ranged from 40 to 104. Mazor et al. [19] have shown an association between the number of responses and the level of assessment, suggesting that dissatisfaction may lead to non-response. We found no such association as a cause of bias in our study (Spearman’s rho = 0.044).
The randomization of the participating doctors should theoretically produce two groups of patients with an equal distribution of characteristics. As summarized in Table 2, patients with different characteristics are equally distributed in the two randomization groups. So, even though the two procedures for including patients yielded different response rates, they produced two respondent populations that were comparable on eight sociodemographic characteristics and agreed in their assessment of the doctors.
For all dimensions, the reminder responses were statistically significantly more critical than the primary responses, and the reminder respondents were younger than the primary respondents. Previous studies [13,20–23] have found that younger patients are more critical in the assessment of their doctor. We therefore adjusted for patient age that did not eliminate the difference. So, the difference in respondent age did not fully account for the differences in the two groups’ assessments, which is in agreement with earlier findings in a hospital setting [24].
With a primary response rate of 72%, the direct expenses because of the reminder procedure accounted for 26% of the overall costs of the patient evaluations. The value added to the patient evaluations by administering reminders seemed to be minimal, whereas the expenses rose significantly. The study design left us with no opportunity to demonstrate the significance of working with a well-described study population and thereby show whether the deviation in reminder respondents’ age is an expression of the reminders skewing the respondent population compared with the study population (bias) or whether the reminders resulted in the population of respondents being representative of the study population.
The bias possibly introduced into this study by the reminders is negligible, and judging from the results of the study, it is of less significance than other possible sources of bias. Future research should aim at evaluating different strategies for including patients, and analyses of non-responders’ characteristics and potential responses should be made on the basis of a well-described study population.
Implications. Reminding non-responding patients resulted in responses from younger patients and patients who were more critical of their doctors. The reminders tended to make the assessments less favourable, but the differences were not statistically significant. Adding a reminder procedure to the face-to-face handing out of questionnaires to patients by the GPs increases the response rate significantly but does not change the results of the patient evaluations.
|
Acknowledgements |
|---|
|
TopMaterials and methodsResultsDiscussionAcknowledgementsReferences |
|---|
This study was carried out as a part of the national project on patient evaluations, DanPEP. The DanPEP study was supported by grants from the Central Committee on Quality Development and Informatics in General Practice and the Danish Ministry of the Interior and Health. Direct expenses incurred by the participating doctors in this study were refunded by the local Committee for Quality Improvement in General Practice in the County of Vejle.
|
References |
|---|
|
TopMaterials and methodsResultsDiscussionAcknowledgementsReferences |
|---|
- Barkley WM, Furse DH. Changing priorities for improvement: the impact of low response rates in patient satisfaction. Jt Comm J Qual Improv 1996; 22: 427–433.[Medline]
- Asch DA, Jedrziewski MK, Christakis NA. Response rates to mail surveys published in medical journals. J Clin Epidemiol 1997; 50: 1129–1136.[CrossRef][Web of Science][Medline]
- Sitzia J, Wood N. Response rate in patient satisfaction research: an analysis of 210 published studies. Int J Qual Health Care 1998; 10: 311–317.
[Abstract/Free Full Text] - Gasquet I, Falissard B, Ravaud P. Impact of reminders and method of questionnaire distribution on patient response to mail-back satisfaction survey. J Clin Epidemiol 2001; 54: 1174–1180.[CrossRef][Web of Science][Medline]
- Smeeth L, Fletcher AE. Improving the response rates to questionnaires. Br Med J 2002; 324: 1168–1169.
[Free Full Text] - Fox RJ, Crask MR, Kim J. Mail survey response rate. A meta-analysis of selected techniques for inducing response. Public Opinion Q 1988; 52: 467–491.
[Abstract/Free Full Text] - Vedsted P, Mainz J, Olesen F. The significance of a reminder and physician’s motivation for the response rate of a questionnaire survey in general practice (in Danish). Ugeskr Laeger 2001; 163: 5529–5532.[Medline]
- Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R. Increasing response rates to postal questionnaires: systematic review. Br Med J 2002; 324: 1183–1191.
[Abstract/Free Full Text] - McColl E, Jacoby A, Thomas L, Soutter J, Bamford C, Steen N. Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technol Assess 2001; 5: 1–256.[Medline]
- der Wiel AB, van Exel E, de Craen AJ, Gussekloo J, Lagaay AM, Knook DL. A high response is not essential to prevent selection bias: results from the Leiden 85-plus study. J Clin Epidemiol 2002; 55: 1119–1125.[CrossRef][Web of Science][Medline]
- Wensing M, Mainz J, Kvamme OJ, Jung HP, Ribacke M. Effect of mailed reminders on the response rate in surveys among patients in general practice. J Clin Epidemiol 1999; 52: 585–587.[CrossRef][Medline]
- Grol R, Wensing M. Patients evaluate general/family practice. The EUROPEP instrument. Nijmegen: EQuiP, WONCA Region Europe, 2000.
- Grol R, Wensing M, Mainz J, Jung HP, Ferreira P, Hearnshaw H. Patients in Europe evaluate general practice care: an international comparison. Br J Gen Pract 2000; 50: 882–887.[Web of Science][Medline]
- Donner A, Klar N. Design and Analysis of Cluster Randomisation Trials in Health Research. London, UK: Hodder Arnold, 2000.
- Stata Statistical Software: Release 8.0. College Station, TX: Stata Corporation, 2003.
- Dupont WD. Statistical Modeling for Biomedical Researchers. A Simple Introduction to the Analysis of Complex Data. Cambridge: Cambridge University Press, 2003.
- Armitage P, Berry G. Statistical Methods in Medical Research, third edition. Oxford: Blackwell Scientific Publications, 1994.
- Rothman KJ, Greenland S. Modern Epidemiology. Philadelphia: Lippincott-Raven Publishers, 1998.
- Mazor KM, Clauser BE, Field T, Yood RA, Gurwitz JH. A demonstration of the impact of response bias on the results of patient satisfaction surveys. Health Serv Res 2002; 37: 1403–1417.[CrossRef][Web of Science][Medline]
- Lewis JR. Patient views on quality care in general practice: literature review. Soc Sci Med 1994; 39: 655–670.[CrossRef][Web of Science][Medline]
- Baker R, Streatfield J. What type of general practice do patients prefer? Exploration of practice characteristics influencing patient satisfaction. Br J Gen Pract 1995; 45: 654–659.[Web of Science][Medline]
- Mainz J, Vedsted P, Olesen F. How do patients evaluate their general practitioners? Danish results from a European study (in Danish). Ugeskr Laeger 2000; 162: 654–658.[Medline]
- Campbell JL, Ramsay J, Green J. Age, gender, socioeconomic, and ethnic differences in patients’ assessments of primary health care. Qual Health Care 2001; 10: 90–95.
[Abstract/Free Full Text] - Perneger TV, Chamot E, Bovier PA. Nonresponse bias in a survey of patient perceptions of hospital care. Med Care 2005; 43: 374–380.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  T. Perneger Ten reasons to conduct a randomized study in quality improvement Int. J. Qual. Health Care, December 1, 2006; 18(6): 395 - 396. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||