Results Of 364 physicians attempting to register, 205 were randomized to the intervention (n= 102) or control arms (n= 103). Baseline and follow-up data were provided by 95 physicians (2127 patients). The proportion of patients with A1c ≤9% was similar at baseline and follow-up in both the control [adjusted odds ratio (AOR): 0.94; 95% confidence interval (CI): 0.61, 1.47] and intervention arms [AOR: 1.16 (95% CI: 0.80, 1.69)]; BP <140/90 mmHg and LDL <130 mg/dl were also similar at both measurement points (P= 0.66, P= 0.46; respectively). We observed no significant effect on diabetes control attributable to the intervention for any of the primary outcome measures. Intervention physicians engaged with the Website over a median of 64.7 weeks [interquartile range (IQR): 45.4–81.8) for a median total of 37 min (IQR: 16–66).
Conclusions A wide-reach, low-intensity, Web-based interactive multi-component intervention did not improve control of glucose, BP or lipids for patients with diabetes of physicians practicing in the rural Southeastern US.
rural health services
continuing process assessment (Health Care)
An estimated 23.6 million adults have diabetes in the USA . Although quality of care process measures such as assessing hemoglobin A1c, blood pressure (BP) and low-density lipoprotein cholesterol (LDL) are performed at high rates, the outcome measures of their appropriate control lag behind. For example, in one US study, >40% adults with diabetes had hemoglobin A1c ≥7%, 70% had BP ≥130/80 mmHg and 55% had total cholesterol ≥200 mg/dl . In the USA, diabetes is more prevalent in the South, but individuals with diabetes in Southern states have worse control [3, 4], especially rural residents [5, 6]. Hence, public health interventions aimed at improving diabetes control in patients living in Southern rural areas are warranted.
Because distance barriers are considerable in rural settings, it is reasonable to explore wide-reach, low-intensity interventions [7–9] to improving diabetes control. One approach that has shown some promise [10, 11] is Web-based continuing medical education (CME) programs. In fact, professional educational programs designed to improve the quality of health care and patient clinical outcomes are of increasing interest [12–15], but evidence on their effectiveness in improving patient outcomes is scant.
Therefore, our objective was to test a multi-component intervention including Web-based CME, performance feedback and quality improvement tools  targeted at primary care physicians designed to improve diabetes care—hemoglobin A1c, BP and LDL control—in the rural Southeastern USA.
Study design and setting
The Rural Diabetes Online Care (R-DOC) study was a cluster-randomized trial (Clinical Trials.gov identifier: NCT00403091). The protocol was approved by the local institutional review boards. Participants were family, general and internal medicine physicians located in rural areas of 11 Southeastern USA (Alabama, Arkansas, Florida, Georgia, Kentucky, Mississippi, Missouri, North Carolina, South Carolina, Tennessee and West Virginia). Details of the recruitment strategy are available elsewhere ; rural areas were identified by population size <25 000. Physicians practicing in rural areas were invited via letters/e-mails to log on to the Website.
Physicians were enrolled sequentially between 28 September 2006 and 13 September 2008 by visiting the study Website. During the first visit to the Website, physicians provided informed consent online and were randomized to the intervention or control arms using block randomization (the block size of four was concealed to the investigators and statistician). Only one physician per practice was enrolled. Participants were offered up to 1 h of American Medical Association Physicians Recognition Award (AMA PRA) Category 1 CME credit™ at no charge upon completion of each section of the Website (maximum 12 h).
Intervention arm description
Website development included a formative evaluation  and focused on helping physicians to achieve A1c, BP and LDL control in their diabetic patients. The Website included case-based learning, personalized audit and feedback, and tools designed to facilitate the provision of high-quality care. Specifically, the intervention site contained: (i) challenging cases; (ii) individualized diabetes performance feedback reports based on the physician's own practice; (iii) practice timesavers; (iv) practical goals and guidelines, including guidance for quality improvement and systems redesign ; (v) patient resources; (vi) and an area to track and view CME credit. Additional details of the Website components are available elsewhere .
Intervention arm physicians received e-mail reminders every 1–3 weeks about Website updates. E-mails were tailored for participants who had not completed specific sections of the Website. In keeping with the wide-reach, low-intensity CME framework, we designed the study to be purely Web-based with only electronic communication and without mail, phone calls or other retention activities that would limit generalizability and prevent scaling up.
Individualized performance feedback reports were based on a review of records of 15 of each physician's patients with diabetes (baseline data, see below). These reports were sequentially added 12–18 months after launching the Website. We provided three feedback reports, graphically comparing each intervention arm participant's personal performance compared with the performance for the top 10% of other intervention arm participants . The feedback reports consisted of: (i) % patients with A1c <7% in the past year, % with systolic BP <130 mmHg at the most recent visit and % with LDL <100 mg/dl in the past year; (ii) counseling on diet or exercise at any of the three most recent visits in the past year; and, (iii) medication added or increased for patients with A1c >8%, BP >140/90 mmHg if on less than four medications and LDL >130 mg/dl.
Control arm description
The control Website contained: (i) links to diabetes practice guidelines and patient education materials; (ii) a list of educational conferences on general medical topics (updated monthly); (iii) an area to track and view their CME credit; and (iv) a link to an external medical blog. Physicians in the control group did not receive performance feedback reports or electronic communications.
All participating physicians provided copies of records of 15 (intervention arm physicians) or 10 (control arm physicians) consecutively seen patients with diabetes at baseline and again at follow-up (representing two cross-sectional views of each physician's practice). The number of records was selected to balance rigor and cost (see also statistical section below). Patient inclusion criteria were having at least two office visits during the past year and no dialysis, dementia, organ transplantation, HIV/AIDS, terminal illness or malignancy (except skin and prostate cancer). Data abstraction was performed by trained personnel on blinded records sent to the study center (or abstracted on site). Data included patient demographics, insurance status, medical conditions, diabetes complications, medications, BP readings and laboratory values. Thus, data were provided on patients fulfilling the above inclusion criteria; we included patients' data for visits prior to randomization (baseline assessment) and prior to study closure (follow-up assessment). Quality control conducted on 5% of records revealed agreement of 90–95% between abstractors. Each physician received a $200 incentive for providing records for baseline data (staff received $50 for photocopying charts) and $500 for providing records for follow-up data.
The primary outcomes were measures of ‘acceptable’ and ‘optimal’ diabetes control [21, 22]. Acceptable control was the proportion of patients with A1c ≤9%, BP <140/90 mmHg and LDL <130 mg/dl. These thresholds were selected to be consistent with thresholds of poor control as defined by the National Committee for Quality Assurance; however, missing values were not counted in the denominator. Optimal control was A1c <7%, BP <130/80 mmHg and LDL <100 mg/dl, reflecting guideline recommendations [21, 22].
Secondary outcomes were the rate of assessment of A1c and LDL over the previous year and 2 years, respectively, and BP at the last visit; and A1c, systolic BP and LDL group mean levels.
For analytical purposes and based on clinically important differences, the study required 100 physicians per trial arm to detect a minimum of 0.4% difference in A1c, 6 mmHg in BP and 6 mg/dl in LDL for the main outcomes (power 80%, alpha = 0.05, up to 20% dropout) (10 patients per physician at baseline and at follow-up). However, intervention arm physicians were asked to provide 15 patient records at baseline because of the need to construct audit and feedback reports for A1c, BP and lipids, reflecting the fact that not all diabetes patients also have hypertension and hyperlipidemia [23, 24].
All analyses followed the intention-to-treat principle. Patient characteristics were compared using bivariate analysis at baseline and follow-up with the Chi-square test or the t-test as appropriate.
The relationship between the outcome variables and intervention effect was examined with generalized linear mixed models (GLMM), accounting for clustering of patients within physicians. Odds ratios for follow-up vs. baseline within the two groups were calculated, adjusting for covariates that differed between baseline and follow-up patient populations (i.e. age, race and clinical diagnosis of hypertension or depression). GLMM was implemented by PROC GLIMMIX in SAS. Thus, the group coefficient represented differences between the intervention and control groups at baseline; the time coefficient represented changes in the control group over time, thus capturing temporal trends. Finally, the group-by-time interaction coefficient represented the difference-of-differences for over-time change in the intervention versus control group, and was a direct comparison of over-time change for the intervention versus the control group. A positive odds ratio meant that the odds of a patient being in control increased more over time for the intervention versus control group. All analyses were performed in SAS version 9.2 (Cary, NC).
We also conducted per-protocol analyses by examining the effect of web engagement (above vs. below median number of weeks of exposure to the Website) on optimal A1c control by using generalized linear latent and mixed models (GLLAMM) in STATA 11.1 (College Station, TX).
Recruitment scheme, patient characteristics and web utilization
Figure 1 displays the recruitment scheme of the study. We obtained baseline and follow-up data on 95 physicians and 1182 of their diabetes patients at baseline and 945 diabetes patients at follow-up. Ninety (94.7%) physicians had access to the Internet in the office. The characteristics of the baseline and follow-up patients collectively are shown in Table 1. At baseline, intervention group physicians provided records for fewer African American patients and more patients with hypertension and depression compared with control group physicians. The median duration of control group physicians' visits to the Website was 5 min [interquartile range (IQR): 3–18] over a median of 0.14 weeks (IQR: 0.14–7.15); intervention group physicians' median duration of visits to the Website was 37 min (IQR: 16–66) over a median of 64.7 weeks (IQR: 45.4–81.8)].
Characteristics of diabetes patients of 95 physicians randomized to intervention (n= 48) or control (n= 47) who completed follow-up
Intervention (n= 715)
Control (n= 467)
Intervention (n= 479)
Control (n= 466)
Race, African American
Non-adherence to appointments
Peripheral vascular disease
Coronary artery disease
Values are mean (SD) or n (%).
aObesity: body mass index ≥30 or clinical diagnosis. bDiabetes complications: retinopathy, neuropathy or nephropathy. cVascular intervention: coronary artery bypass grafting, stent, percutaneous coronary angioplasty.
Consolidated standards of reporting trials (CONSORT) diagram.
Main outcomes—A1c, BP, LDL control
Acceptable A1c control (<9%) was similar at baseline and follow-up in both trial arms [control, adjusted odds ratio (AOR): 0.94; 95% confidence interval (CI): 0.61, 1.47; intervention, AOR: 1.16 (95% CI: 0.80, 1.69)], Figure 2 (top panel). Analogously, acceptable BP (<140/90 mmHg) and LDL control (<130 mg/dl) were also similar at baseline and follow-up in both trial arms [BP control arm, AOR: 1.19 (95% CI: 0.89, 1.58), P= 0.24; BP intervention arm, AOR: 1.06 (95% CI: 0.82, 1.38), P= 0.66] [LDL control arm, AOR: 1.08 (95% CI: 0.69, 1.69), P= 0.74; LDL intervention arm, AOR: 1.16 (95% CI: 0.78, 1.73), P= 0.46].
Main outcomes (top, middle panels). Proportions of patients with acceptable and optimal hemoglobin A1c, BP (BP, mmHg) and cholesterol (LDL, mg/dl) control for intervention (n= 48) or control (n= 47) physicians, at baseline (pre, n= 1182 patients) and follow-up (post, n= 945 patients). Secondary outcomes (bottom panel). Proportion assessed of hemoglobin A1c (past year), BP (last visit) and LDL (past 2 years). *P < 0.001, **P= 0.05.
Optimal A1c control (<7%) was also similar at baseline and follow-up in both trial arms [control, AOR: 0.82 (95% CI: 0.60, 1.13); intervention, AOR: 1.02 (95% CI: 0.77, 1.34)], Figure 2 (middle panel). Optimal BP (<130/80 mmHg) was also similar at baseline and follow-up in both arms [BP control, AOR: 1.20 (95% CI: 0.88, 1.63); BP intervention, AOR: 1.15 (95% CI: 0.88, 1.50)]. For optimal LDL control (<100 mg/dl), the control arm was modestly improved at follow-up (AOR: 1.35 (95% CI: 0.96, 1.90)] but the intervention arm at baseline and follow-up was similar [AOR: 0.95 (95% CI: 0.71, 1.28)].
We observed no significant effect attributable to the intervention as reflected in the group-by-time interaction term for either acceptable (A1c: P= 0.45, BP: P= 0.46 and LDL: P= 0.68) or for optimal control measures (A1c: P= 0.28, BP: P= 0.74 and LDL: P= 0.19). Because of the large variability in time spent on the Website, we also examined whether time engaged in the Website was associated with improved patient outcomes. Although not statistically significant, as web engagement increased in the intervention group, the proportion of patients with optimal A1c control increased (above vs. below median number of weeks, AOR: 1.32 (95% CI: 0.90, 1.94), P= 0.16).
Secondary outcomes—process measures
We observed notable increases in A1c assessment for both trial arms at follow-up relative to baseline (control, P < 0.001; intervention, P < 0.001), Figure 2 (bottom panel). BP assessment was high at both baseline and follow-up in both trial arms (control, P= 0.48; intervention, P= 0.05). Similar to A1c, LDL assessment increased for both trial arms at follow-up relative to baseline (control, P < 0.001; intervention, P < 0.001).
Secondary outcomes—group mean A1c, BP and LDL
The group mean A1c did not differ between baseline and follow-up for both trial arms (control, P= 0.19; intervention, P= 0.70), Figure 3. Similarly, mean systolic BP was similar at baseline and follow-up across trial arms (control, P= 0.40; intervention, P= 0.68). Finally, LDL also did not differ significantly at baseline and follow-up across trial arms, although there were non-significant trends toward improvement in both arms (control, P= 0.31; intervention, P= 0.20).
Secondary outcomes. Hemoglobin A1c, systolic and diastolic BP (mmHg) and cholesterol (LDL, mg/dl) for patients in the control and intervention groups at baseline (pre) and follow-up (post) (values are means and 95% confidence intervals).
In this cluster-randomized trial, a wide-reach, low-intensity Web-based multi-component intervention for primary care physicians in the rural Southern USA had no significant effect on their diabetes patients' glycemic, BP or lipid control. These findings have important implications for the recent enthusiasm in the CME community for education/quality improvement programs that aim to improve not only learning outcomes, but also higher level patient outcomes. While the type of intervention tested here could potentially be widely disseminated, the results suggest that in order to improve patient outcomes, perhaps a higher intensity approach, and concomitant cost, may be needed.
One plausible explanation for the lack of improvement in patient outcomes was the relatively low Web engagement by physician participants. This observation has important implications for Web-based CME programs, which by one estimate made up 7–9% of CME in 2008, but may make up as much as 50% by 2018 . As we previously reported, intervention physicians whose patients were in the worst quartile of A1c control—in greatest need for improvement—spent substantially less time on the Website than physicians whose patients were in the best quartile of A1c control (33 vs. 80 min, respectively, P= 0.003) . The low engagement occurred in spite of highly tailored retention activities and personalized feedback reports reflecting the physician's own practice. We purposely designed this intervention to be relatively low in intensity but wider in reach , maximizing generalizability and dissemination possibilities. However, our findings suggest that to impact patient outcomes, more intensive approaches may be required to better engage physicians, especially those physicians whose quality measures are suboptimal.
A second contributor to our findings could be the higher-than anticipated attrition rate (50 vs. 20% projected). High attrition is not unusual in Web-based interventions . The marked loss to follow-up undoubtedly was related to time constraints, since the Health Insurance Protection and Portability Act required us to request practice staff to blind records sent to us. Notably, practice closures were also higher than we anticipated, with a total of 20 practices closing over the 18 months between baseline and follow-up, reflecting economic conditions in the South. Attrition has important implications for performance improvement CME, which is even more demanding of physician time than this study. We conducted the record review and practice staff-copied records, but in performance improvement CME, physician themselves are often asked to conduct the review. While this could increase engagement, it could also result in substantial barriers to participation and retention, as we experienced in our study.
Our findings are consistent with some but not all physician Web-based education studies. For example, process but not outcome measures improved in a group-randomized controlled trial of a Web-based provider decision support tool among 26 urban academic medical center practitioners and 598 of their patients . Intervention providers could access patient data, treatment recommendations and Web-based resources. Annual foot examinations and A1c and LDL testing increased in the intervention group. However, measures of A1c, BP or LDL control did not improve in either study arm. In our study, physicians in both trial arms demonstrated both statistically significant and clinically important improvements in the assessment of hemoglobin A1c and LDL cholesterol. These findings suggest that low-intensity, wide-reach CME programs may be more effective at improving processes but not outcomes of care; however, the lack of significant differences between the intervention and control groups suggests a Hawthorne's effect (behavior change simply due to the study itself, rather than the specific intervention).
More recently, in a pre–post study of 18 rural physicians, a Web-based educational intervention was associated with improved patient outcomes  in three cross-sectional samples of patients. The Web-based intervention was composed of 13 educational chapters, practice tools, patient education resources and self-assessment. Glucose control (A1c <7%) increased by 5% at 24 months; BP control (≤140/90 mmHg) increased by 11%; and LDL control (<100 mg/dl) increased by 5%. In contrast to our CME program, this quality improvement intervention was far more intensive, and the lack of a control group makes the reported results difficult to interpret.
Our study has several important implications. First, the quest for effective wide-reach and low-intensity interventions, applicable to rural settings, is important. The needs to update practicing physicians on the rapidly expanding body of knowledge as well as to optimize the quality of care they provide are very real, and finding effective strategies to accomplish continuing professional development for physicians practicing in remote locations is an important objective. Web-based CME activities are widely accessible and are therefore attractive options. However, our study suggests that more work is needed to identify strategies that can achieve better physician engagement . We hoped to have physicians apply quality improvement principles in their practices; however, they were particularly disengaged in this area. If better engagement can be achieved, tests of whether Web-based programs can actually improve patient outcomes are needed. Notable is the paucity of rigorous evaluation of CME programs per se. Only studies with control groups and rigorous design can provide information on the effectiveness of such programs, and our study is an example of how such programs can be tested.
Our study had several notable limitations. The non-random sampling of physicians may have introduced selection bias. Second, physicians were asked to provide records of consecutively seen patients, and while we were unable to monitor compliance with this request, the wide range of A1c, BP and LDL values suggests that not only well-controlled patients were selected, but poorly-controlled patients as well. The fiscal realities of our project prevented a more comprehensive assessment of each physician's practice, and it is possible that 10 or 15 records did not represent the physician's diabetes patients in general. The high attrition may have introduced biases. Although it was designed with wide reach and scaling up in mind, our budget for Website development and implementation likely exceeded that available in most academic CME divisions, raising concerns about sponsorship of such programs.
In conclusion, a rigorous test of a state-of-the-art multi-component Web-based intervention designed to improve the diabetes care of rural primary care physicians showed no improvement in their patients' glycemic, BP and lipid control. Despite low engagement in the Web-based program, process of care measures did improve in both arms of the study. Our study raises important issues for wide-reach, low-intensity CME programs that seek to improve patient outcomes.
Awards by the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) 5R18DK065001 to Dr J.J.A. Drs A.H.S and C.A.E. were supported by the Veterans Affairs National Quality Scholars Program.
Sharina Person, PhD, for prior statistical analysis assistance.
Published by Oxford University Press in association with the International Society for Quality in Health Care 2011.
Centers for Disease Control and Prevention. National Diabetes Fact Sheet: General Information and National Estimates on Diabetes in the United States, 2007. Atlanta, GA: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention; 2008.
. Trends in medication use among US adults with diabetes mellitus: glycemic control at the expense of controlling cardiovascular risk factors. Arch Intern Med 2009;169:1718-20. doi:10.1001/archinternmed.2009.296.
. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res 2006;21:688-94. doi:10.1093/her/cyl081.
. Are there differences in engagement in a web-based intervention among physicians of patients with diabetes: the Rural Diabetes Online Care (RDOC) Study. Acad Med 2010;85:1511-7. doi:10.1097/ACM.0b013e3181eac036.
Carlos A.Estrada, Monika M.Safford, Amanda H.Salanitro, Thomas K.Houston, WilliamCurry, Jessica H.Williams, FernandoOvalle, YonginKim, PamelaFoster, Jeroan J.AllisonInt J Qual Health Care(2011)23 (6):
682-689DOI: http://dx.doi.org/10.1093/intqhc/mzr053First published online: 10 August 2011 (8 pages)