International Journal for Quality in Health Care Advance Access originally published online on May 20, 2008
International Journal for Quality in Health Care 2008 20(4):264-270; doi:10.1093/intqhc/mzn021
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Developing an automated speech-recognition telephone diabetes intervention
1 Center for Primary Care and Prevention, The Warren Alpert Medical School of Brown University/Memorial Hospital of Rhode Island, Pawtucket, RI, USA
2 Department of Ambulatory Care and Prevention, Harvard Medical School and Harvard Pilgrim Health Care, Boston, MA, USA
3 Department of Veterans Affairs Center for Practice Management and Outcomes Research, University of Michigan, Ann Arbor, MI, USA
4 Department of Family Medicine, The Warren Alpert Medical School of Brown University, Providence, RI, USA
Address reprint requests to: Steven Simon; Tel: + 1-617-509-9938; Fax: + 1-617-859-8112; E-mail: steven_simon{at}hphc.org
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Objective. Many patients do not receive guideline-recommended care for diabetes and other chronic conditions. Automated speech-recognition telephone outreach to supplement in-person physician–patient communication may enhance patient care for chronic illness. We conducted this study to inform the development of an automated telephone outreach intervention for improving diabetes care among members of a large, not-for-profit health plan.
Design. In-depth telephone interviews with qualitative analysis.
Setting/participants. Individuals with diabetes (n = 36) enrolled in a large regional health plan in the USA.
Main outcome measure. Patients' opinions about automated speech-recognition telephone technology.
Results. Patients who were recently diagnosed with diabetes and some with diabetes for a decade or more expressed basic informational needs. While most would prefer to speak with a live person rather than a computer-recorded voice, many felt that the automated system could successfully supplement the information they receive from their physicians and could serve as an integral part of their care. Patients suggested that such a system could provide specific dietary advice, information about diabetes and its self-care, a call-in menu of information topics, reminders about laboratory test results and appointments, tracking of personal laboratory results and feedback about their self-monitoring.
Conclusions. While some patients expressed negative attitudes toward automated speech recognition telephone systems generally, most felt that a variety of functions of such a system could be beneficial to their diabetes care. In-depth interviews resulted in substantive input from health plan members for the design of an automated telephone outreach system to supplement in-person physician–patient communication in this population.
Keywords: information technology, intervention development, qualitative methods, quality of care
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Randomized controlled trials have identified effective strategies for treating chronic diseases [1–3], but important gaps remain in the translation of research evidence into practice. Although guidelines exist for managing conditions such as asthma, hypertension and diabetes mellitus, many patients do not get the care they should receive [4]. Diabetes serves as a prototypical example of a condition for which there exist critical shortcomings in the translation of research into practice [5–7]. Suboptimal management of diabetes leads to serious adverse outcomes, and there is growing appreciation of the benefits of aggressive diabetes control [1, 8, 9]. These factors make diabetes an important condition to target for quality improvement interventions that promote self-management skills and adherence to physicians' recommendations.
Management of diabetes and other chronic conditions can benefit in multiple ways from interventions using computer-automated telephone technology [10–14]. Automated telephone outreach has most often been linked to follow-up with a live practitioner or counselor [15–18]. As speech recognition and interactive capabilities rapidly improve, possibilities emerge for interventions that are highly dependent on patient–computer interaction [13, 19, 20]. To be successful, computer-automated telephone outreach interventions should be tailored to the specific needs and characteristics of the target population, including their tolerance for interacting with speech recognition software [21].
We conducted in-depth interviews among patients with diabetes in a large health plan to inform the development of an automated telephone outreach intervention, which has recently been implemented. We undertook this qualitative study to identify potential approaches for engaging health plan members with diabetes in an automated telephone outreach program and to capture the range of content that these individuals might embrace. This report describes how the findings were incorporated in the population-based intervention.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Study design, setting and population
This qualitative study consisted of 36 individual, in-depth telephone interviews with patients from Harvard Pilgrim Health Care, a large not-for-profit health plan in New England. Among 22 000 adult health plan members who met diagnostic criteria for diabetes [22], we sampled 300 members stratified in approximately equal numbers across strata of quality measurement (high vs. low) and household income (<$50 000 vs. $50 000 + ). We excluded individuals who had previously informed the health plan that they did not want to be contacted for research studies and individuals whose primary care physicians informed us that they did not have diabetes. We attempted to contact 161 members and completed 36 interviews. Our recruitment was purposive—e.g., we intensified recruitment of Spanish-speaking and low-income subjects when early recruitment did not yield adequate representation of these characteristics.
We identified two subsets of individuals with active health plan enrollment: (i) patients in the ‘low-quality’ HEDIS® measures group were those without a dilated eye examination in the preceding 12 months and with no more than one of the following recommended laboratory tests performed in the prior 12 months: LDL-cholesterol; HbA1c; urine microalbumin; and (ii) patients in the ‘high-quality’ group had an eye examination in the preceding year and at least two of the three other designated tests performed. Interviews were conducted by a bilingual (English/Spanish) research assistant trained in open-ended interviewing. Interviews lasted for 15–30 min, and participants were given a $25 honorarium. Interviews were recorded, and audiotapes were professionally transcribed. The study protocol was approved by Harvard Pilgrim Health Care's Human Studies Committee.
Interview questions
We developed an interview guide with open-ended questions to explore attitudes toward diabetes, its management and health plan interventions to support members with diabetes (Box 1). The interviewer followed core questions with appropriate, spontaneous probes [23]. During the interview, the interviewer played a 90-second segment from an actual de-identified automated telephone outreach call and solicited participants' feedback.
| Box 1. Content areas from in-depth interviews with health plan members Content area Personal history with diabetes Experience with health plan programs to improve diabetes care and self-management Needs and preferences for information about diabetes Experience with automated telephone outreach and speech recognition in health care and other industries Speculation on use of automated telephone outreach programs for diabetes Reactions to recorded sample of automated telephone outreach with speech recognition Suggestions for improving and implementing the automated telephone outreach intervention Barriers to and facilitators of obtaining recommended care for diabetes Factors influencing adherence to physicians' recommendations and prescriptions
|
Data analysis
Using the analytic method of immersion/crystallization [24], a general internist/health services researcher (S.R.S.) and anthropologist (R.E.G.) listened to selected interview tapes, read all interview transcripts and met regularly to discuss 4–6 new transcripts per analysis session. We became thoroughly ‘immersed’ in the data within the context of each interview in its entirety. We repeatedly looked at alternative explanations for interpretations of the data, and ruled them out or modified our initial interpretations.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Study population
Characteristics of the 36 participants are shown in Table 1. Overall, the study sample included more women than men. For most characteristics, there were small differences in the numbers of participants meeting the high and low quality of care criteria.
|
Overview of results
Patients who were recently diagnosed with diabetes as well as those who had been diagnosed more than 5 years previously expressed basic informational needs. While most would prefer to speak with a live person rather than a computer-recorded voice, many felt that the automated speech recognition system could be used to supplement the information they receive from their physicians and could serve as an integral part of their care.
Patients' knowledge and desire for more information
Both those who were new to the disease and many who had had diabetes for at least 5 years and as long as 20 years had informational needs that might be addressed through an automated speech recognition telephone system. Patients' knowledge was frequently uneven; many were unaware of the gaps in their understanding of diabetes and the need for regular preventive examinations. For example, a 68-year-old man with 9 years of diabetes claimed that he had no need for further information as he reads avidly and has his diabetes ‘under control’, although records show that he has low HEDIS® measures. A 50-year-old man with low HEDIS® measures who was diagnosed 7 years previously apparently misunderstood the role of yearly eye examinations, and commented how the automated telephone system could be helpful:
Well, it could be used for people who are just starting to have diabetes—they could be asking questions—like blurry vision .... I thought for sure it was the diabetes. It might have nothing to do with my diabetes. These questions could be asked of the computer.The largest category of information that patients thought could be provided by the computer—how to control diabetes through diet—was addressed in equal numbers among those with low- and high-quality HEDIS® measures, and was mentioned more often by patients with greater than 5 years of diabetes than by those diagnosed more recently. Many wanted an individual nutrition plan with specific, detailed information about what to eat and how to prepare foods.
Several patients with high HEDIS® measures, particularly those who had been diagnosed in the past 5 years, were interested in their long-term prognosis: what they will experience in the future, and how soon they will die from this disease. Information requested by a few people in both HEDIS® categories included: how to perform home glucose monitoring properly, the impact of diabetes on other health conditions, and when a cure for diabetes will be found.
Attitudes toward interactive, automated telephone technology for diabetes management
About half of the patients interviewed anticipated that they would react favorably toward automated telephone outreach technology, and all of these remained positive after hearing the taped sample, expressing surprise at the realistic automated voice. Among these patients, most preferred to speak to a human being but were able to identify reasons why the automated system could be helpful.
As 1-year-old woman with high HEDIS® measures who had had diabetes for 12 years declared, ‘Health is personal!’ However, she said she would be willing to have the computer supplement her discussions with her doctor: ‘Gosh, probably would be nice to receive a call saying, ‘I'm just calling to make sure that you're staying on track’. Additionally, some patients who were doubtful about the usefulness of this system admitted that they would participate if their doctors encouraged them, and especially if they could receive cutting-edge, ‘late-breaking’ information to supplement what they learned from their doctor.
Of the 16 participants who initially felt negatively about the idea of receiving an automated telephone call (largely due to unpleasant experiences with other kinds of automated telephone systems or feeling that the computer is too impersonal), half remained negative after hearing the sample recording. However, all of these negative participants (except two who had had diabetes for more than 20 years) offered ways that an automated system might be helpful. For example, a 40-year-old woman diagnosed 4 years previously and with low HEDIS® measures declared: ‘I think it would be stupid. If you're really that concerned about me, you'll call me personally’. Later, she commented that the system would be useful if it would provide a mechanism for her to receive feedback on her home blood glucose readings.
Patients who revised their perspective positively about automated telephone outreach after hearing the example call felt that automated calls would be worthwhile if they contributed to improving one's health, and that they would be useful for supplementing, not replacing, in-person doctor visits. For example, one 63-year-old woman was initially very negative toward computerized telephone systems: ‘I just feel those [systems] can be very intimidating to people. ... You don't warm up to a machine. You just don't. You just don't’. However, her perspective changed after hearing the sample recording: ‘That's not bad. That's not bad at all. I liked it. I think it's great that people are reaching out to people with diabetes’.
Some patients who retained their original negative opinions of the automated calls nevertheless spoke more favorably toward the end of the interview about the potential of the system as they thought of ways the system could be helpful. For example, a 22-year-old man who had type-1 diabetes since childhood stated early in the interview:
I absolutely hate [automated systems]. I don't mean to [show disrespect to] any programs that you guys are developing, but I think they're stupid. These automated systems—they're robots—people hate them. I mean, don't you? You're not talking to a real person.Yet, later in the interview he remarked that reminders to get examinations would be helpful, and he would listen to the automated updates if his doctor said it was part of his treatment program.
Perceptions of how automated telephone technology can help improve diabetes care
Regarding the potential uses of automated telephone technology, there was little difference among the responses of patients in the low- and high-quality HEDIS® measure groups and minor differences between patients who had been diagnosed within 5 years and greater than 5 years previously. Patients expressed interest in individualizing the telephone outreach calls to address their specific conditions. People in all groups were interested in having the system answer their questions. A service that appeared to sway people toward more positive opinion about the system and was most commonly suggested by newly diagnosed patients was the possibility of dialing in at their own convenience to hear information from a menu of topics. A function suggested by an equal percentage of newly and longer diagnosed patients was to allow people to leave a recorded message with questions that they would like to have answered in between visits with their doctor. A 77-year-old woman with low HEDIS® measures and 7 years with diabetes liked the idea of calling the computer with questions because she would not ‘call the doctor as much and pester him’. And a 60-year-old nurse with 13 years of diabetes and high HEDIS® measures explained:
I think you'd learn more. You could ask questions that you might forget to ask your doctor, or maybe things that are stupid questions, and it would be talking to a computer rather than a person, so you wouldn't feel [embarrassed].The function most commonly mentioned by patients in both low and high HEDIS® groups and slightly more among recently diagnosed people than for those with longer diabetes experience was for the computer to call patients with reminders about upcoming medical appointments or the need to have laboratory testing, particularly if these calls were tailored to the person's specific care. They would like to have the computer provide personalized feedback on their laboratory results. People in all groups, with a higher percentage of those who were diagnosed more than 5 years previously, suggested that the computer could help them monitor and log their daily blood sugar levels and provide both computerized and human feedback about their results. A 55-year-old man with high HEDIS® measures who had had diabetes for 4 years explained:
I'd like to know exactly where my levels are in terms of whether they're considered good, bad, or whether they need improvement.Patients suggested having the computer make calls to motivate them and track their progress regarding diet and exercise. Other suggested functions included the provision of advice on diet and weight loss. Though some felt that a recorded or computerized call would not be as motivating as talking to a doctor, others felt that they did not get enough specific dietary advice from doctors. As a 57-year-old woman with high HEDIS® measures and diabetes for 12 years explained:
Doctors don't help you. They just say, ‘Well if you don't start watching what you eat and do more exercise you're going to have to go on insulin,’ instead of just telling me [exactly how to avoid insulin]. It's like a threat all the time.Patients also mentioned that the calls could explain procedures such as the dilated eye examination to reduce people's fear, explain the role of screening tests and inform about health insurance coverage of the costs associated with tests and specialty visits.
Patients' practical suggestions for an automated telephone diabetes outreach system
An overarching theme among patients' comments was the need to ensure that the automated system is integral to their care plans. Some would be willing to participate if it was clear from the first contact that the call was from their health plan and not from a market researcher. Many noted the proliferation of automated telemarketing calls, and emphasized how important it would be for the program to distinguish itself from this type of call by announcing the program first through mailed materials from the health plan, mentioning the health plan in the first sentence of the telephone call, and stating that the call is not a research survey. Patients described identifying closely with and trusting their physicians, and frequently suggested having physicians' endorsements to encourage patients to participate in the program.
Built-in flexibility for timing of calls was recommended, with preference for a system that could leave a message for patients to call back and participate at their convenience. To keep calls short, patients suggested being given the option to hear more about the topic on the phone at that moment, or to receive written information in the mail. Reflecting the desire for personalized information, patients recommended sending them a follow-up letter with feedback about health data they provided and topics discussed during the call.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
This series of in-depth interviews among health plan members with diabetes revealed initial skepticism about automated telephone outreach interventions, but apparent willingness to participate. This study underscored the persistent need among health plan members with diabetes for information about their disease. Even patients who had been diagnosed many years previously had knowledge gaps related to diabetes care and self-management. Patients want more information and feedback about their own behaviors and self-monitoring. These observations are well-aligned with the growing emphasis on self-management strategies for diabetes and other chronic conditions [25, 26].
Consistent with other studies [14, 27, 28], our results indicate that, while not suitable for every patient, automated telephone outreach may play an important role in providing information to patients with diabetes and helping them manage their condition. While some patients claimed they would never participate in a program that includes telephone calls from a computer, most—even those who were initially negative toward the idea—felt that an automated system could be helpful. As others have found [29], patients were amenable to receiving computerized reminder phone calls even when they were generally opposed to computerized phone systems. Patients stressed the importance of flexibility, as Stricklin et al. found [30], so they could take the calls at their convenience and initiate calls when they had time, specific questions or interest in hearing information from a menu of pre-recorded topics. Diverse patients, especially those who had been diagnosed with diabetes more than 5 years prior, noted that they would appreciate having the computer or a live person call them periodically to provide feedback on their condition. Although our study did not address the comparison directly, it is logical to suspect that some patients will prefer and respond more favorably to a human interaction as an adjunct to the automated calls, although this human interaction would add cost to such a program.
The request for specific and detailed advice has been found previously [15]. Nonetheless, the knowledge gaps identified in this study suggest that outreach interventions cannot rely solely on patients' preferences for content, as patients may not know what information they are lacking. From these observations, we postulate that providing individualized information on topics of personal interest to participants may serve as a ‘hook’ for achieving and maintaining their participation in an intervention program that has multiple objectives. While some patients expressed interest in being able to dial in, at their own initiation, to listen to health information that they would select from a menu, prior experience suggests that this passive approach would not be sufficient to change behavior; this kind of ‘health library’ may be considered as an adjunct to a computerized outreach program.
Study limitations include the relatively small sample size, which does not allow extensive comparative analyses according to patient characteristics. Perceptions of the automated telephone outreach intervention may vary by patient characteristics but could not be definitively determined given the sample size. Additionally, all participants were members in the same health plan, and there were unequal numbers of individuals in subgroups. However, our methods of purposive sampling enabled us to capture a range of perspectives from a diverse array of patients. While the specific findings may not be generalizable to all health care organizations, the approach to intervention design would be broadly applicable in settings addressing any chronic condition with established measures of quality, patients who need to practice self-care as well as follow up with professionals, and a need for continuing education.
This study demonstrates the value of in-depth interviews with health plan members prior to designing an intervention. Analyzing participants' own words allowed us to identify and understand the areas of concern and interest among our intervention population and contributed substantially to tailoring the design and content of an automated telephone outreach intervention program, recently implemented among 600 individuals with diabetes within our health plan (Table 2). This intervention was individualized based on patients' previously completed referrals and laboratory testing; existing technology will enable future iterations to incorporate patient-provided information (e.g. weights, blood sugar levels, diet preferences) to provide additional personalization. Population-based programs to improve the care of chronic conditions should consider in-depth interviews and qualitative analysis of constituents' preferences to inform the development of quality improvement interventions.
|
|
Funding |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
This work was supported by a Junior Faculty Career Development Award (01-05-JF-40) from the American Diabetes Association to Dr. Simon. Dr. Piette is a VA Research Career Scientist.
|
Acknowledgements |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
The authors gratefully acknowledge the support of the Eliza Corporation (www.elizacorp.com), which provided the audio recorded interactive voice response sample used in this study.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
- The Diabetes Control and Complications Trial Research Group. The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. N Engl J Med (1993) 329:977–86.
[Abstract/Free Full Text] - The ALLHAT Officers and Coordinators for the ALLHAT Collaborative Research Group. Major outcomes in high-risk hypertensive patients randomized to angiotensin-converting enzyme inhibitor or calcium channel blocker vs diuretic: the antihypertensive and lipid-lowering treatment to prevent heart attack trial (ALLHAT). JAMA (2002) 288:2981–97.
[Abstract/Free Full Text] - Sacks FM, Pfeffer MA, Moye LA, et al. The effect of pravastatin on coronary events after myocardial infarction in patients with average cholesterol levels. N Engl J Med (1996) 335:1001–09.
[Abstract/Free Full Text] - McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med (2003) 348:2635–45.
[Abstract/Free Full Text] - Saaddine JB, Engelgau MM, Beckles GL, et al. A diabetes report card for the United States: quality of care in the 1990s. Ann Intern Med (2002) 136:565–74.
[Abstract/Free Full Text] - Safford M, Eaton L, Hawley G, et al. Disparities in use of lipid-lowering medications among people with type 2 diabetes mellitus. Arch Intern Med (2003) 163:922–8.
[Abstract/Free Full Text] - Adams AS, Mah C, Soumerai SB, et al. Barriers to self-monitoring of blood glucose among adults with diabetes in an HMO: a cross sectional study. BMC Health Serv Res (2003) 19:1–6.
- UK Prospective Diabetes Study (UKPDS) Group. Effect of intensive blood-glucose control with metformin on complications in overweight patients with type 2 diabetes (UKPDS 34). Lancet (1998) 352:854–65.[CrossRef][Web of Science][Medline]
- UK Prospective Diabetes Study (UKPDS) Group. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). Lancet (1998) 352:837–53.[CrossRef][Web of Science][Medline]
- Barnett TE, Chumbler NR, Vogel WB, et al. The effectiveness of a care coordination home telehealth program for veterans with diabetes mellitus: a 2-year follow-up. Am J Manag Care (2006) 12:467–74.[Web of Science][Medline]
- Maljanian R, Grey N, Staff I, et al. Intensive telephone follow-up to a hospital-based disease management model for patients with diabetes mellitus. Dis Manag (2005) 8:15–25.[CrossRef][Medline]
- Biermann E, Dietrich W, Rihl J, et al. Are there time and cost savings by using telemanagement for patients on intensified insulin therapy? A randomised, controlled trial. Comput Methods Program Biomed (2002) 69:137–46.[CrossRef][Web of Science][Medline]
- Krishna S, Balas EA, Boren SA, et al. Patient acceptance of educational voice messages: a review of controlled clinical studies. Methods Inf Med (2002) 45:360–9.
- Boren SA, De Leo G, Chanestsa FF, et al. Evaluation of a diabetes education call center intervention. Telemed J E Health (2006) 12:457–65.[CrossRef][Web of Science][Medline]
- Piette JD. Moving diabetes management from clinic to community: development of a prototype based on automated voice messaging. Diabetes Educ (1997) 23:672–80.
[Free Full Text] - Sacco WP, Morrison AD, Malone JI. A brief, regular, proactive telephone ‘coaching’ intervention for diabetes: rationale, description, and preliminary results. J Diabetes Complications (2004) 18:113–18.[CrossRef][Web of Science][Medline]
- Vollmer WM, Kirshner M, Peters D, et al. Use and impact of an automated telephone outreach system for asthma in a managed care setting. Am J Manag Care (2006) 12:725–33.[Web of Science][Medline]
- Biem HJ, Turnell RW, D'Arcy C. Computer telephony: automated calls for medical care. Clin Invest Med (2003) 26:259–68.[Web of Science][Medline]
- Mundt JC, Moore HK, Bean P. An interactive voice response program to reduce drinking relapse: a feasibility study. J Subst Abuse Treat (2006) 30:21–29.[CrossRef][Web of Science][Medline]
- Levin E, Levin A. Evaluation of spoken dialogue technology for real-time health data collection. J Med Internet Res (2006) 8:e30.[Medline]
- Piette JD. Interactive behavior change technology to support diabetes self-management: where do we stand? Diabetes Care (2007) 30:2425–32.
[Free Full Text] - National Quality Measures Clearinghouse. Complete summary: comprehensive diabetes care: percentage of members with diabetes mellitus (type 1 and type 2) who had an eye exam (retinal) performed (2008) http://www.qualitymeasures.ahrq.gov/summary/summary.aspx?ss=1&doc_id=5766&string= (27 February 2008, date last accessed).
- Goldman RE, Hunt MK, Allen JD, et al. The life history interview method: applications to intervention development. Health Educ Behav (2003) 30:564–81.
[Abstract/Free Full Text] - Borkan J. Immersion/crystallization. In: Doing Qualitative Research—Crabtree B, Miller W, eds. (1999) 2nd edn. Thousand Oaks, CA: Sage Publications.
- Wagner EH. What will it take to improve care for chronic illness? Eff Clin Pract (1998) 1:2–4.[Medline]
- Bodenheimer T, Lorig K, Holman H, et al. Patient self-management of chronic disease in primary care. JAMA (2002) 288:2469–75.
[Abstract/Free Full Text] - Piette JD, Weinberger M, Kraemer FB, et al. Impact of automated calls with nurse follow-up on diabetes treatment outcomes in a Department of Veterans Affairs Health Care System: a randomized controlled trial. Diabetes Care (2001) 24:202–8.
[Abstract/Free Full Text] - Piette JD, Weinberger M, McPhee SJ. The effect of automated calls with telephone nurse follow-up on patient-centered outcomes of diabetes care: a randomized, controlled trial. Med Care (2000) 38:218–30.[CrossRef][Web of Science][Medline]
- Dini EF, Linkins RW, Sigafoos J. The impact of computer-generated messages on childhood immunization coverage. Am J Prev Med (2000) 19:68–70.[CrossRef][Medline]
- Stricklin ML, Jones S, Niles SA. Home Talk/Healthy Talk: improving patient's health status with telephone technology. Home Health care Nurse (2000) 18:53–61.[CrossRef]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||