OUP user menu

Comparing physician and patient perceptions of quality in ambulatory care

Roger Levine, Karen Shore, James Lubalin, Steven Garfinkel, Margarita Hurtado, Kristin Carman
DOI: http://dx.doi.org/10.1093/intqhc/mzs023 348-356 First published online: 22 May 2012

Abstract

Objective If quality improvement interventions, value-based purchasing and/or certification are based on patient assessments, a common definition of quality should be shared by clinicians and patients. The study's objectives were to determine (i) how patients and clinicians define quality care, (ii) in what ways patients' and physicians' definitions differ and (iii) whether patients' definitions vary by ethnicity.

Design We used the critical incident technique to interview participants about behaviors that resulted in office visits being considered either good or poor quality and compared the prevalence of different types of ‘quality’ behaviors reported to identify commonalities and differences.

Setting Hawaii and Chicago.

Participants A total of 168 patients and 39 clinicians.

Results We developed a taxonomy, comprising 9 major categories and 106 subcategories of behaviors responsible for quality visits. Almost all clinicians and patients agreed that clinical skill, rapport and health-related communication behaviors were key elements. Patients were more likely to report behaviors demonstrating thoroughness in routine examinations, spending enough time with them, engaging them and being treated with courtesy and respect as drivers of a quality office visit than were physicians.

Conclusions Increased clinician awareness of the behaviors that patients believe are the drivers of a quality office visit can help clinicians improve patients' experience of care and experience-based measures of quality.

  • quality of health care
  • patient satisfaction
  • physician-patient relations
  • task performance and analysis

Introduction

Measuring the quality of care—as an end in itself and for value-based purchasing—is a fundamental force in health care today. How quality is defined and measured has implications for patients' well-being and physicians' compensation and board certification. Although experts are converging on the technical, administrative and patient-centeredness concepts that define quality at a high level, patients and physicians must reach a common understanding of what defines quality in the daily practice of medicine if quality improvement and value-based purchasing are to work well for all parties [16].

This is especially so for patient experience of care, which is now widely accepted by payors, accrediting organizations and regulators as a valid component of quality [1]. Specialty societies and certification boards are actively engaged in efforts to measure patient experience, but a high proportion of physicians remain skeptical of its relevance to quality. Physicians must understand the criteria that patients use to assess performance if they are to score well on patient assessments that will be used as pay-for-performance and board certification criteria.

The objective of our research is to determine at the micro level (i) how patients and physicians define quality care, (ii) in what ways patients' definitions differ from physicians' definitions and (iii) whether patients' definitions vary by patient race/ethnicity. With the advent of patient-centered care concepts [1, 2] and the ability to measure patient assessments of health care in consistent ways [35], clinicians, payors, accrediting organizations and regulators have focused on patient experiences as a complement to clinical measures [69]. Nevertheless, physicians sometimes express skepticism about the value of patient experience data [10] or patient-centered practice [11]. Often this skepticism is based on the assumption that patient assessments relate to the physician's interpersonal skills and not to specific behaviors that represent quality practices. Physicians may only take patient experience data seriously if they believe these data are measuring quality as they understand it. Consequently, this study asked the question: do patients assess the quality of care they receive in ambulatory care settings using the same or different criteria as physicians?

Previous research has examined the differences in physician and patient assessments of health-care encounters as well as differences in patient assessments as a function of patient race/ethnicity. One study found that physicians and patients both commented on listening and explanation skills and length of the visit, but physicians also focused on issues such as visit management and non-verbal behavior [12]. Another study found an agreement that the three most important competencies were technical skill, ethical conduct and effective communication [13]. Differences in the Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys, as a function of patient ethnicity, have been observed. In the CAHPS Health Plan Survey, Spanish language respondents had worse reports of actual health experiences even though their ratings of their personal doctors, specialists, overall health care and health plan were higher than comparable ratings provided by English-speaking respondents [14]. And, in the CAHPS Hospital Survey, in spite of extensive efforts to insure conceptual equivalence in Spanish translations of these instruments, Spanish language respondents continued to provide higher overall ratings. Additionally, their responses to items about specific health experiences were often more positive than those provided by English language respondents [15]. These studies are limited in what they tell us about the underlying criteria used to define the quality in medical encounters. Consequently, our study used the critical incident technique (CIT) to determine if physicians and patients have common or disparate understandings of quality.

The CIT has been used for more than half a century to describe and classify human behaviors in many fields of employment including health care [1618], to assess patient experiences [1921], patient responses to chronic illness and treatment [2224], quality indicators [2528], nurses' perceptions of quality of palliative care [29], surgical training [30], survey and scale development [31, 32], asthma self-management [33] and adherence to drug treatment regimens [3436]. The CIT provides a methodology for collecting descriptions of specific behaviors that are causally related to an outcome of interest, analyzing these descriptions (critical incidents) and developing a comprehensive taxonomy of such behaviors, as well as assessing the reliability and comprehensiveness of that taxonomy.

Methods

We developed interview guides and conducted in-depth telephone interviews (45–60 min) with patients and clinicians (physicians, nurse practitioners and physician assistants). Respondents were asked to think about recent office visits and to provide detailed information about behaviors that made each visit good or bad. We used this information to write descriptions of these behaviors, the context in which they occurred and the result of these behaviors (called critical incidents) using a standardized template (see Fig. 1) The standardized template reflects the types of questions and probes that were used to elicit this information but is not the exact wording. These critical incidents were used to develop a taxonomy of behaviors.

Figure 1

Examples of patient and clinician critical incidents.

Respondents were identified primarily through two health plans, in the Chicago area and Hawaii, and through a recruiting firm. (Four interviews were conducted with patients in Florida whose predominant language was Spanish, to enable the collection of critical incidents that might be idiosyncratic to such patients.) Because of HIPAA regulations, health plans recruited patients by sending out letters inviting interested people to contact us. Since we did not have control over the number of letters sent out, it is not possible to calculate a participation rate.

We attempted to interview 40 clinicians and 160 patients and to distribute the patients equally among four racial–ethnic groups (Caucasian, African American, Asian American and Hispanic) and between the geographic areas. In our final sample, 90 patients and 19 clinicians were from the Chicago area, 74 patients and 16 clinicians were from Hawaii and 4 patients and 4 clinicians were from other parts of the country. The reason for oversampling minorities was to increase the likelihood of capturing quality behaviors reflective of issues that might be culturally specific. Table 1 provides the number and characteristics of interview respondents. The clinicians, 30 of whom were physicians, each received $125 for participating. We interviewed 168 patients, who each received $35.

View this table:
Table 1

Demographics, by type of respondent

 Type of respondent
Patient (n = 168)Clinician (n = 39)
Race/ethnicity, n (%)
 Asian48 (29)9 (23)
 Black36 (21)0 (0)
 Hispanic31 (19)1 (3)
 Multi-racial9 (5)2 (5)
 White44 (26)27 (69)
Gender, n (%)
 Female118 (70)12 (31)
 Male50 (30)27 (69)
Age, n (%)
 21–3034 (20)2 (5)
 30–3926 (15)0 (0)
 40–4930 (18)15 (38)
 50–5934 (20)17 (44)
 60–6922 (13)4 (10)
 70 and over21 (13)1 (3)
 Missing1 (1)0 (0)
  • Race/ethnicity was determined through respondent's response to items listing the federal reporting categories for race and ethnicity. Respondents who answered affirmatively to more than one race category were classified as ‘multi-racial’.

Eleven interviewers were trained in the CIT, conducted phone interviews with respondents and prepared critical incidents from their detailed notes and a review of interview tapes or transcripts. Three of the interviewers were bilingual, enabling Spanish-speaking respondents to be interviewed in Spanish. Each incident was reviewed by another researcher to ensure it met the criteria required for a good critical incident. Incidents failing to meet these criteria were sent back to the interviewer who would review the source tape or transcript to enable edits. If adequate edits could not be made, the incident was discarded. A total of 14 incidents (0.5%) were discarded.

A set of 200 randomly selected incidents was used to develop the initial taxonomy. Two teams of two senior researchers read each incident and prepared a description of the incident's focal behavior, grouping together critical incidents whose focal behaviors were judged to be virtually identical. The development of critical incident taxonomies is grounded in the data and is not based on a predefined conceptual model or theory. Categories of critical incidents describing related behaviors were grouped together into larger aggregations. The researchers met and reconciled their respective taxonomies. The researchers then reviewed subsequent sets of incidents. New categories and subcategories were created for incidents that did not fit into existing categories. Codes were compared for agreement. Inconsistencies were used to inform revisions of category descriptions and taxonomic organizational revisions.

The classification scheme was reviewed until there was an agreement that the behaviors grouped together at the finest level were homogeneous groups and that the taxonomy was logical, internally consistent and appropriate for its intended functions [18]. Saturation was achieved when the final 100 incidents resulted in the creation of no major categories and two or fewer subcategories. The reliability of this saturated taxonomy was established through inter-rater reliability checks: Cohen's weighted kappa for agreement at the subcategory level was 0.68.

The prevalence of incidents in the major categories and subcategories reported by clinicians and by patients was compared through the chi-square statistic, using SAS Version 9.1. We limited subcategory comparisons to those subcategories for which 10 or more respondents reported an incident. To compare the prevalence as a function of patient race/ethnicity, we estimated a multivariate probit model, using SAS Version 9.1, for each category and subcategory (with at least 10 respondents) with race/ethnicity, age and gender as explanatory variables. These models determined if persons in each of the four self-identified racial–ethnic groups were more or less likely than self-identified Caucasians to report an incident in each category or subcategory. Similar procedures and models were employed to compare clinicians who were physicians with other clinicians.

Results

The composite taxonomy

Our first objective was to establish an exhaustive taxonomy of clinician and staff behaviors responsible for a quality visit. A total of 2997 incidents were identified from 207 respondents. Clinicians provided an average of 15.6 incidents per respondent and patients provided 12.7. Examples are shown in Fig. 1.

The incidents were grouped into nine broad behavioral categories (Fig. 2). More than 80% of the incidents fell into three categories: clinical skills (33%), rapport (27%) and health-related communication (21%). Office practices (8%) and patient advocacy (4%) included most of the rest. We identified 106 first-tier subcategories, mostly within the first three broad categories. Figure 3 displays the subcategory structure within one of the major categories, clinical skills.

Figure 2

Major taxonomic categories. A detailed copy of the full taxonomy may be obtained by contacting the corresponding author.

Figure 3

Subcategories included in major category of clinical skills.

Convergence between clinicians and patients

Our second objective was to determine if clinicians and patients differ in their conceptualization of quality. To assess the extent of convergence, we compared the number of clinicians and patients who provided critical incidents in each category and subcategories for which 10 or more respondents reported an incident. Results are presented in Table 2.

View this table:
Table 2

Comparisons of patient and clinician reports of behaviors in major categories and subcategories

PatientsPercentCliniciansPercentProbability
Clinical skills158943897
 Provides appropriate, efficient, thorough examination for acute problem(s)965730770.02
 Prescribes or recommends appropriate/effective medication or other treatment81481949
 Provides good follow-up care603621540.04
 Gives thorough routine examination54326150.04
 Gives appropriate referrals50301231
 Gives thorough care271612310.04
 Makes accurate diagnosis2716820
 Allows patient to participate in decisions about care2213513
 Minimizes pain and discomfort211213
 Performs effective treatment191125
 Conducts visit quickly and efficiently14700
 Pursues or recommends conservative course of treatment first10638
 Motivates compliance behavior74821<0.001
 Notices problems, characteristics or issues unrelated to patient's reason for visit42615<0.001
 Health-related communication133793590
 Gives patient complete and accurate information and/or instructions regarding tests, treatment, treatment options, treatment plan, medications, follow-up care and post-operative care68412256
 Explains medical condition thoroughly and in terms patient/caregiver can understand57341744
 Provides information on non-medical ways patient can care for self or for condition492918460.04
 Explains procedures right before/while they are being performed301838
 Effectively answers patient's questions or addresses patient concerns251538
 Discusses patient condition or treatment in a direct, straightforward manner159615
 Gives good explanation to emotionally prepare patient for future medical tests, treatment, procedures or surgery15938
 Provides literature, diagrams, models or refers to websites1481128<0.001
 Checks to see if patient has questions14823
 Explains test or surgery result to patient117410
 Encourages patient to seek preventive care or screening646150.01
Rapport154923487
 Treats patient with courtesy and respect102611231<0.001
 Engages the patient955713330.01
 Takes time with patient8953820<0.001
 Reassures, calms or verbally comforts patient4124615
 Listens to patient3320615
 Asks interested and appropriate questions2615410
 Acknowledges errors/mistakes/tardiness171038
 Distracts patient during testing, treatment or procedure by talking with them9513
Advocacy: ensuring patients get needed care523121540.01
 Facilitates access to medications or medical supplies3521820
 Facilitates access to other providers or services1381231<0.001
Office practices, office and ancillary staff85502564
 Office practices, policies and procedures442617440.03
 Care from ancillary staff including nurses, technicians, therapists, etc.4426513
 Office staff conduct281738
Information seeking29171539<0.01
 Obtains relevant medical information about patient24141436<0.01
Unexpected extras1598200.04
Accessibility/availability5231821
 Is available on short notice211338
 Encourages patient to come in or call anytime for follow-up care2012513
 Returns phone calls or e-mails13800
Ethical behavior10625

Major categories

Clinicians and patients were about equally likely to cite each of the four most common major categories (clinical skills, health-related communication, rapport and office practices/office and ancillary staff). Clinicians were more likely to provide incidents for three categories: advocacy: ensuring patients get needed care (54% of clinicians vs. 31% of patients), information-seeking by the clinician (39% vs. 17%) and unexpected extras (21% vs. 9%).

At the major category level, it appears that clinicians and patients differ in describing quality in terms of behaviors that are not generally apparent to patients—clinicians advocating with others on their patients' behalf (advocacy), clinicians seeking information about their patients from others (information-seeking) and clinicians doing things beyond normal expectations (unexpected extras). Moreover, these are not the most salient dimensions of quality for either clinicians or patients. Additionally, it should be noted that clinicians are reporting many more care experiences than patients. For example, many clinicians will advocate for a patient at some point, but many patients may be unaware when they receive this service. So, it is not surprising that the proportion of clinicians reporting advocacy behaviors is greater than the proportion of patients who report such behaviors.

Subcategories

Some interesting differences emerged among the first-tier subcategories. Within the domain of clinical skills, clinicians were more likely than patients to ‘use examinations for acute care problems, providing good follow-up care, motivating compliance behavior and noticing something unrelated to the presenting complaint to define quality’. Patients were more likely to cite thorough routine examinations. Thus, clinicians seem to recognize that their clinical skills come into play when they have to deal with something that is not routine, but patients, who most often receive routine care, are less likely to report on more challenging aspects of care.

Within the health-related communication domain, significant differences were observed for subcategories ‘providing information to patients on non-medical ways to care for their condition’ (46% of clinicians vs. 29% of patients); ‘providing patients with the literature, diagrams, models or referrals to websites (28% vs. 8%) and encouraging the patient to seek preventive care or screening’ (15% vs. 4%). In all other subcategories involving health-related communication, there were no significant differences between clinicians and patients.

Within the rapport domain, patients were significantly more likely than clinicians to report an incident involving ‘engaging the patient’ (33% vs. 57%), ‘treating them with courtesy and respect’ (31% vs. 61%) or ‘taking time with the patient’ (21% vs. 53%). There were no subcategories in which clinicians were more likely than patients to report incidents reflecting quality.

Within the office practices, office and ancillary staff domain, there was only one significant difference: clinicians were more likely than patients (44% vs. 26%) to identify ‘office practices, policies and procedures’ as reflecting quality.

Clinicians were significantly more likely than patients to report ‘contacting other clinicians or entities on behalf of patients’ (31% vs. 8%) in the advocacy domain and ‘obtaining relevant medical information’ (36% vs. 14%) in the information-seeking domain. These behaviors frequently occur behind the scenes of a medical encounter and are less likely to be perceived by patients.

Variation by patient race/ethnicity

Race/ethnicity was significant in only 1 of the 45 categories or subcategories (data not shown). This is slightly less than would be expected by chance, suggesting that race/ethnicity is relatively unimportant in conceptualization of quality. Thus, we treated patients as a single group when comparing them to clinicians.

Variation by clinical training

We also compared the physicians with the non-physicians (nurse practitioners and physician assistants. Significant differences were found in only two subcategories (data not shown), suggesting minimal differences in the conceptualization of quality by type of training. Accordingly, we treated clinicians as a homogeneous group when comparing them to patients.

Discussion

In reporting on the quality of their ambulatory health-care encounters, nearly all clinicians and patients described at least one incident involving clinical skills. At least three-quarters of both clinicians and patients described at least one incident involving health-related communication and at least one incident involving rapport. These three dimensions account for more than three-quarters of all patient- and clinician-reported incidents. Clinicians were more likely than patients to report incidents using dimensions of medical care that go on behind the scenes. Thus, clinicians reported more incidents than patients that involved: office practices or staff, advocacy, information seeking on behalf of patients and actions beyond normal expectations in providing care.

The important story, however, is at the subcategory level, where we gain a greater understanding of what patients and clinicians focused on in making their judgments. Nearly all patients and physicians share the view that clinical or technical skill is a key factor in ambulatory health-care quality and is reflected by physicians who: do thorough examinations; diagnose problems accurately; allow patients to participate in care decisions; provide appropriate medications, treatments or referrals and provide good follow-up care. Where they differ is that patients focus more on routine care, while clinicians focus more on challenging aspects of care, including acute care, thorough care, motivating compliance and noticing problems unrelated to the presenting complaint.

A substantial majority of patients and clinicians think that health-related communications are important and share the view that quality is reflected by clinicians who: explain patients' medical conditions thoroughly and in ways patients can understand; give patients complete and accurate information and/or instructions about their care; explain procedures right before or while they are being performed and effectively respond to questions and concerns. Clinicians are more likely than patients to consider quality care to include providing patients with information about non-medical ways patients can care for themselves as well as with literature, diagrams, models and websites.

The vast majority of both patients and clinicians view rapport to be important to quality. They share the view that good rapport is reflected by clinicians who: listen to patients; ask interested and appropriate questions and reassure, calm or verbally comfort patients. Patients are more likely than clinicians to consider being treated with courtesy and respect, engagement by the clinician and adequate time with the clinician as important aspects of quality care.

While most clinicians focus on office practices, policies and procedures when they think about how their office influences the quality of care, nearly half of all patients are more focused on the care they receive there from ancillary staff. Again, this likely reflects the different experiences of the two types of respondents, because clinicians are less likely than patients to directly observe services provided by ancillary staff.

Clinicians are more likely than patients to see behaviors in other subcategories as reflecting health-care quality. They are more apt to describe incidents in which they: advocate on behalf of patients with others, obtain medical information about patients and engage in behaviors that are above and beyond usual practice. Patients are more likely than clinicians to make attributions of quality when clinicians provide services at convenient times and locations in ways that are responsive to patient needs. It is intriguing to speculate on the reasons for the existence of differences in patient and clinician perceptions of quality ambulatory care. Differences in values and differences in perceptions almost certainly are contributing factors.

These findings should be considered in the light of several limitations. We used a convenience sample of patients and clinicians, selected from two small regions of the country. Future research based on representative national samples would be a valuable next step. Second, clinicians reported more incidents on an average than patients because they experience many more encounters than patients. This could explain why clinicians had the greater likelihood of reporting an incident in subcategories that showed significant differences between clinicians and patients. Finally, we conducted a large number of statistical comparisons. Some of these significant differences may be due to chance.

The CIT is purely inductive and exhaustive. No a priori domain structure is assumed and analysis continues until the respondent-generated taxonomy is saturated. Thus, it is unlikely that patients would have been constrained from reporting at least one incident in every salient subcategory. Furthermore, there is a consistent, plausible interpretation for the instances in which clinicians are significantly more likely to report an incident: clinicians are privy to events that patients do not observe.

Previous methods used to identify the components of a quality ambulatory care visit include literature reviews, qualitative interviews and empirical quantitative studies, but these approaches do not provide reasonable assurances that the ultimate listing of factors is comprehensive. Furthermore, building on a priori conceptualizations and schema can lead to a failure to recognize unanticipated factors. The CIT addresses these shortcomings by eliciting a comprehensive set of behaviors that patients and clinicians describe in recounting actual medical encounters as either good or poor quality care.

In sum, we have expanded our understanding of the commonalities and differences in patient and clinician assessments of ambulatory health-care quality. The key question with which we began was whether clinicians and patients use the same or different criteria in assessing the quality of ambulatory care encounters. This is an important question for clinicians because performance-based compensation arrangements and specialty-board certification programs increasingly include patient survey measures. The growing impact of patient surveys means that, more than ever, clinicians must understand how patients conceive quality.

Our results indicate that clinicians and patients have largely similar conceptions of the important dimensions of quality. They were equally likely to cite incidents in the four major categories of quality that account for over 90% of reported incidents: clinical skills, health-related communication, rapport and office practices, office and ancillary staff. Among the other major categories, clinicians were significantly more likely than patients to cite incidents in three: advocacy, information seeking and unexpected extras.

Patients were more likely than clinicians to report incidents in four subcategories: thorough routine examinations, being treated with courtesy and respect, being engaged by the clinician and having the clinician take time with them. Since patients provided fewer incidents and have a more limited set of experiences on which to report, these subcategories should be areas of focus for enhancing patients' care experiences. It is noteworthy that these areas are all measured by the most common patient surveys, particularly the CAHPS Clinician and Group Survey.

Funding

This work was supported by the Agency for Healthcare Research and Quality's Consumer Assessment of Healthcare Providers and Systems II [1 U18 HS13193-01].

Acknowledgements

We thank Ms Judy Mitchell and Ms Rebecca Morales for their assistance in the development of our critical incident taxonomy. Dr Shore led the data collection and taxonomy development; Drs Lubalin and Garfinkel had lead responsibility for data interpretation; Dr Levine provided methodological leadership and led the data analyses. Drs Levine, Shore, Lubalin and Hurtado contributed substantially to the writing and review of the manuscript. Dr Levine had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. All authors contributed substantially to the design of the study. There are no conflicts of interest for any of the authors. This project was reviewed and approved by the Institutional Review Board of the American Institutes for Research.

References

View Abstract