International Journal for Quality in Health Care 14:329-336 (2002)
© 2002 International Society for Quality in Health Care
Methodology Matters |
Qualitative research methods
School of Public Affairs, Baruch College, New York, USA
The use of rigorous qualitative research methods can enhance the development of quality measures the development and dissemination of comparative quality reports as well as quality improvement efforts. This paper describes how such methods have been and can be used and identifies how to improve the use of such methods in studying quality of care. Focus groups and cognitive interviews are now a standard part of the development of valid and reliable survey instruments. They are particularly useful in developing surveys to gather data on the experiences and responses of patients and consumers to plans services and providers. These two methods have also been adapted and applied to improve the development and dissemination of comparative quality reports to consumers and other audiences while key informant interviews and focus groups have been critical in the exploratory assessment of stakeholder responses to reports and their effects on consumers. Interviews have also been used to identify best practices found in health plans receiving high scores on the Consumer Assessment of Health Plans Surveys and measures of effectiveness in the Health Employer Data and Information Set. It would be valuable to widen the use of qualitative methods especially structured observations to document in detail the delivery of services designed to improve quality so the implementation of complex processes can be more carefully measured and related to outcomes. The design and conduct of rigorous qualitative research takes a skilled and experienced team. Issues commonly faced in quantitative work must also be addressed in qualitative studies including study design specification of the unit of analysis sampling instrument design and administration and in particular data analysis. It is especially critical that the analysis and interpretation process be deliberate and thorough to avoid the use of initial impressions rather than detailed examination of the raw data.
Keywords: CAHPS®, consumer surveys, focus groups, HEDIS®, key informant interviews, qualitative data analysis, qualitative methods, structured observations
The use of rigorous qualitative research methods has been on the rise in health services and health policy research. A workshop held in late 1998 jointly sponsored by the federal Agency for Healthcare Research and Quality (AHRQ) and the Robert Wood Johnson Foundation, showcased a wide range of studies by highly regarded investigators that used qualitative methods either on their own or in tandem with quantitative methods. In a Special Issue of the journal Health Services Research issued a year later papers commissioned for this workshop highlighted both the contributions that qualitative methods can make [1–4] and strategies that can ensure they are used systematically and rigorously [5,6]. In his introduction to the Special Issue editor Stephen M. Shortell noted that the growth in the use of these methods ‘is consistent with developments in the social and policy sciences at large reflecting the need for more in-depth understanding of naturalistic settings the importance of understanding context and the complexity of implementing social change’ [7].
Health care is delivered in naturalistic settings and in a wide range of professional organizational and community contexts. There has been of course very rapid change in health care. Areas of rapid change include the measurement of quality the dissemination of comparative quality information and efforts to improve quality. Work in this area has already benefited from the increased use of qualitative and mixed method research and can do so even more in the future. This paper discusses a wide range of applications of qualitative methods in the broad quality arena using actual studies as examples and illustrating a wide range of specific methods available. It will close with an analysis of the ingredients that distinguish the creative rigorous use of qualitative methods from those that are poorly thought out or executed.
Qualitative methods in measurement development
Perhaps the qualitative methods most familiar to those in the field of health care quality are focus groups and cognitive interviews since both are increasingly used as a precursor to the development of survey instruments including those intended to measure quality. Focus groups were initially used in market research to test potential consumer responses to both products and their presentation in advertising campaigns [8,9]. They are now viewed by social scientists as a flexible and cost effective method for exploring attitudes experiences and responses of non-random samples of people who fit a particular profile [10–12].
Especially when developing a patient or consumer survey it is valuable to begin by identifying the issues that are most meaningful to consumers and patients as well as the language they do and do not use in talking about these issues [13]. Even if the investigator has a clear conceptual framework that s/he is using to guide survey development the most effective way to operationalize the framework can be illuminated by hearing in the course of a structured but open discussion how potential respondents think and talk about the issue. In the example given below the Consumer Assessment of Health Plans Study (CAHPS®), both focus groups and cognitive interviews were used to develop a core set of items intended to be common across the entire ‘family’ of CAHPS® instruments [14]. One or both of these methods were also used to develop and test items specific to a particular those CAHPS® survey such as for people who spoke Spanish [15] were on Medicaid [16] (see glossary) or Medicare [17] (see glossary) or for children [18]. Both were also used to develop a related instrument, the Consumer Assessment of Behavioral Health Services [19].
Immediately before the onset of CAHPS® the Agency for Health Care and Policy Research (the former name of AHRQ) commissioned a series of focus groups around the US that included consumers who varied in terms of age gender income education health status and source of insurance. The CAHPS® initiative was designed to develop valid and reliable survey instruments that could be used to collect information from consumers about their health care experiences under different insurance plans. Perhaps the most distinctive characteristic of CAHPS® was that the surveys were to be used to generate comparative scores that could be reported back to consumers for their use in choosing between available insurance plans [20]. Therefore the purpose of the focus groups was to identify (1) how group members as consumers viewed and defined health care quality; and (2) what kinds of quality information from fellow consumers they would and would not consider relevant useful and trustworthy [21]. The focus groups made clear that consumers defined quality in terms of access to care (e.g. finding a regular doctor getting referred to a specialist getting needed tests and treatments getting both regular and urgent care promptly not having excessive waits for appointments), the nature of interpersonal interactions (e.g. with physicians other clinicians medical office staff and health plan customer service staff) as well as technical/clinical quality convenience of facility location and other dimensions. However the groups also made it clear that while consumers thought that other consumers particularly those who were ‘like them’ could provide meaningful and trustworthy information on access and interpersonal interactions they could not provide such information on technical/clinical quality or even on convenience. With respect to technical/clinical quality consumers were well aware of the limits of their own clinical expertise and therefore of the expertise of other consumers. Cost and convenience were viewed as determined by the individual’s particular context so that someone else’s experience would be irrelevant to one’s own. While these were important dimensions of quality to consumers they recognized that a survey of other consumers would not generate the information they needed; factual data relevant to costs location hours etc. would be more useful. The results of these focus groups had a substantial impact on the dimensions of quality and ultimately the specific items that were and were not included in the core set of items that are a part of every CAHPS® instrument.
Cognitive interviews are considered by many to be an essential element of testing the reliability and validity of survey instruments in particular to determine whether or not items and response options are understood and consistently interpreted by potential respondents as intended by the investigator [22]. Feedback from the interview respondent can be obtained in many ways. In the ‘think aloud’ approach a person similar to a potential respondent completes the survey in the presence of the interviewer and verbalizes their responses to the question stem and the response options often restating the question in their own language. Alternatively the interviewer can administer the survey orally either by probing at the end of each question why the respondent chose the particular response option they did or by asking follow-up questions to make sure the question and response options were really understood. Debriefing can also be done after the respondent has completed the entire survey. Often the method chosen depends on the characteristics of the respondent group. For example cognitive interviews were conducted by the staff of The MEDSTAT Group [23] last year to develop a consumer experience survey for people, including persons with some degree of cognitive impairment or developmental disability who were enrolled in Medicaid home and community-based waiver services. In those interviews probes were administered after each question since asking about the entire interview at the end would have overwhelmed the respondents cognitively. Although a cognitive interview is likely to be far more structured than a typical qualitative interview it is still designed to be ‘open—ended’—responses are presumed to be unknown and the interviewer’s job is to elicit those responses rather than get the person to choose from among a previously determined set of responses.
In the CAHPS® project an important element of the ‘vetting’ of each survey instrument has been conducting cognitive interviews [14]. Cognitive interviews were conducted by all the CAHPS® teams. They pointed to a wide range of issues particularly around the use of terms that as health service researchers or health care professionals we tend to take for granted. Survey developers discovered, for example, that they had to be extremely explicit in specifying what we meant by a ‘regular source of medical care’ and define what we meant by ‘specialist’. They also learned that some dimensions of care were far more difficult to include on a consumer survey than others especially if only a single item was available. For example it proved infeasible to include a single item to tap consumers’ experience around receiving fragmented rather than coordinated care. The item tested asked whether consumers had ever received different or conflicting information from different providers. While most professionals would perceive the receipt of conflicting information as problematic a substantial subset of consumers interpreted the question differently and thought that getting different information from different providers was desirable akin to getting a ‘second opinion’.
Qualitative methods in developing and assessing reports and dissemination of comparative quality information
The reporting and dissemination of comparative quality information is a relatively new enterprise that has only been possible since reliable and relevant data have become available. In part because this is such a pioneering set of activities the exploratory discovery-oriented aspects of qualitative methods have made them a critical element of dozens of such efforts. Focus groups have been used to assess the relevance and comprehensibility of a wide range of quality measures intended to be reported as well as preferences for displaying data graphically. Thus to give just a few examples: (1) the National Committee for Quality Assurance (NCQA, www.ncqa.org) commissioned focus groups to assess candidate indicators for the measures of effectiveness it would include in the third iteration of its Health Employer Data and Information Set (HEDIS®) with Medicaid enrollees and people with Medicare [24]; (2) the Foundation for Accountability a non-profit organization whose mission is to identify and study ways to disseminate consumer-relevant comparative quality measures [25] commissioned groups to assess whether people without a specific medical condition would or would not respond to condition-specific quality indicators [26]; while (3) the Centers for Medicare & Medicaid Services (CMS formerly the US Health Care Financing Administration) used focus groups to identify which specific measures on a CAHPS® Medicare survey should be disseminated in its Medicare & You handbook [27]. These focus groups have illuminated not only such basic issues as which dimensions of quality are considered important or unimportant by consumers but also subtle and surprising issues such as which quality indicators consumers view as influenced by a health plan as compared with a physician or themselves [28] the relationship between the comprehensibility of an indicator and its perceived salience [29] and the confusion generated by presenting data displays that provide information on both comparative and absolute scores.
In a recent project supported by The Commonwealth Fund and designed to develop and test consumer reports on the quality of Medicare managed care plans (see glossary) in New York City we began by using focus groups to test the overall framework of performance dimensions that was most engaging to consumers. An important learning was that given the current ‘mental map’ of people with Medicare a report that limited itself only to comparative quality information and did not prominently include information on premiums and services covered, as well as explaining key differences between managed care plans and fee for service health care delivery would simply not work. The focus groups also clarified our dissemination strategy by demonstrating that the most interested audience for the materials we planned would be relatively low-income people with Medicare who were most likely to view managed care plans as an affordable way to get better service coverage. Participants from higher income levels were negatively disposed toward joining managed care plans and preferred to continue buying a Medigap (see glossary) supplement (an expense that many lower-income participants could ill afford). We also conducted focus groups of counselors whose responsibility it was to help people deal with Medicare-related problems. These groups helped us to gauge their own exposure to and understanding of quality information (highly limited) and their opinions of the ability of their clients to deal with quality information without personal assistance (also highly limited) [30].
Ultimately these focus groups helped us to develop an innovative strategy for creating materials. Rather than incorporate all the comparative information into one booklet or develop a booklet just on quality we decided to create a set of six related booklets that would cover a variety of managed care plan performance dimensions of interest to consumers. Rather than organize quality information by source (an option since we were providing both CAHPS® scores and scores on HEDIS® indicators) we decided to ‘map’ different quality measures onto the existing mental map we discerned through the focus groups. For example in focus groups both consumers and counselors confirmed the overwhelming interest of people on Medicare in the prescription benefits available in managed care plans. We therefore created a brief booklet that compared prescription benefits for all Medicare managed care plans in New York City and also included the scores recorded for a number of health care plans on a CAHPS® Medicare Managed Care survey item that asked about members’ experiences in getting the prescriptions they needed from their plan. We linked these two kinds of information with a discussion of the use of formularies and the preference for prescribing generic rather than brand name medications among managed care plans. Similarly we learned that consumers were very concerned about access to physicians in managed care plans. This led us to create a booklet titled ‘Getting a Plan With Good Doctors’, which included a wide range of CAHPS® items that address physician-related issues (ease of finding a good doctor a composite of items on physician communication and the overall rating of all health care providers in the managed care plan).
Cognitive interviewing
The methodology of cognitive interviewing has also been adopted by those developing materials to present comparative quality information. These methods were used in CAHPS® [31] and were also used in our New York City project [32]. Two rounds of cognitive interviews were conducted to test our materials both with people on Medicare and with the kinds of counselor we hoped would help us disseminate and explain the materials. As considerable research demonstrates in reporting the devil is often in the details and it is these details that can best be addressed through cognitive interviews. The method can and has been used to test both specific elements of reports such as titles labels data displays table layouts fonts colors photographs or other illustrations etc. and broader issues such as how people scan a multi-page document or whether they perceive the purpose of the overall document and actually understand the information correctly. Interviews provide material developers with significant clues to how a reader cognitively processes a document in a manner far more subtle than can be obtained merely by doing a test of its ‘readability’ or ‘literacy level’. Cognitive tests gave us literally pages of specific ways of improving the materials but also confirmed that for many people with Medicare personal assistance would be essential. We also learned a good deal about what kind of information would have to be provided to counselors if they were to be effective at providing such assistance.
Observations and key informant interviews
This leads to the use of qualitative methods beyond the stage of comparative quality materials development. In our New York City project we also used two other classic qualitative methods: observations and key informant interviews. Our design called for training counselors. Using a structured observation protocol pilot training sessions were observed and the observations used to refine the training. We also held follow-up support sessions for the counselors we trained. These sessions were also observed using a structured approach and provided considerable information on the progress of the counselors in using the materials. In this mixed method project we also used a brief survey to get quantifiable information on several important intermediate outcomes including counselor response to training knowledge gains and understanding of how to use the booklets. But we gained considerably more information through a series of telephone interviews using a semi-structured protocol both with the counselors and their supervisors. It had proved to be quite difficult for counselors to incorporate the use of materials into their day-to-day practice with clients. If we had simply looked at the numerical information about how many booklets were distributed we would have gained little understanding of the underlying dynamics of our approach. The open-ended interviews in contrast provided rich and detailed information about the myriad factors that created difficulties for counselors ranging from inaccurate initial assessments of how many clients needed the help to organizational changes such as shifts in their job assignments to surprise at how difficult it was to provide this kind of assistance compared with most of their other work. The one factor that it appeared was not particularly significant in discouraging the use of booklets was in fact the booklets themselves. Thus qualitative methods helped us discover both the weaknesses and the strengths of our approach which is far more useful than evidence that something unspecified about the approach was seriously amiss.
Indeed qualitative methods have become an essential element in the methodological repertoire of evaluation researchers who are interested in documenting and examining the ‘independent variable’—that is the intervention—in a more nuanced and complete manner [33]. Evaluations conducted of CAHPS® demonstration projects have typically included focus groups composed of members of the target audiences for CAHPS® reports as well as extensive key informant interviews conducted with the multiple demonstration project participants [34]. In addition focus groups and interviews have been used to assess the impact of comparative quality reports on other audiences such as health plans [35,36] and hospitals insurers and purchasers [37]. These methods are virtually essential to determine first the extent to which an intervention was implemented as planned (for example were comparative quality materials actually disseminated) and which specific aspects of the intervention caused difficulties for which specific groups of consumers (for example in a process evaluation of an Oregon CAHPS® demonstration we actually found a group of engineers who wanted more detailed tables rather than simpler graphics). Since this work is still at an early stage this kind of information is essential to move the field forward.
Qualitative methods in the study of delivery system quality
These same questions about the details of processes and the implementation of innovations are likely to be present in a wide range of studies of how they affect quality of care outcomes. Unfortunately many health service researchers are still defining the independent variable (i.e. the intervention) in such studies as dichotomous: either a process or program was received by a given patient or it was not. In fact a patient might have received all none or some elements of an intervention but the assignment to a group is taken as sufficient evidence that the intervention was (or was not) delivered. As noted above evaluation researchers in other fields have moved beyond this approach recognizing that it is essential to observe and thus illuminate the ‘black box’ of the process or intervention to assess variations in implementation and if possible to identify the ‘active ingredients’ that really make a difference. The full range of qualitative methods but in particular observations can be used to great advantage to achieve these objectives. Of course when we know enough about a process to specify it very carefully and we can operationalize whether or not it has been consistently implemented we can and should move to a more closed-ended and quantifiable inquiry. But all too often we do this prematurely and rarely learn exactly what we should be looking for to determine whether a process has occurred or why a process has not taken place.
In fact identifying and describing specific activities so their consequences can be carefully studied and so the activities themselves can be well replicated is a research area ripe for enhancement through qualitative methods. Some work is of course already going on. For example, two studies supported by The Commonwealth Fund tried to work backwards from the quantitative CAHPS® and HEDIS® measures to identify best practices in health plans that were associated with high scores. These studies both used qualitative field research methods in particular key informant interviews. One study visited plans that received both especially high and especially low scores on the CAHPS® Medicare Managed Care Survey [38]. As the authors state "We did not find a one-to-one correspondence between specific CAHPS® scores and discretionary plan-level practices that "explained" those scores... However we identified several discretionary practices that contributed to improved or sustained performance within each CAHPS® dimension. These practices affected the quality of members’ experiences with benefits and services administered directly by the plan as well as with clinicians and clinicians’ offices.’
In the other study investigators visited plans that performed well on six HEDIS® indicators of clinical effectiveness [39]; they interviewed key informants with multiple perspectives and collected and analyzed a wide range of documents. The authors note ‘...We find that factors that contribute to high clinical performance transcend the specific improvement activities or programs. Delivering high-quality care is a primary driving force for nearly all the high-performing plans we studied. There is a culture of respect for clinicians in these plans. The plans view their role as assisting clinicians to perform clinical processes better and use clinicians as important resources for accomplishing quality improvement. The plans also invest in and use clinical data often and appropriately both to manage clinical improvement and to demonstrate it.’ While these studies made good use of qualitative methods they might have been enhanced by the use of additional focus groups with plan members as well as if possible given confidentiality concerns direct observations of the delivery of care. This is a form of methodological ‘triangulation’ often recommended by qualitative researchers either among different qualitative methods or between qualitative and quantitative approaches.
Qualitative methods can also be used to help plans and providers move from the relatively general information about their performance that is captured in CAHPS® and HEDIS® measures to more specific information about what they can do to improve their performance. For example if information were made available to plans so they could identify a profile of members who are particularly dissatisfied they could use that profile to recruit people into focus groups to probe into the details of member expectations and experiences as well as to obtain their advice about how to make their experiences better. Plans and providers can also use focus groups and one on one interviews to ‘reality test’ improvement strategies to see whether they make sense both to clients and to front line staff. In summary the study of how high-quality care is delivered could be greatly enhanced by more extensive use of qualitative methods.
Key factors to improve the use of qualitative methods
When qualitative research is designed or executed poorly the results are neither credible nor useful. Given the skepticism that many researchers have about the validity and reliability of the methods it is critical that the challenges they pose to unwary or careless investigators be understood. Firstly it is important to realize that these methods probably to a greater degree than quantitative approaches are highly dependent on the knowledge and skills of all members of the research team. A focus group will only succeed if it is based on a well-structured guide and conducted by a skilled moderator flexible enough to work both through and around the structure. A key informant interview will harvest much less useful information if the interviewer does not have enough background knowledge to know when a response deserves extensive probing or sufficient experience to assess either the candor of the respondent or the extent to which s/he actually has knowledge and experience relevant to the questions. A novice interviewer needs training which can best be provided by having him or her accompany an experienced interviewer in the field or listen carefully probably multiple times to tapes of effectively conducted interviews.
Secondly conducting a good qualitative study like conducting a good quantitative project requires attention to issues of study design the unit of analysis sampling techniques instrument development and administration and analysis plans. Are you looking for developments over time? Then you need a study designed to collect data at multiple points in time rather than depending on recollection. Are you studying the quality of interactions between providers and patients? Then you need to think through whether your unit of analysis is the provider or the interaction or how to deal with the nesting of interactions within providers.
If you are conducting a focus group study sampling will be critical. You need to define carefully the inclusion and exclusion criteria for participants and the dimensions on which you want a particular group to be homogeneous and heterogeneous. For example in studying family members and friends of people with Medicare who help them make health care decisions we conducted eight groups [40]. It took considerable thought to develop a screening tool that would discriminate individuals who were actually helping with or were actually making health care decisions. This was an inclusion criterion for participants in all eight groups and helped ensure that we did not waste resources talking with people who had nothing meaningful to share or create a situation in the group in which the discussion of extraneous issues would have to be continuously discouraged. However the groups were evenly divided by market to explore differences in experience and attitude between markets with high and longstanding as opposed to new and limited managed care penetration. Within each market we recruited into one group only helpers who were themselves on Medicare into another group only people who actually made decisions for their friend or relative and then divided the remaining two groups by educational level. In all groups we sought a mix of ethnicities and gender. This complex design helped us explore both shared and distinctive experiences.
Instrumentation is as critical in qualitative as in quantitative research. It takes training and practice to write open-ended questions the hallmark of a qualitative interview and then to keep from transforming them into closed-ended questions especially with a resistant subject when actually conducting the interview. Few individuals have experience in developing and using observation protocols or in conducting and writing up the notes from structured observations. Observation protocols should make explicit the particular dimensions of an interaction (e.g. a staff meeting) or an event (e.g. a training session) that are of interest. For example if you are observing a meeting you typically want to know who was expected to attend who actually showed up whether there was a leader and who it was whether there was an agenda and whether it was followed what roles the leader and other members took on who asked questions and who answered them who requested input and who provided it how and by whom it was determined that a decision needed to be made whether how and by whom decisions were made whether conflict arose and what kind whether the conflict was acknowledged by others whether and how the conflict was resolved and so on. It is often a good idea to draw a map of the meeting showing who sat where if for no other reason than to help visualize the experience afterwards. It is also useful to note aspects of the physical environment (e.g. sound level temperature density of the room) that might influence the process. And then in addition to paying attention to meeting dynamics the observer will also have to follow and take notes on the actual content of the discussion. Clearly the use of this technique requires skills experience and endurance.
Many might wonder whether it is not inevitable as well as problematic for the presence of an observer to change the way the meeting or event occurs in essentially unknowable ways. This is more or less likely depending on circumstances and skill. In terms of circumstances for example if you are the only woman or person of color at an event it is harder to remain unnoticed. It is perhaps surprising however that even if participants appear at the outset to be somewhat stiff or on what they think is supposed to be their best behavior they often forget about the observer fairly quickly unless they are completely bored and disengaged from the meeting (which is also interesting data). To be as unobtrusive as possible the observer has to melt into the background and keep a straight face showing no reaction to the events since that might indeed generate an unnatural response.
Data analysis
The analysis of qualitative data is probably the most challenging aspect of the use of these methods. On the one hand it is not appropriate to treat qualitative data in a quantitative manner. Statements made by focus group participants should not be counted and displayed in a table. In addition to being at odds with the underlying principles of the method these presentations can lead people to assume incorrectly that the data are generalizable to similar populations which is simply not the case. Data from qualitative research are typically suggestive rarely if ever conclusive. Nevertheless the analysis process should be highly deliberate and systematic. The temptation in qualitative work is to simply generate impressions based on an initial review of notes or tapes and move quickly to written summaries that blur the distinction between what was observed heard or read (the raw data) and the patterns and themes that an investigator has discerned (which can often best be termed an interpretation of the data). Just as in quantitative work we are careful to present our findings separately from our conclusions in qualitative work we have to distinguish between our observations and our interpretations of those observations. To achieve this the research team needs explicit processes for tracking and managing raw qualitative data for coding the data (not numerically but rather using phrases) in ways that permit data to be looked at both in its textual context and outside of it for checking on the reliability of coding by using multiple coders for recognizing and articulating emergent ideas about patterns themes explanations and hypotheses and for then conducting a conscious search both for ‘rival’ patterns and explanations and for data that in some way disconfirm or refine the patterns and explanations. This is time-consuming and resource intensive but it is a hallmark of respect for the data collected and for the willingness to be proven incorrect that is a key element of being ‘scientific’.
In closing it is important to note that this paper is not describing what many basic social scientists would call pure ethnography [41,42] or grounded theory research [43,44]. The approach taken here is to apply qualitative methods to a highly applied field. Clarity about research questions and the use of a conceptual framework to guide data collection and analysis are not considered inappropriate. The discovery-oriented character of qualitative methods can persist even when conducting research in a systematic and purposeful manner. The ‘openness’ in more structured qualitative research pertains not to what you want to learn but rather to what you actually do learn. The best qualitative researchers remain wide open to surprises. Some even measure the effectiveness of their research by how much surprise they experience when they go into the field.
Glossary
A Managed Care Plan is an organization that provides health care to an enrolled population in return for pre-set monthly premium payments. Most managed care plans provide care through a specified network of doctors hospitals and other medical professionals. Coverage is reduced or eliminated when members use providers of care not included in the network. Care is generally coordinated by a primary care physician (‘gatekeeper’) who makes referrals to network specialists hospitals and other service providers.
Medicaid and Medicare are two health care financing programs in the United States that were established in 1965 as amendments to the Social Security Act.
Medicaid is a jointly funded Federal-State health insurance program that covers approximately 36 million low-income and needy people.
Medicare is a Federal health insurance program that covers people aged 65 and over as well as those who have permanent kidney failure and certain people with long-term disabilities.
Medigap is the name given to a form of supplemental insurance that is designed to fill some of the gaps in coverage left by Medicare. It is sold by private insurance companies not the US government and is available to those who are eligible for Medicare.
Address reprint requests to Shoshanna Sofaer School of Public Affairs Baruch College 17 Lexington Avenue Box C-401 New York NY 10010 USA. E-mail: shoshannasofaer{at}baruch.cuny.edu 
Author’s Note. A brief glossary is appended of a number of terms specific to the US health care system that may not be familiar to all readers.
Received for publication . Accepted for publication April 17, 2002.
REFERENCES
- Hurley RE. Qualitative research and the profound grasp of the obvious. Special Supplement on Qualitative Methods in Health Services Research, December 1999, Part II. Health Serv Res 1999; 34: 1119–1136.[Web of Science][Medline]
- Ragin CC. The distinctiveness of case-oriented research. Special Supplement on Qualitative Methods in Health Services Research, December 1999, Part II. Health Serv Res 1999; 34: 1137–1151.[Web of Science][Medline]
- Sofaer S. Qualitative research methods: what are they and why use them? Special Supplement on Qualitative Methods in Health Services Research, December 1999, Part II. Health Serv Res 1999; 34:1101–1118.[Web of Science][Medline]
- Yin RK. Enhancing the quality of case studies in health services research. Special Supplement on Qualitative Methods in Health Services Research, December 1999, Part II. Health Serv Res 1999; 34: 1209–1224.[Web of Science][Medline]
- Devers KJ. How will we know ‘good’ qualitative research when we see it? Beginning the dialogue in health services research. Special Supplement on Qualitative Methods in Health Services Research, December 1999, Part II. Health Serv Res 1999; 34: 1153–1188.[Web of Science][Medline]
- Patton MQ. Enhancing the quality and credibility of qualitative analysis. Special Supplement on Qualitative Methods in Health Services Research, December 1999, Part II. Health Serv Res 1999; 34: 1189–1208.[Web of Science][Medline]
- Shortell SM. The emergence of qualitative methods in health services research. Special Supplement on Qualitative Methods in Health Services Research, December 1999, Part II. Health Serv Res 1999; 34: 1083–1090.[Web of Science][Medline]
- Edmunds H. (American Marketing Association). The Focus Group Research Handbook. Lincolnwood, IL: NTC Publishing Group, 2000.
- Templeton JF. Focus Group: A Strategic Guide to Organizing Conducting and Analyzing the Focus Group Interview. Chicago, IL: The McGraw-Hill Companies 1996.
- Bloor M, Frankland J, Thomas M, Robson K. Focus Groups in Social Research. Thousand Oaks, CA: Sage Publications, 2001.
- Krueger RA, Casey MA. Focus Groups: A Practical Guide for Applied Research. Thousand Oaks, CA: Sage Publications, 2000.
- Morgan DL. Focus Groups as Qualitative Research (Second edition). Thousand Oaks, CA: Sage Publications, 1998.
- Morgan DL. Successful Focus Groups: Advancing the State of the Art. Thousand Oaks, CA: Sage Publications, 1993.
- Harris-Kojetin LD, Fowler FJ Jr, Brown JA, Schnaier JA, Sweeny SF. The use of cognitive testing to develop and evaluate CAHPS 1.0 core survey items. Med Care 1999; 37(Suppl. 3): MS10–21.[Web of Science][Medline]
- Weidmer B, Brown J, Garcia L. Translating the CAHPS 1.0 survey instruments into Spanish. Med Care 1999; 37(Suppl. 3): MS89–96.[Web of Science][Medline]
- Brown JA, Nederend SE, Hays RD, Short PF, Farley DO. Special issues in assessing care of Medicaid recipients. Med Care 1999; 37(Suppl. 3): MS79–88.[Web of Science][Medline]
- Schnaier JA, Sweeny SF, Williams VS et al. Special issues addressed in the CAHPS survey of Medicare managed care beneficiaries. Med Care 1999; 37(Suppl. 3): MS69–78.[Medline]
- Homer CJ, Fowler FJ, Gallagher PM et al. The Consumer Assessment of Health Plans Study (CAHPS) survey of children’s health care. Jt Comm J Qual Improv 1999; 25: 369–377.[Medline]
- Eisen SV, Shaul JA, Clarridge B, Nelso D, Spink J, Cleary PD. Development of a consumer survey for behavioral health services. Psychiatr Ser 1999; 50: 793–798.
- Crofton C, Lubalin JS, Darby C. Consumer Assessment of Health Plans Study (CAHPS). Foreword. Med Care 1999; 37(Suppl. 3): MS1–9.[Web of Science][Medline]
- Agency for Healthcare Research and Quality. Frequently asked questions: the CAHPS reports. http://www.ahrqlgov/qual/cahps/faqrepor.htm. Accessed March 2002.
- Sudman S, Schwarz N, Bradburn NM. Thinking About Answers: The Application of Cognitive Processes to Survey Methodolog. San Francisco, CA: Jossey-Bass, 1995.
- The MEDSTAT Group USA: http://www.medstat.com. Accessed 15 April 2002.
- McGee J, Sofaer S, Kreling B. Final Report: Findings from Focus Groups Conducted for the National Committee for Quality Assurance Medicare and Medicaid Consumer Information Projects, July, 1996.
- Foundation for Accountability USA: http://www.facct.org. Accessed 15 April 2002.
- Hibbard JH, Sofaer S, Jewett JJ. Condition-specific performance information: assessing salience comprehension and approaches for communicating quality. Health Care Financ Rev 1996; 18: 95–109.[Web of Science][Medline]
- Goldstein E, Fyock J. Reporting of CAHPS quality information to Medicare beneficiaries. Health Serv Res 2001; 36: 477–488.[Web of Science][Medline]
- Hibbard JH, Jewett JJ. What type of quality information do consumers want in a health care report card? Med Care Res Rev 1996; 53: 28–47.
- Jewett JJ, Hibbard JH. Comprehension of quality indicators: differences among privately insured publicly insured and uninsured. Health Care Financ Rev 1996; 18: 75–94.[Web of Science][Medline]
- Kreling B, Sofaer S. Medicare Consumer Information Project: A Report on Focus Group. Report to The Commonwealth Fund School of Public Affairs Baruch College May 2001.
- McGee J, Kanouse DE, Sofaer S, Hargraves L, Kleimann S, Hoy EW. Making survey results easy to report to consumers: how reporting needs guided survey development in CAHPS. Med Care 1999; 37: MS32–MS40.[Web of Science][Medline]
- Dewart T, Sofaer S. Report on Cognitive Tests of Medicare HMO Materials with People on Medicare and Medicare Information Intermediaries in New York City. Report to The Commonwealth Fund School of Public Affairs Baruch College May 2000.
- Patton MQ. Qualitative Research & Evaluation Methods (Third edition). Thousand Oaks, CA: Sage Publications, 2001.
- Carman KL, Short PF, Farley DO, Schnaier JA, Elliott DB, Gallagher PM. Epilogue: early lessons from CAHPS demonstrations and evaluations. Med Care 1999; 37(Suppl. 3): MS97–105.[Medline]
- Scanlon DP, Darby C, Rolph E, Doty HE. The role of performance measures for improving quality in managed care organizations. Health Serv Res 2001; 36: 619–641.[Web of Science][Medline]
- Smith F, Gerteis M, Downey N, Lewy J, Edgman-Levitan S. The effects of disseminating performance data to health plans: results of qualitative research with the Medicare Managed Care plans. Health Serv Res 2001; 36: 643–663.[Medline]
- Bentley JM, Nash DB. How Pennsylvania hospitals have responded to publicly released reports on coronary artery bypass graft surgery. Jt Comm J Qual Improv 1998; 24: 40–49.[Medline]
- Gerteis M, Harrison T, James CV, Manocchia M, Smith F, Edgman-Levitan S. Getting Behind the Numbers: Understanding Patients’ Assessments of Managed Care. New York: The Commonwealth Fund, 2000.
- Felt-Lisk S, Kleinman L. Effective Clinical Practices in Managed Care: Findings from Ten Case Studies. New York: The Commonwealth Fund 2000.
- Sofaer S, Kreling B, Kenney E, Swift E, Dewart T. Family members and friends who help people on Medicare make health decisions. Health Care Financ Rev 2001; 23: 105–121.
- Atkinson P, Hammersley M. Ethnography: Principles in Practice (Second edition). London, UK: Routledge 1995.
- Denzin NK. Interpretive Ethnography. Thousand Oaks, CA: Sage Publications, 1996.
- Glaser BG, Strauss AL. The Discovery of Grounded Theory. New York, NY: Aldine de Gruyter, 1967.
- Strauss AL, Corbin JM. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (Second edition). Thousand Oaks, CA: Sage Publications, 1998.
- Bickman L, Rog DJ (eds). Handbook of Applied Social Research Methods. Thousand Oaks, CA: Sage Publications 1998.
- Strauss A, Corbin J. Basics of Qualitative Research (Second edition). Thousand Oaks, CA: Sage Publications 1998.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  P Hudelson, A Cleopas, V Kolly, P Chopard, and T Perneger What is quality and how is it achieved? Practitioners' views versus quality models Qual. Saf. Health Care, February 1, 2008; 17(1): 31 - 36. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. Tong, P. Sainsbury, and J. Craig Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups Int. J. Qual. Health Care, December 1, 2007; 19(6): 349 - 357. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. Bolivar-Munoz, A. Daponte-Codina, N. Pascual-Martinez, F. Barranco-Ruiz, J. Sanchez-Cruz, C. Martin-Castro, and E. Gil-Pinero Use of emergency transport by patients with cardiopathies: a focus group study Int. J. Qual. Health Care, December 1, 2007; 19(6): 407 - 413. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. J. Hern, T. A. Beery, and D. G. Barry Experiences of College-Age Youths in Families With a Recessive Genetic Condition Journal of Family Nursing, May 1, 2006; 12(2): 119 - 142. [Abstract] [PDF]
|
|
|
|
 |
|
 D. H. Fernald and C. W. Duclos Enhance Your Team-Based Qualitative Research Ann. Fam. Med, July 1, 2005; 3(4): 360 - 364. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||