Skip Navigation


International Journal for Quality in Health Care Advance Access originally published online on July 21, 2005
International Journal for Quality in Health Care 2005 17(6):497-504; doi:10.1093/intqhc/mzi063
This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
17/6/497    most recent
mzi063v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrow Search for citing articles in:
ISI Web of Science (1)
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Fickel, J. J.
Right arrow Articles by Thrush, C. R.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Fickel, J. J.
Right arrow Articles by Thrush, C. R.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?

International Journal for Quality in Health Care vol. 17 no. 6 © The Author 2005. Published by Oxford University Press on behalf of International Society for Quality in Health Care; all rights reserved

Policymaker use of quality of care information

Jacqueline J. Fickel1 and Carol R. Thrush2

1 Department of Psychiatry, University of Arkansas for Medical Sciences, and 2 Veterans Administration Health Services Research and Development, Center for Mental Healthcare and Outcomes Research, Little Rock, AR, USA

Objective. This study describes public policymakers’ experiences with the feasibility of using information from quality of care assessment activities. The objective is to improve the ability to match quality evaluation tools with policymakers’ information needs.

Design. US state administrative policymakers were interviewed about use of quality of care information and knowledge, attitudes, and experiences with information from specific types of measures.

Participants. A purposive sample of 82 key informants from Medicaid program administrations in 48 states.

Main measures. Users of information from each of eight targeted types of quality of care measurement methods were compared with non-users based upon their levels of knowledge, perceived characteristics of quality of care information, and perceived characteristics of the policy situation.

Results. Participants indicated that some types of quality measurement methods have been useful, whereas others have not. Extent of quality assessment information use, and the measurement methods utilized, varied widely. Two factors were associated with the use of information from particular quality assessment methods: information needs of the policymakers and their perceptions of the characteristics, including strengths and weaknesses, of particular measurement methods.

Conclusions. These policymakers had positive attitudes about quality assessment, were knowledgeable about types of methods, and had a variety of potential uses for quality-related information. Yet, perceptions and experiences with different types of measurement methods varied. We describe a set of quality assessment methods with complementary characteristics that could provide a relatively inclusive picture of quality of care and better address policymaker information needs.

Keywords: healthcare quality improvement, managed care, policy making, quality information

Address reprint requests to Jacqueline J. Fickel, University of Arkansas for Medical Sciences, 4301 West Markham Street (Slot 755), Little Rock, AR 72205, USA. E-mail: jjfickel{at}uams.edu

Accepted for publication May 29, 2005.


   Introduction
Top
 Introduction
Methods
 Results
 Discussion
 Limitations
 Conclusion
 References
 
Public policymakers are healthcare stakeholders who can use information about quality of care. It is still uncertain, however, whether quality assessment activities are translating into action at the program and system level [1]. Although there are a number of types of quality of care assessment methods that they may use [2,3], evidence is lacking on how well existing measures match policymakers’ information needs.

State Medicaid programs in the United States provide an example of a setting where policymakers may use quality of care assessment information. These publicly-funded programs finance health care for people in low-income, high-risk populations, such as children and pregnant women. State Medicaid agencies have many options within the federal guidelines [4], and the type and extent of quality evaluation activities vary a great deal between states [5–9]. Most agencies operate part or all of their programs via a managed care model and require collection of data from their contractors for various quality indicators [5,10].

Medicaid agencies fulfill various roles and functions [11,12], and administrators have many possible uses for quality of care information. In operating programs, activities include monitoring delivery of services and reporting. As purchasers of managed care, they perform contracting, monitoring, and feedback functions and are concerned with value of health care benefits. As regulators, they are responsible for enforcement of federal and state program regulations. And as stakeholders in their state health systems, they have an interest in public health intervention activities. In these activities, administrators face challenges similar to policymakers from many other settings: obtaining reliable, relevant, timely information about quality of care and whether to use such information.

There is little published research on policymaker use of information about health care quality or their knowledge and attitudes about available measures [12,13]. Most studies to date have dealt with consumers or employers who purchase health benefits and suggest that actual use of information about quality of care is limited [14–18]. Studies that have described quality assessment activities in Medicaid at the agency level have not focused on the choice of particular quality assessment methods or on policymaker knowledge, attitudes, and practices [6–9].

Questions addressed by this study are meant to explore and clarify relationships between quality of care assessment activities conducted in managed care programs, administrators’ use of the resulting information, and factors conceptualized to influence that use. Ultimately, our findings are intended to improve the ability to match quality evaluation tools to policymaker information needs.


   Methods
Top
 Introduction
 Methods
Results
 Discussion
 Limitations
 Conclusion
 References
 
Design and participants
We conducted semi-structured interviews with a national cross-section of US state Medicaid administrative policymakers, using both open- and close-ended questions. This design is suited to capturing the full range of knowledge and perceptions about quality of care assessment methods and to exploring attitudes and experiences in sufficient depth to clarify the factors associated with information use.

Eligible participants were identified based on their positions: program directors, managed care directors, and directors of quality assessment. These positions are key management roles relative to collecting and using quality of care assessment information and are common to most of the 49 state agencies that were using managed care models [10]. We identified 138 potential informants using the National Association of State Medicaid Directors membership list [19], information from each agency’s internet site, and preliminary phone calls.

Data collection
An experienced investigator (JF) conducted interviews from March through September 2001. Interviews lasted an average of 45 minutes and were recorded with notes on a standardized interview guide form. Topics covered in the interviews are listed in Table 1 and produced two types of data: quantitative ratings and qualitative statements. We used the qualitative information to enrich our understanding of the patterns noted in the quantitative analysis.


View this table:
[in this window]
[in a new window]
  		Table 1 Interview topics and types of data gathered for study

 

Factors conceptualized to potentially influence administrator use of information about quality were synthesized from existing literature. These factors paralleled those observed to influence use of quality information by employers [17,18,20] and consumers [14,15,21] and policymaker use of other types of information [22–24]. They fell into three categories: knowledge of quality information, perceptions about characteristics of quality measurement information, and perceptions of the policy situation and environment.

We assessed knowledge through both stated familiarity and comprehension, using methods similar to Hibbard et al. [25] in which participants’ statements of what the measure is intended to mean were compared with the definition as established in the literature. Participants identified whether they were familiar with a listing of 22 types of quality measurement methods (Table 2), selected based on published frequency of collection by state Medicaid agencies [2,5,11]. Eight of these (Table 2) were targeted for more detailed questioning, based on greatest frequency of previously-reported collection [5]. When data for a familiar, targeted measurement method were being collected by the participant’s agency, the participant was asked to rate his or her perception of its usefulness for quality of care assessment on a Likert scale of 1 (not at all) to 5 (very good) [25]. They rated the policy situation and environment on five-point Likert scales as well. Participants were identified as ‘users’ or ‘non-users’ of information from a particular measure based upon self-report, plus supporting description of the ways in which they used the information.


View this table:
[in this window]
[in a new window]
  		Table 2 Respondents use of information from quality of care measures and ratings of overall usefulness for targeted quality measures

 

Analysis
Participants who used information from a particular measure were compared with those who did not, based upon each of the key factors identified from the literature (Table 1), plus relevant characteristics of the individual participants and of the state Medicaid programs, included because of their face validity. For variables that were categorical in nature, Chi-square tests were used to compare groups. Point-biserial correlations were used when variables were continuous. Power analysis indicated 90% power for moderate (0.5) effects using a two-sided test at an alpha level of 0.05 with a sample size of at least 45. Qualitative statements were coded by the interviewer for identification of the key topics according to an a priori coding scheme and examined for patterns and themes. In addition, a systematic post hoc analysis of the qualitative statements was conducted to identify unanticipated themes.


   Results
Top
 Introduction
 Methods
 Results
Discussion
 Limitations
 Conclusion
 References
 
Administrators, their knowledge, and attitudes
Eighty-two individuals completed interviews, a 59% response rate. Fourteen held positions as state Medicaid directors, 37 were directors of Medicaid managed care programs, and 31 were managers over quality of care activities. Participants had their positions for an average of 4 years, ranging 2 months to 18 years. They worked in their agencies for an average of nine years, ranging 2 months to 30 years. They worked in 48 state Medicaid agencies; 26 states had two or three respondents each.

Participants had a variety of responsibilities. In addition to general administrative duties for Medicaid Programs, typical responsibilities included program operations, contract procurement and monitoring, and coordination of quality assessment and improvement programs. Many had multiple areas of responsibility, with profiles reflecting the size and structure of each agency. All but one reported dealing with quality of care in their jobs; 55% did so for at least half their activities.

Most participants reported positive attitudes about the importance of assessing quality of care. Sixty-four (78%) stated that it was either very important or extremely important and described their reasons. Moreover, there was a high rate of familiarity with and understanding of the types of measurement methods. At least 80% stated that they were familiar with 20 of the 22 types included in the interview protocol. For the eight targeted types, over half of the respondents who completed ratings had partial or adequate levels of understanding of what that type of measure was supposed to tell them.

Use of quality of care information
Most participants reported that they use information from their programs’ quality of care assessment activities. However, the extent of use and the measurement methods chosen were far from uniform. Eighty percent of participants described using information from several (two to six) different types of quality of care measures. They relied most heavily on a few of the different types (Table 2), especially enrollee satisfaction surveys (67%) and utilization/encounter data (59%). Most of the eight targeted types were rated as useful in assessing quality of care by the majority of participants who were familiar with them and had the opportunity to use information from them in their jobs (Table 2). Administrators reported using information about quality of care in several ways in their jobs (Table 3). Many described specific examples of how they and their agencies are using this information. Selected examples are presented in Table 4.


View this table:
[in this window]
[in a new window]
  		Table 3 How respondents reported using targeted types of quality of care information, n = 82

 

View this table:
[in this window]
[in a new window]
  		Table 4 Examples of how respondents and their agencies are using quality of care information

 

Associations with use
Significant positive associations (P < 0.05) were noted between use of five types of measures and the purposes for which participants used quality of care information. Different profiles of uses were associated with each measure, the largest variety with enrollee satisfaction surveys, including managed care organization (MCO) contract monitoring, reporting to other governmental entities, to enrollees, and to the public. Use of information from reviews of quality improvement programs was associated with MCO contract decisions and with monitoring. Use of information from HEDIS measures was associated with MCO contract decisions, and use of information from both focused clinical studies and health outcomes measures was associated with providing feedback to MCOs. Unsurprisingly, significant positive associations (P < 0.05) were also noted between use of six types of measures and participants’ ratings of overall usefulness (Table 2).

There were no statistical associations between use of any of the eight targeted methods and participants’ familiarity or understanding, nor with ratings of the nature of the issues or the policy environment. There were also no significant patterns of associations with the descriptive characteristics of the individual participants, or the states and Medicaid programs.

Strengths and weaknesses of types of measures
Participants who rated usefulness were probed for their perceptions of the particular strong points or problems and their experiences in attempting to use the information. Many participants described actions and situations which were helpful or problematic for their agencies in producing useful information. A summary of the most commonly-mentioned strengths and problems is presented in Table 5. Some of these were seen as inherent with the particular type of measure, others as related to implementation issues. These patterns of strengths and weaknesses are related to one of the themes that emerged from the qualitative data: that using several different measures in conjunction is helpful in getting a more complete picture of quality of care.


View this table:
[in this window]
[in a new window]
  		Table 5 Respondents’ perceptions of problems and strengths of targeted quality care measures

 


   Discussion
Top
 Introduction
 Methods
 Results
 Discussion
Limitations
 Conclusion
 References
 
Although state Medicaid administrative policymakers were interested in using quality of care assessment information in a variety of applications, each type of measure was noted to have its own strengths and weaknesses. The factors that were related to information use—the purposes for which information is needed and perceptions of strengths and weaknesses of specific measures—may assist in developing a framework for understanding how to match quality of care activities with policymaker needs. The most helpful methods would be those which would provide for the full complement of policymakers’ information needs while taking advantage of a variety of operational characteristics.

Using multiple measures in an integrated approach
One of the main themes raised by participants who had positive experiences with quality assessment was that many measures have some utility on their own—but, sets of measures offer a more comprehensive, integrated approach and provide a better picture of quality of care. As one state Medicaid director puts it, ‘If you don’t compare one type of information with other sources, it can be used by plans to show themselves in an unbalanced light, i.e. much better than they are in the big picture.’ This agrees with the recommendations of other researchers, that systems of care which promote quality will need to rely upon multiple strategies, rather than a single approach or fragmented efforts [26–28].

Synthesis of the quantitative and qualitative results of this study suggests that a set of measurement methods for assessing quality of care at the program level would need to satisfy two main considerations. For one, the set would need to meet the specific information needs of the prospective users. In our example of state Medicaid agencies, a set of quality measures would have to help them in their various roles as program operators, purchasers, regulators, and promoters of public health. The other main consideration would be that a set of measures would need to have complementary characteristics across five domains discussed in previous literature and highlighted by policymakers in this study: (i) address both technical and perceptual aspects of care [29,30], (ii) draw upon both administrative and clinical sources of data [31], (iii) include information that can provide quick, ‘red flag’ feedback on issues of interest, as well as longer, in-depth studies [3], (iv) touch on the three measurable aspects of quality of care—structures, processes, and outcomes—and take advantages of linkages between these aspects when possible [27], and (v) encourage the learning and engagement of the MCOs in the quality assessment and improvement process, rather than simple compliance with state-established performance goals. As an example of using this framework, six of the quality of care measurement activities described by participants in this study could fit into an integrated approach to quality assessment and improvement: enrollee satisfaction surveys, enrollee complaint tracking, focused clinical studies, utilization/encounter data, reviews of quality improvement programs, and provider profiling.

Perceptual aspects of quality of care can be assessed through enrollee satisfaction surveys and tracking of complaints. Surveys have the advantages of standardization and sampling, although not of quick turnaround. They also do not show the causes of low satisfaction ratings, nor necessarily discriminate between MCOs. But, they can be useful for program-level monitoring, especially identification of possible problems, and for external reporting. Tracking enrollee complaints is primarily useful as a ‘red flag’ indicator of possible problems (in addition to its basic client service function) and can be a source of information on both overall operations and particular MCOs.

Technical aspects of care may be best reflected by in-depth clinical studies that are focused on a particular population or condition. These studies can include process measures, outcome measures, and compliance with clinical practice guidelines. Turnaround time is long, and the picture is retrospective, but the specificity and depth of information are not available with any other method. In addition, doing the studies may provide organizational learning experiences.

Encounter or claims data can be a relatively quick, administrative source of information. They can be used to calculate utilization rates of particular services that are related to processes of care and in monitoring operations at a program and system level. They can also be used to calculate performance measures related to MCO contracting, monitoring, and feedback, provider profiling and feedback, in regulatory monitoring, and as a source of information for public health initiatives.

Administrative reviews of quality improvement programs and provider profiling are both suited to encouraging organizational learning and quality improvement potential. Reviews of the MCOs’ quality improvement programs can also help agencies with contracting decisions and fulfill regulatory purposes. Some participants noted that on-site reviews as well as administrative document reviews are needed to really understand how the quality improvement structures are working. Several participants mentioned the value of provider profiling for giving professional feedback, providing operational information at the program level, and use in provider contracting and monitoring.

In summary, the findings from this study suggest this example of a set of measures that could provide a relatively inclusive picture of quality of care in Medicaid managed care programs. These six measures provide information with complementary characteristics that can be used for various purposes. And, despite their limitations, they are currently being used with some success in quality of care assessment activities in Medicaid managed care programs.


   Limitations
Top
 Introduction
 Methods
 Results
 Discussion
 Limitations
Conclusion
 References
 
Potential sources of bias in this study were primarily those typical of interview methodologies (e.g. response bias, social desirability bias). To minimize potential response bias, we encouraged involvement of prospective participants who stated that they might not have helpful responses (because they were not necessarily using quality of care information). To minimize potential social desirability bias, interviews were structured to cross-check and follow up ratings and statements as much as possible, and there were opportunities for open-ended statements on all key points in the interview. The range of responses provided by participants, individually and across the entire study group, helped increase confidence in the validity of their responses. Participants tended to speak favorably about some points and methods and less favorably about others; very few were strikingly positive or negative overall. Finally, the specific findings of this study were from US state Medicaid programs. To utilize the recommendation for matching characteristics of a set of measurement methods with information needs in other types of health care programs or systems, further work would be needed.


   Conclusion
Top
 Introduction
 Methods
 Results
 Discussion
 Limitations
 Conclusion
References
 
In conclusion, several types of quality of care assessment tools that are currently available are proving to be useful for the administrative policymakers in this program. However, there are still many challenges, with individual measures and for quality measurement overall. Understanding the relative characteristics of different quality measurement methods, and matching them with the information needs of policymakers in a given setting, should help in identifying sets of measures that produce information which policymakers are able to use and improve the efficiency of quality assessment activities.

The authors thank the administrators who participated in this study, plus Joseph C. Thompson, M.D., J. Greer Sullivan, M.D., and John Fortney, Ph.D., for their helpful comments on this manuscript.


   References
Top
 Introduction
 Methods
 Results
 Discussion
 Limitations
 Conclusion
 References
 

  1. LoebJM. The current state of performance measurement in health care. Int J Qual Health Care 2004; 16 (suppl. 1): i5–i9.

  2. McGlynnEA. Choosing and evaluating clinical performance measures. Jt Comm J Qual Improv 1998; 24: 470–479.[Medline]

  3. EddyDM. Performance measurement: problems and solutions. Health Aff 1998; 17: 7–25.[Abstract]

  4. Health Care Finance Administration, Medicaid Bureau. A Health Care Quality Improvement System for Medicaid Managed Care. Washington, DC: U.S. Department of Health and Human Services, 1993.

  5. National Academy for State Health Policy. Medicaid Managed Care: A Guide for States. Portland, Maine: NASHP, 1999.

  6. LandonB, Tobias C, Epstein A. Quality management by state Medicaid agencies converting to managed care: plans and current practice. JAMA 1998; 279: 211–216.[Abstract/Free Full Text]

  7. LandonBE, Huskamp HA, Tobias C, Epstein AM. The evolution of quality management in state Medicaid agencies: a national survey of states with comprehensive managed care programs. Jt Comm J Qual Improv 2002; 28: 72–82.[Medline]

  8. LandonBE, Schneider EC, Tobias C, Epstein AM. The evolution of quality management in Medicaid managed care. Health Aff 2004; 23 (4): 245–254.

  9. SchneiderEC, Landon BE, Tobias C, Epstein AM. Quality oversight in Medicaid primary care case management programs. Health Aff2004; 23 (6): 235–242.

  10. Centers for Medicare & Medicaid Services (CMS) of the U. S. Department of Health and Human Services. National Summary of State Medicaid Managed Care Programs: Program Descriptions as of June 30, 1998: http://www.cms.hhs.gov/medicaid/managedcare/nsglos98.asp Accessed 13 June 2005.

  11. BoothM, Riley T, Mitchell E. Building Quality in Medicaid Managed Care: What Policymakers Need to Know (and Do). Portland, Maine: NASHP, 1998.

  12. RileyT. The role of states in accountability for quality. Health Aff 1997; 16: 41–43.[CrossRef][Medline]

  13. BrennanTA. The role of regulation in quality improvement. Milbank Q 1998; 76: 709–731.[Medline]

  14. RobinsonS, Brodie M. Understanding the quality challenge for health consumers: the Kaiser/AHCPR study. Jt Comm J Qual Improv 1997; 23: 239–244.[Medline]

  15. FarleyDO, Short PF, Elliott MN, Kanouse DE, Brown JA, Hays RD. Effects of CAHPS health plan performance information on plan choices by New Jersey Medicaid beneficiaries. Health Serv Res 2002; 37: 985–1007.[CrossRef][Web of Science][Medline]

  16. GabelJ, Hunt KA, Hurst K. When Employers Choose Health Plans: Do NCQA Accreditation and HEDIS Data Count? New York: The Commonwealth Fund, 1998.

  17. MaxwellJ, Temin P, Watts C. Corporate health care purchasing among Fortune 500 firms. Health Aff 2001; 20: 181–188.[Free Full Text]

  18. HargravesJL, Trude S. Obstacles to employers’ pursuit of health care quality. Health Aff 2002; 21: 194–200.[Abstract/Free Full Text]

  19. National Association of State Medicaid Directors (NASMD). Membership List: http://www.nasmd.org/members.htm Accessed 1 July 2005.

  20. HibbardJH, Jewett JJ, Legnini MW, Tusler M. Choosing a health plan: do large employers use the data? Health Aff 1997; 16: 172–180.[Abstract]

  21. HibbardJH, Slovic P, Jewett JJ. Informing consumer decisions in health care: implications from decision making research. Milbank Q 1997; 75: 395–417.[CrossRef][Web of Science][Medline]

  22. OhCH. Information searching in governmental bureaucracies: An integrated model. Am Rev Public Adm 1996; 26: 41–70.

  23. FranklinAL, Cawley M, O’Sullivan-Kachel P. Renovations and innovations in program evaluation: the Arizona experience and its potential for more widespread applicability. Public Productivity Manage Rev 1998; 22: 88–106.

  24. GreenbergD, Mandell M, Onstott M. The dissemination and utilization of welfare-to-work experiments in state policymaking. J Policy Anal Manage 2000; 19: 367–382.[CrossRef]

  25. HibbardJH, Sofaer S, Jewett JJ. Condition-specific performance information: assessing salience, comprehension, and approaches for communicating quality. Health Care Finance Rev 1996; 18: 95–109.

  26. DonabedianA. Explorations in Quality Assessment and Monitoring, Vol. II, The Criteria and Standards of Quality. Ann Arbor: Health Administration Press, 1982.

  27. HammermeisterKE, Shroyer AL, Sethi GK, Grover FL. Why it is important to demonstrate linkages between outcomes of care and processes and structures of care. Medical Care 1995; 33: OS5–OS16.[Web of Science][Medline]

  28. AyanianJZ, Quinn TJ. Quality of care for coronary heart disease in two countries. Health Aff 2001; 20: 55–67.[Abstract/Free Full Text]

  29. DonabedianA. Explorations in Quality Assessment and Monitoring, Vol. I, The Definition of Quality and Approaches to its Assessment. Ann Arbor: Health Administration Press, 1980.

  30. Edgman-LevitonS, Cleary PD. What information do consumers want and need? Health Aff 1996; 15: 42–56.[Abstract]

  31. RosenheckR, Fontana A, Stolar M. Assessing quality of care: administrative indicators and clinical outcomes in posttraumatic stress disorder. Medical Care 1999; 37: 180–188.[CrossRef][Medline]


Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us    What's this?



This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
17/6/497    most recent
mzi063v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrow Search for citing articles in:
ISI Web of Science (1)
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Fickel, J. J.
Right arrow Articles by Thrush, C. R.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Fickel, J. J.
Right arrow Articles by Thrush, C. R.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?