International Journal for Quality in Health Care 15:169-177 (2003)
© 2003 International Society for Quality in Health Care
Methodology Matters |
Reporting quality of nursing home care to consumers: the Maryland experience
1Abt Associates Inc., Cambridge, MA
2Maryland Health Care Commission, Baltimore, MD
3Rhode Island Quality Partners, Providence, RI, USA
Objective. To design and implement a reporting system for quality of long-term care to empower consumers and to create incentives for quality improvement. To identify a model to approach this technically and politically difficult task.
Approach. Establishment of a credible and transparent decision process using a public forum. Development of the system based on: (1) review of the literature and existing systems, and discussions with stakeholders about strengths and weaknesses; (2) focus on consumer preferences in the design; and (3) responsiveness to industry concerns in the implementation.
Lessons learned. None of the existing systems appeared to be a suitable model. We decided to develop an entirely new system based on three key design principles that allowed us to tailor the system to consumer needs: (1) designing a decision tool rather than a database; (2) summarizing rather than simplifying information; and (3) accounting for the target audience in the creative execution. Industry concerns focused on the burden of the system, the potential for errors, and the possible communication of a negative impression of the industry. As methodological and data limitations prevented us from resolving those concerns, we addressed them by using cautionary language in the presentation and by making a commitment to incorporate improvements in the future. All stakeholders regarded the final design as an acceptable compromise.
Conclusions. Despite its potentially controversial nature and many methodological challenges, the system has been well received by both the public and the industry. We attribute this success to two key factors: a collaborative decision process, in which all critical design and execution choices were laid out explicitly and debated with stakeholders in a public forum, and realism and honesty regarding the limitations of the system.
Keywords: nursing home, performance measurement, public accountability, public reporting, quality indicators, report cards
The interest in public reporting of quality of care is growing. Most notably, the US federal government is moving towards publicly releasing information on a variety of health care providers, with the recent release of a nursing home reporting system as the first step [1]. Nursing home care offers itself as an area where public reporting may well add value for consumers, as decisions on long-term care can usually be made well in advance. Therefore it is not surprising that, particularly in the US, a variety of governmental, private, and public organizations are currently making data on long-term care facilities publicly available or are in the process of developing reporting systems.
Designing and implementing such reporting systems is a difficult task, both technically, because of the complexity involved in measuring quality of care, and politically, because of the sensitive nature of the information released [2]. Thus, much can be learned from the successful introduction of such systems. This paper describes our approach to the design and implementation of an internet-based performance measurement system for long-term care in the State of Maryland, the Maryland Nursing Home Performance Evaluation Guide, and reflects upon lessons learned in this process. The system covers 
220 nursing homes with 25 000 residents. It was publicly launched on 7 August 2001 and is currently operational and accessible on the internet [3], while the project team continues to incorporate improvements to the system based on public comments and methodological improvements.
Approach to the development of a nursing home performance system
Measuring quality of care is a difficult task, because the concept of quality is multidimensional and the methodology is an evolving field. Communicating quality information to consumers adds further to the complexity, since it requires both condensing complex information into intuitively understandable categories and educating a lay audience about methodologies and their limitations [4,5]. In addition, while the use of performance data for research, policymaking, and internal improvement purposes is now widely accepted, their public reporting is still in its infancy and is met with scepticism, if not outright hostility, by providers [2,6,7]. While this negative stance may be partly explained by unease about new and untested methods, there are also objective downside risks of public performance reporting, such as detrimental effects on staff morale and focus of the media on negative results [9,10].
Keeping these reservations in mind, we did not assume that any final product would escape scrutiny by providers and consumers and be immune to methodological challenges. In addition, we were aware that introducing the system without collaboration with providers would create a controversial climate, which would not be conducive to quality improvement. It therefore became crucial to have a credible and transparent process to yield a product that unites the multiple and partly conflicting demands placed upon it. The elements of this process were:
- discussions with key stakeholders, representing research institutions, consumer groups, government agencies, operational experts, and trade organizations.
- Focus on consumer preferences in the design.
- Responsiveness to industry concerns in the implementation.
Review of existing evaluation systems and evidence
We identified existing systems for nursing home evaluation through an internet search and through a survey of experts (Table 1). Most noticeable was the similarity of these systems’ content in contrast to the wide variety of reporting styles and formats. The content consisted mainly of nursing home inspection data, combined with general information such as a facility locator or links to related websites.
|
Although the existing systems had many features that we judged useful for consumers, such as simplicity in format, language, and navigability, the availability of internal search capabilities, decision-making guides, and links to other relevant web sites, we felt that no single site had successfully united all the elements. In particular, we frequently observed substantial gaps in the sites’ content or flaws in their execution (Table 2). In identifying those limitations, we deliberately took the position of an inexperienced internet user to emulate the situation of the potential target audience. In the absence of a best example to use as a blueprint, we decided to create an entirely new system that would satisfy our demands.
|
In addition to the analysis of existing internet-based evaluation systems, we reviewed the literature on public reporting of quality of care. Important insight was gained from consumer focus groups [11] and a study describing decision-making styles of the elderly seeking long-term care placement [12]. We included reports on lessons learned from the development of evaluation systems for hospital care [13]and health plans [14], as well as publications that describe general design considerations for the elderly [15,16]. Finally, we reviewed the state-of-the-art in measuring nursing home quality and assessed the feasibility of existing performance measures for the reporting system.
Design considerations
Based on our review, we identified specification requirements for the design of a consumer-focused performance report. Three core design principles were adopted. First, the system needed to be designed as a decision tool for consumers rather than a database. While quality of care is the main focus of the web site, other relevant information for selecting a nursing home, such as payment sources or facility services, are provided. The second was to summarize rather than simplify information. Users should be guided through the key decision points in selecting a nursing home, but would be allowed to decide how much information they would like to retrieve at each point, since the presentation of the quality information in a ‘one-size-fits-all’ format is not likely to meet the majority of users’ needs. A single approach might either overwhelm some users or leave others lacking the detail they desire. Layering the level of detail so that users may select the level they wish to access is more likely to meet a greater proportion of users’ interests [5]. For example, the site provides a main page for each nursing home in the state, which shows basic information such as name, address, and a contact phone number. Visitors can then drill down to resident and facility characteristics, deficiency reports, or quality indicator summaries, print a summary report, or obtain a map to locate the facility (Figure 1). Performance scores are presented on a summary page, from which one can drill-down to scores on individual indicators. Layering the information is of particular importance for other potential users of the reporting system, such as discharge planners or social workers, who are able to understand more technical details, or current nursing home residents who want to monitor particular aspects of care in their facility.
|
Finally, the third design principle was to account for the target audience characteristics in the presentation. Taking into consideration that many elderly people using the system might not have much experience with the internet, we strove for simplicity both in site architecture and presentation of content. Elderly-friendly design principles, such as the use of well-readable fonts and type sizes, were followed [15,16].
The internet appeared to be the ideal media solution for the demands put on the system, because it places almost no limits on the amount of information provided so that a variety of material to guide consumers in their decisions about nursing homes can be incorporated. On the other hand, visitors are able to customize what part of the information they want to retrieve through drill-downs and crosslinks. Interactive components, such as error messages or context-sensitive help, support inexperienced users in their search for information. An internet-based system also allows to investigate whether the design principles were properly executed, by encouraging visitors to the site to provide feedback through e-mail and by providing the opportunity for analysing user patterns. Finally, updates both of the underlying data and the content and its presentation are easily and cost-effectively implemented.
Development of the performance scoring system
The development of a performance scoring system can be deconstructed into two steps. The first is the selection of the measures, and the second is the decision how to rate providers on the basis of those measures. For this project, decisions had been made that the system should be based on existing data sources to save the taxpayer and providers the cost and burden of dedicated data collection, and that only established measures should be used.
The available data sources permitted the construction of two types of performance measures. The first type of performance measures are deficiency scores from the deficiency database of the Maryland Office of Health Care Quality (OHCQ) that classify violations of state and federal regulations according to scope and severity. The second are the Quality Indicators (QIs), which are rates of adverse events experienced by nursing home residents. These events are identified from the Minimum Data Set (MDS), an assessment instrument developed and mandated by the US Federal Government [17]. It is a resident-level data set that captures clinical and functional status. MDS data are routinely submitted by nursing homes to the State and are used for a variety of purposes, such as care planning, rate setting, and quality improvement.
Of the large variety of existing MDS-based QIs, we decided to use QIs developed by the University of Wisconsin Center for Health Systems Research and Analysis (CHSRA) [18]. These QIs are commonly regarded as a well established system for performance measurement [19]. A subset of the CHSRA QIs is currently used in the inspection and certification process for participation in Medicare and Medicaid, the US public insurance programs for the elderly and the poor, respectively. Thus, the nursing home industry is accustomed to them and is aware of its performance as measured by them, thereby reducing the burden imposed by the reporting system. In our opinion, these advantages outweighed the major drawback of the CHSRA QIs: their lack of fully developed risk adjustment. We selected a subset of 27 QIs, which is largely identical to the set currently used by the federal government.
The second step in the development of the system was to derive scores based on the selected measures. While the deficiency data have a pre-defined scoring system, the QIs are reported as event rates, which may be difficult to interpret for some consumers [19]. We thus made the decision to score each facility on its QI rates compared with all other facilities in the state, and report those performance categories rather than the actual rates or facility rankings. The main problems with rankings, which are intuitively easy to understand, is that they transform differences of varying magnitude into differences of equal relevance and may therefore convey misleading judgements. In the centre of the data, small and statistically meaningless differences may result in substantially different ranking, whereas large differences in rates may only lead to minimal changes in rank in the tails of the data. By contrast, performance categories group facilities according to a pre-specified cutoff.
For reporting, the 27 QIs were grouped into four categories: clinical, psychosocial, medication prescribing, and functional quality. On an overview page for each nursing home, the number of QIs, on which it was rated in the top, average, or bottom categories, is listed for the four categories separately (Figure 2). Visitors can then drill down to view the rating on each individual QI.
|
Considerations of industry concerns
While the design of the reporting system was tailored primarily to meet the needs of consumers and their intermediaries, a goal of no lesser importance for this public reporting system was to create incentives for providers to improve quality. We were aware that a system that antagonized providers would not succeed in creating a climate conducive to improvement efforts. Consequently, we strove to take into consideration industry concerns and to create a collaborative atmosphere in the implementation phase. Industry representatives were invited to join a Steering Committee that guided choices in the conceptual design phase. Prior to implementing the system publicly, each facility received a pre-release of the information and was asked to comment on data accuracy as well as on conceptual issues. Industry concerns centered mainly on three issues, all of which have been described as common elsewhere [2]. The first was the burden of the reporting system for providers, the second was that the report could potentially communicate incorrect judgements about individual facilities, and the third was that the system could convey an overall negative impression of the industry.
Burden of the reporting system
As with any regulatory intervention, the system inevitably entails a burden for the industry. As current and prospective residents and their intermediaries will be judging facilities on the reported data, facility staff need to be aware of the ratings and be able to respond to performance-related inquiries. Because of the high visibility of the system, improving performance along the published criteria may become a priority target for nursing homes so that resources may be diverted from other, potentially more fruitful, improvement efforts. We tried to minimize the burden by devising a system that overlapped existing reporting requirements so that no additional data collection effort is required. In addition, the QI scoring system bears a resemblance to information that the government is currently feeding back to facilities for their internal use.
Potential for errors
Given that the selected QIs are based on adverse outcomes, there is always the concern that factors other than quality of care, such as casemix, are reflected in them. Since more ill and more functionally impaired residents are at greater risk of experiencing adverse outcomes, even under optimal care, facilities that accept more severe cases could be penalized by the reporting system. While the QIs used in the performance reporting system are to some degree risk-adjusted, e.g. by excluding high-risk patients or stratifying the QI into low- and high-risk subgroups, these adjustment methods are by no means perfect. As the constraints of the project prevented us from developing improved risk adjustment methods, the only option was to acknowledge this information openly. We strove to educate consumers to use this information cautiously and to interpret QI scores in the context of other sources of information, such as their care providers or visits to a facility. This goal was achieved by incorporating cautionary language and explanations in laypersons’ language in the indicator reports. We also made a commitment to the industry that we would stay abreast of methodological advances and would adopt better risk adjustment techniques when they became available.
Negative bias
We were limited in our selection of performance measures to existing instruments. Unfortunately, both the QIs and the deficiency scores focus on negative outcomes, whereas measures for positive outcomes of nursing home care, such as quality of life or resident satisfaction, were not available. Industry representatives voiced concerns that this emphasis on negative results could contribute to an overall negative perception of nursing home care by the public. As with the question of risk adjustment, we had to address these concerns in the design of the reporting system because we were unable to collect primary data or develop new measures for these outcomes. The proposed scoring method assigned at least 90% of the facilities a good or average score on every QI. This relatively generous treatment helped overcome concerns about a potentially punitive character of the reporting system and ensured continuous industry participation in its development.
Conclusions
The Maryland Nursing Home Performance Evaluation Guide constitutes one of the first efforts to develop a public performance reporting system for quality of care that focuses explicitly on consumers’ needs. Since both public reporting of quality and communicating quality to consumers are in their infancy, we believe that the lessons learned during this project provide helpful guidance for future related efforts. In spite of the potentially controversial nature of releasing performance information publicly, the reporting system has largely received positive responses and, with 200 hits on the website per day, can be regarded as a success.
We believe that, in particular, two factors contributed substantially to the success of the project. The first factor was an honest and collaborative decision and implementation process. For all critical design and execution choices, we presented different options with their pros and cons, and debated their relative merits with stakeholders in a public forum. We had decided to give stakeholders a menu of potential approaches rather than to ask for their proposal, since we felt this would overburden such a consensus process. Prior to the public launch of the website, all nursing facilities received a pre-release of their report and were given the opportunity to comment on content and presentation style. In response to their comments, substantial changes were made to correct data errors and to display quality information in a less controversial manner. Limitations of the methods and data used were openly acknowledged, and consumers cautioned against judging a facility only on its indicator scores. The second factor was to tailor content and presentation to consumers as the primary target audience. This implied designing the website as a decision tool rather than a data report, providing a rich context for the data presented, and using elderly-friendly design principles in the execution.
It is our hope that the collaborative nature of the process contributes to stimulating interest in and commitment to the continuous improvement of care delivery in Maryland nursing homes, and thus in better care for residents, as the ultimate measure of success for such a project. And, while the content of our reporting system is certainly very particular to the local situation addressed, we believe that the components that contributed to its success may inform other groups who embark on such a task.
Financial disclosure
During this project, Drs Mattke and Reilly were employees of Abt Associates Inc., a for-profit policy research firm that developed the Maryland Nursing Home Performance Evaluation Guide as a contractor for the State of Maryland. Dr Gifford worked as a consultant to Abt Associates Inc. on the project.
Address reprint requests to Sören Mattke MD, DSc, Organization for Economic Co-operation and Development, OECD/DEELSA/SPD, 2 rue Andre Pascal, 75775 Paris Cedex 16, France. E-mail: soeren.mattke{at}oecd.org 
Accepted for publication December 18, 2002.
References
- Http://www.medicare.gov/NHCompare/home.asp. Accessed 28 November 2002.
- Leatherman S, McCarthy D. Public disclosure of health care performance reports: experience, evidence and issues for policy. Int J Qual Health Care 1999; 11: 93–98.
[Free Full Text] - Http://www.state.md.us/mhcc. Link to Nursing Home Report. Accessed 26 September 2002.
- Hibbard JH. Being strategic with public reports. Int J Qual Health Care 2001; 31: 353–355.
- Jewett JJ, Hibbard JH. Comprehension of quality care indicators: differences among privately insured, publicly insured, and uninsured. Health Care Financ Rev 1996; 18: 75–94.[ISI][Medline]
- Marshall MN, Shekelle PG, Leatherman S, Brook RH. The public release of performance data. What do we expect to gain? A review of the evidence. J Am Med Assoc 2000; 283: 1866–1874.
[Abstract/Free Full Text] - Kassirer JP. The use and abuse of practice profiles. New Engl J Med 1994; 330: 634–636.
[Free Full Text] - Abt Associates Inc. Review of Existing Nursing Home Performance Reporting Systems. Consulting Report. Contract No. MHCC 00-028. October 2, 2000.
- Mebius C. Public disclosure of health care performance reports: comments and reflections from Sweden. Int J Qual Health Care 1999; 11: 102–103.
[Free Full Text] - Collopy BT. Public disclosure of health care performance reports: what do we lose in giving information. Int J Qual Health Care 1999; 11: 99–101.
[Free Full Text] - Seniors Research Group. Phase 2: Final Report No. 5. Nursing Home Compare Web Site Database, 1 September 2000.
- Maloney SK, Finn J, Bloom DL, Andersen J. Personal decision-making styles and long-term care choices. Health Care Financ Rev 1996; 18: 141–155.[ISI][Medline]
- Rogers G, Smith DP. Reporting comparative results from hospital patient surveys. Int J Qual Health Care 1999; 11: 251–259.
[Abstract/Free Full Text] - McGee J, Kanouse DE, Sofaer S, Hargraves JL, Hov E, Kleimann S. Making survey results easy to report to consumers. How reporting needs guided survey design in CAHPSTM. Med Care 1999; 37: MS32–MS40.[CrossRef][ISI][Medline]
- Vastag B. Easing the Elderly Online in Search of Health Information. J Am Med Assoc 2001; 285: 1563–1564.
[Free Full Text] - National Institute on Aging and National Library of Medicine. Making Your Web Site Senior Friendly. A Checklist. February 2001. Http://www.nlm.nih.gov/pubs/checklist.pdf. Accessed 27 June 2002.
- Http://cms.hhs.gov/medicaid/mds20/default.asp. Accessed 27 September 2002.
- CHSRA. PIP/ORYX: QI definitions. June 6, 2001. Http://www.chsra.wisc.edu/CHSRA/PIP_ORYX_LTC/QI_Matrix/main.htm. Accessed 27 June 2002.
- Zimmerman D, Karon S, Arling G et al. Development and testing of nursing home quality indicators. Health Care Financ Rev 1995; 4: 107–127.
- Gibbs DA, Sangl JA, Burrus B. Consumer perspectives on information needs for health plan choice. Health Care Financ Rev 1996; 18: 55–73.[ISI][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  A. S. Groenewoud, MSc, N. J. A. van Exel, MSc, M. Berg, PhD, and R. Huijsman, PhD Building Quality Report Cards for Geriatric Care in The Netherlands: Using Concept Mapping to Identify the Appropriate "Building Blocks" From the Consumer's Perspective Gerontologist, February 1, 2008; 48(1): 79 - 92. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. O'Meara, M. Kitchener, E. Collier, M. Lyons, A. de Billwiller-Kiss, L. P. Simon, and C. Harrington Case Study: Development of and Stakeholder Responses to a Nursing Home Consumer Information System American Journal of Medical Quality, January 1, 2005; 20(1): 40 - 50. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||