What methods do stakeholders prefer for feeding back performance data: a qualitative study in palliative care
Department of Palliative Care and Policy, King’s College, London, UK
Objective. To investigate the opinions of stakeholders (service commissioners and providers) on how performance data should be presented, in order to develop effective feedback methods to facilitate the use of these data in decision making.
Design. A qualitative analysis of semi-structured face-to-face and telephone interviews. League tables and fictional box plots were presented as an illustrative guide. The themes covered in the interviews were the effectiveness of these two feedback formats, their positive and negative characteristics, and ideas for new and improved feedback mechanisms.
Participants. Thirty-six stakeholders representing a range of clinical and non-clinical roles within palliative care and the wider health care system across a variety of statutory and non-statutory organizations from London and the West Midlands.
Results. Box plots were received more positively than league tables, and qualitative information was considered more appropriate than pictorial feedback. Conventional methods such as league tables and box plots were judged to lack essential information on which important decisions could be based, such as additional contextual information and the methodological assumptions of the instrument. Both feedback methods were considered useful as an impetus to further discussion. There was a consensus that feedback should be constructive and able to be adapted to the organizational realities in which UK health services function.
Conclusion. Qualitative research was viewed as the right evidence for gaining an understanding of the quality of end of life care. Stakeholders highlighted the importance of the lay perspective, which requires approaches that illuminate the subjective meanings of patient experience.
Keywords: box plots, feedback methods, league tables, palliative care, performance data, quality of care, stakeholders
Address reprint requests to Marjolein Gysels, Department of Palliative Care and Policy, King’s College, London, UK. E-mail: marjolein.gysels{at}kcl.ac.uk
Accepted for publication June 7, 2004.
Research on the quality of health care has had limited effect on decision making with respect to policy [1–3]. Often the policy process is guided by sources of evidence other than research findings [2,4]. Reliable measures and feedback mechanisms are needed to ensure that research findings are consulted and acted upon. The need to take account of the views of stakeholders on how research evidence should be presented in order to facilitate policy making has been indicated in several publications [5–8]. In this paper, we aim to determine stakeholders’ views on different feedback methods and which features are most helpful for decision making. We focus on the feedback of findings from research and audit studies on the performance of palliative care services in the UK.
In the UK, the dominant system is the National Health Service (NHS), which offers free medical care at the point of delivery to the entire population. Voluntary initiatives exist to meet needs for which the NHS does not provide sufficiently [9]. Modern hospices appeared almost simultaneously with the emergence of the NHS as independent charities in response to inadequacies in the care of the dying [10]. The modern hospice movement and specialist palliative care has grown up in the UK alongside and inside the NHS [10]. Innovations and changes in culture and organizational structures in the NHS have occurred, especially after 1989, when the essentially paternalist service took on a consumerist approach, leading to a more managerial model in the ‘New NHS’. In 2001, another major change was heralded towards a network-based approach. An emphasis on governance and public accountability with a concern to promote evidence, standards, and quality has been evident in both the NHS and palliative care in recent years. These developments form the backdrop of this study’s concern with adequate mechanisms for basing decision making upon evidence.
The findings of this study have wider resonance for other areas of health care, both nationally and internationally, even reaching as far as other public services.
 |
Methods |
|---|
 Top Methods ResultsDiscussionReferences |
|---|
This study is one component of a wider project that is developing clinical and service evaluation tools for palliative care for national adoption [11]. Pilot sites for testing the feasibility and appropriateness of the tools were the West Midlands and London health regions. Representatives of organizations in these health regions were purposively sampled to participate in a semi-structured face-to-face or telephone interview about assessing the quality of end of life care and feeding back these findings. This paper focuses on the insights gained into feedback. Research ethics approval was granted. The interviews took place in the autumn of 2000. The participants represented a range of clinical and non-clinical roles within palliative care and the wider health care system across a variety of statutory and non-statutory organizations. They included general practitioners, directors of public health, local health authority and social services commissioners, specialist palliative care nurses, and chief executives of local and national voluntary hospice and palliative organizations (see Table 1).
|
The researchers presented two systems of feedback to the participants as an illustrative guide: a league table (see Table 2) and a box plot (see Figure 1). They asked them to discuss which format would be most likely to impact on care or change practice. The participants were also asked about ideas or suggestions for alternative ways to give feedback. Participants were already familiar with league tables, but they were shown a box plot and where necessary given some explanation. In the case of telephone interviews, the example of a box plot was faxed or e-mailed.
|
|
Interviews were taped and transcribed, and confidentiality was assured. The accuracy of transcriptions was checked against the original recordings. Data familiarization was achieved through reading and re-reading transcripts, and discussing the issues raised during the interviews within the research team. Themes within the transcripts were identified and coded. Coding aimed to combine previous theoretical knowledge and assumptions with the inductive generation of original concepts. These codes were then sorted for the purposes of interpretation and presentation of the results. The quotes included in this paper are illustrative of the issues raised, but not exhaustive representations of the interviews.
|
Results |
|---|
|
TopMethodsResultsDiscussionReferences |
|---|
Participants
Of the 59 stakeholders approached, 36 were interviewed. Twenty-two stakeholders represented organizations based in London and 14 represented organizations from the West Midlands. Fifteen stakeholders had primarily clinical roles and 21 had non-clinical roles.
League tables
Reasons for using league tables
These were related to transparency and accessibility; they are easy to understand and the data are easy to collate. They were seen as an objective and explicit tool. They were also seen as effective and as a guarantee for action.
Reasons to oppose the use of league tables
There were a variety of objections raised by the opponents of league tables, who tended to be strongly opposed to their use. Opponents did not readily accept that the kind of information that was judged was worth including, and they were critical of how decisions were made to exclude other kinds of information. They contended that this process is politically motivated and they questioned its validity. A frequently cited criticism was the incomparability of the different leagues:
‘I don’t think like is compared with like. You might have, say, six hospices, or six hospitals, and they all produce excellent services but they are one decimal point out. You know, 1.1 to 1.6, but that is of no significance, but it is still at the bottom.’ (Commissioning Manager)‘I think that there’s such a variety of services provided around the country that you can’t really compare a wealthy area with a nice, big teaching hospital with a very poor inner city area that is struggling to keep going.’ (Chief Executive)
League tables were not deemed suitable for representing quality of care. Criticism focused on the emphasis on quantitative data, which the participants thought was incapable of grasping complex phenomena. Also, league tables were seen as giving a wrong and misleading picture of reality, not adapted to the current organization of the NHS.
‘...I don’t know how you measure things like attitude and approach... I think that’s one of the really important things: the kind of approach of the organizers, the philosophy; the approach that staff have with people...’ ( Chief Executive)‘I would strongly recommend not to have league tables, because using quantitative measures for such complex phenomena means that you extrapolate all the assumptions that go into those sorts of figures, into some sort of league table, then you’re just falling into the trap of misleading information.’ [General practitioner (GP)]
A major objection to the use of league tables was that the hierarchical design was an impediment to positive feedback, leading to demotivating and threatening competition among health services.
‘Because most people are working hard to provide a good service, it would be very discouraging to find that you’re at the bottom, when you’ve put in a lot of work.’ (General Manager)‘My concern is that what will happen is that we’ll get driven into targeting specific things, and there’s the danger that people won’t be open about their practice. Some things will become hidden because people don’t want to be blamed. ...I just feel there’s a danger that league tables create competition.’ (Service Manager)
Examples of caveats
League tables were judged useful under certain circumstances, subject to specific restrictions:
‘I think they’re of mixed use. It’s a bit like school league tables, you’re not all starting from the same position. I think they could be useful within one institution, and possibly to compare it to similar institutions, and probably more useful over a period of time.’ (GP)‘[League tables are not useful] because one of the fundamentals of palliative care is to assess and hopefully to respond to the needs and wants of each individual, and those are going to vary considerably. For straightforward things like pain control, yes. But if you’re looking at the subtleties of spiritual care, the thought of being in a league table doesn’t have any advantage.’ (GP)
Box plots
In general, stakeholders received box plots more positively than league tables. Their advantages were seen in the following possibilities they offered:
1. An accessible tool. Box plots were judged to be an easy, accessible tool because they represented information in a pictorial way.
‘It’s visual. You know exactly where you are right away.’ (GP)‘It is not so complicated. I know you have to collate the evidence...but for somebody looking, I think it’s easier to the eye.’ (Nursing Manager)
2. A pretext for further thought and a stimulus to practice. Many participants thought that the box plots would be the start of deeper inquiry into how to improve services, and that they would provide an initial impetus to change shortcomings in the provision of care.
‘I think it’s a very good way of demonstrating to people very simply where they need to focus their service development.’ (Commissioning Coordinator)
3. An anonymous and non-judgmental tool. The stakeholders found the box plots especially useful because the information about quality of care they provided was confined to the institution it was fed back to. The institution could then compare itself with an anonymous average, ensuring confidentiality for the services that scored lower. This was seen as positive as it avoided disapproval and imposition of a sense of failure without the further consideration that league tables entail.
‘It is a comparison with other organizations, but it’s not judgmental. It’s about where you are on the bar. And it’s very clear to demonstrate whether you’re near average or below average, and how far away you are from average.’ (Commissioning Coordinator)
Perceived disadvantages
Although some participants were familiar with box plots, the majority of those interviewed needed an explanation about how box plots work. This was also seen as the most important impediment to their effectiveness.
‘I didn’t understand it. I think it’s quite useful, but it’ll need [laughter] a bit more explanation for idiots like me!’ (Administrator)‘Given some explanatory text, it would be quite helpful to have that sort of information. There needs to be some technical support, explaining what’s meant by the box plot.’ (Director of Public Health)
There was still the apprehension that was seen with regard to league tables. The participants frequently raised the question of validity, or wanted to know more about the criteria on which the presented information was based, or about the context of particular measures.
‘...a smear is good or it’s bad. Pain control isn’t so clear cut...into that diamond goes an awful lot of things... All you need is somebody with severe neuropathic pain and they’re off the scale.’ (Commissioning Manager)‘...we realise that pictorial representations don’t necessarily say everything. ...it’s easier if someone has a surgical operation, then when you ask them sort of: how satisfied were you with the service?’ (GP)
Suggestions for good feedback
When the stakeholders were asked if they had other ideas regarding the forms in which feedback could be given, they did not. They spoke about the government’s initiatives to monitor service delivery, such as performance indicators and starring systems, but they could not offer new ways of giving feedback that would be more appropriate or overcome the limitations of the existing ones. They did, however, have clear ideas about what features an ideal feedback instrument should have. They pointed out what would benefit them, as stakeholders, most.
1. The stakeholders thought that it was important to present feedback in a positive and careful way, suggesting how services could be improved, rather than using the ‘naming and blaming’ approach inherited from previous governments.
‘I wouldn’t have a problem in looking at myself in a comparative light, but I think that we need to present it in a very careful, positive way, because often you’re working with institutions that are feeling quite demoralised already, and to be sent a chart saying that you’re doing really badly is not constructive and helpful. And at the end of the day, the people who tend to suffer from that are the people on the front-line who are delivering the service.’ (Manager)
2. The participants wanted benchmarking, a framework of what is a minimum service. They wanted best practice examples, which would be a source of inspiration. They thought this would lead to useful contacts and the exchange of ideas.
‘...we are looking at benchmarks that...in Social Services that is very much the name of the game, and we look at equivalent areas. To look at how they’ve solved their problems. A sort of benchmark, a framework of what is a minimum service. So we phone them up and say: How is it you’re able to do it? You know, is it, in fact that you’ve got virtually all your staff off on stress sick leave! Because you overwork them? And really you’re not a good model after all! Or is it that you’ve got systems in place that really are very efficient and we should just adopt them?’ (Commissioning Manager)
3. Comparison using box plots was generally preferred above numerically based mechanisms like league tables, and examples were considered even more suitable than comparison.
‘I have worked on performance indicators in the past and I know that the impression of ‘how are we doing compared to the rest?’ has more impact than a numerically based system.’ (Commissioning Manager)‘...a type of person’s individual view of how things went, the bereaved person. So that’s helpful information in a significant event, audit-type environment.’ (GP)
‘And examples of good practice are often better than trying to compare the services with services.’ (Chief Executive)
4. There was overwhelming support for qualitative feedback. Twenty-two of the 36 stakeholders expressed the need for qualitative information. They thought that more personalized and in-depth feedback would help to facilitate clinical and policy decisions. They believed that the description of cases, results of focus groups, or personal accounts from patients or carers would be good evidence against which to weigh their decisions. They found this kind of feedback appealing and thought that these methods carried a great persuasive force. They wanted qualitative information to supplement the box plots, but also in its own right.
‘Those quotes, that’s an extremely powerful way of getting people sitting up straight and thinking: God, I’ve got to do something about this.’ (Commissioning Coordinator)‘What about summaries of patient storytelling? What about key scenarios or whatever? I mean, sometimes they’re just as powerful as all these numbers and things. And they would be [powerful] to the relevant Health Authority.’ (Lead Cancer Nurse)
The stakeholders thought it was not enough to simply make the feedback available to the relevant institutions and leave it at that. They suggested that an active dissemination strategy was essential. Interaction between researchers and decision makers was seen as an especially effective mechanism for reaching informed decisions. It is also important to focus feedback on the service user. Patients should be the centre of the information and therefore it is important to include their views and experiences.
The stakeholders pointed out that their preference for qualitative feedback was motivated by the actual context in which health services were functioning. The feedback instruments had to be capable of reflecting the organizational realities with all their complexities.
‘...it might be that you need to recognize that each individual person who died, will have had a number of different services. And therefore, it may only be possible to evaluate the services corporately, rather than individually. So you might have Macmillan nurses giving advice, you might have GP’s, you might have a palliative care physician call in occasionally, there could be a whole team of people.’ (GP)
The stakeholders thought that palliative care was also best served by qualitative feedback, due to the nature of palliative care itself.
‘...it’s very, very difficult with this particular sort of, palliative care, terminal illness, because the emotions are that much more raw. I think, some sort of involvement at local level of patients and their families is needed.’ (Director of Nursing)
|
Discussion |
|---|
|
TopMethodsResultsDiscussionReferences |
|---|
Measurement of organizational performance, the use of comparative data, minimal standards, and benchmarking are key issues in the management of quality in health care. Feeding back these data is an essential aspect of the successful monitoring of service provision. This is the first study to consult stakeholders on how they themselves think performance data or research findings are best presented to them in order to reach informed decisions.
In general, box plots were discussed much more positively than league tables, and qualitative information was considered more appropriate than box plots. The use of league tables was considered especially unsuitable for representing quality of care. Box plots were characterized as being more capable of representing a variety of factors in more detail. However, the reservations about validity, although a lot stronger with regard to league tables, were still considered an issue. The transparency of box plots was seen as a major advantage and they were perceived as a simple and visual tool, although many asked for an explanation of what the different components represented. The need for explanation was the most important impediment to their effectiveness.
Both league tables and box plots were judged to lack essential information on which important decisions, such as those relating to resource allocation, could be based. They felt that these should be accompanied by additional information that would clarify the context of the institutions represented and the methodological background of the instrument. The stakeholders pointed out that the usefulness of both instruments should rather be seen as an impetus to further discussion and debate.
The instruments were discussed in the context of the organizational realities in which UK health services function. Stakeholders had been able to witness the various, but not always appropriate, uses and effects of league tables in other sectors, or in a different political climate [12,13]. Measure fixation, for example, is one of the unintended dysfunctional effects of league tables [14], where effort is invested in less productive domains for the sake of sheer scoring and maintaining one’s league position. They embody the philosophy that competition would form a stimulus for improved quality [15]. Their hierarchical design has a confrontational effect, which was rejected as ‘detrimental’, ‘dangerous’, ‘demotivating’, and ‘threatening’ by the stakeholders. There was consensus that methods of feedback should be constructive and encouraging. This is why one of the perceived advantages of box plots was anonymity. Stakeholders do value the ability to make comparisons, but they want them to be anonymous and to a specific standard, i.e. benchmarks or best practice, rather than in a ranking system that shows one or two services at the bottom of a list.
However, reflecting on the ideal approach it became apparent that stakeholders preferred openness, but with the purpose of naming and sharing rather than the previous policy of naming and shaming. Also, a good feedback system was judged to be adaptable to the new structures in which the NHS was developing in its move away from insular providers into more grouped services, like Primary Care Trusts and Networks. An adequate measurement tool should be able to take account of the functioning of units in a bigger whole.
The overwhelming request for qualitative feedback was seen in the light of the kind of care it was supposed to represent. Palliative care transgresses the usual boundaries of health care and it requires a multi-professional approach as it involves a diversity of areas: symptom and pain management, communication, emotional and holistic care, and continuing support for the family after bereavement. Qualitative research that collects and interprets in-depth data was viewed as providing the right evidence for gaining an understanding of what is appropriate care at the end of life. The importance of the service user’s view, which the stakeholders highlighted, requires approaches that illuminate the subjective meanings of patient’s behaviour and experience. The stakeholders saw that the lay perspective could make an important contribution to focus attention on key policy issues that have previously been excluded from the priorities of decision makers. Of course these qualitative data, which are to be used for decision making, must be collected, analysed, and reported in accordance with rigorous standards for qualitative studies. Validity is a common concern to all research methods. However, for qualitative research this will relate to interpretive validity. The involvement of the researcher in the research process is needed to carry out adequate qualitative research, which makes the concern with bias irrelevant. Rather, it requires that the researcher makes explicit in which ways his or her questions and observations may have shaped the data. Therefore, it is a question of making the research process as transparent as possible to produce valid qualitative research.
Similar concerns emerge outside the borders of the UK. In Europe, there is a development towards ‘health care convergence’, where the challenges and solutions in quality improvement are strikingly similar. Stakeholders in various countries are increasingly demanding accountability and transparency, and several of the trends witnessed in the UK, such as increased regulation and league tables, are manifest in other countries. In this light, the use of conventional feedback mechanisms should be reconsidered to adequately reflect performance in diverse working practices and environments, and to promote open exchange of information in an international context.
This article was produced as part of the ongoing work of the Project to imPROve Management Of Terminal illnEss (PROMOTE), which is supported financially by the Community Fund.
|
References |
|---|
|
TopMethodsResultsDiscussionReferences |
|---|
- Black N. Evidence based policy: proceed with care. Br Med J 2001; 323: 275–279.
[Free Full Text] - Klein R. From evidence-based medicine to evidence-based policy? J Health Serv Res Policy 2000; 5: 65–66.[Medline]
- Weiss CH. Improving the linkage between social research and public policy. In Lynn LE, ed., Knowledge and Policy: The Uncertain Connection. Washington DC: National Academy of Sciences, 1978; pp. 23–81.
- Trostle J, Bronfman M, Langer A. How do researchers influence decision makers? Case studies of Mexican policies. Health Policy Plan 2001; 14: 103–104.
- Harries U, Elliott H, Higgins A. Evidence-based policy-making in the NHS: exploring the interface between research and the commissioning process. J Publ Health Med 1999; 21: 29–36.
[Abstract/Free Full Text] - Lipton D.S. How to maximize utilization of evaluation research by policymakers. Ann Am Acad 1992; 521: 175–188.
- Stocking B. Why research findings are not used by commissions and what can be done about it? J Publ Health Med 2001; 17: 380–382.
- Advisory Group to the NHS Central Research and Development Committee. Methods for the Implementation of the Findings of Research: Priorities for Evaluation. Leeds: Department of Health, 1995.
- Clark D, Seymour J. Reflections on Palliative Care. Buckingham: Open University Press, 1999.
- Small N. The changing National Health Service, user involvement and palliative care. In Monroe B, Oliviere D, eds, Patient Participation in Palliative Care: a Voice for the Voiceless. Oxford: Open University Press, 2003.
- Hughes R, Higginson IJ, Addington-Hall J, Aspinal F, Thompson M, Dunckley M. Project to improve management of terminal illness (PROMOTE). J Interprof Care 2001; 15: 398–399.[Medline]
- Smith GD, Ebrahim S, Frankel S. How policy informs the evidence. Br Med J 2001; 322: 184–185.
[Free Full Text] - Tymms E, Wiggins A. Schools’ experience of league tables should make doctors think again [letter]. Br Med J 2000; 321: 1467.
- Keasey K. MPDD. Performance measurement and the use of league tables: some experimental evidence of dysfunctional consequences. Account Bus Res 2000; 30: 275–286.
- Brook R. Health care reform is on the way: do we want to compete on quality? Ann Intern Med 1994; 120: 84–86.
[Free Full Text]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||