International Journal for Quality in Health Care 15:187-188 (2003)
© 2003 International Society for Quality in Health Care
Editorial |
Questioning practices in health care research: the contribution of social surveys to the creation of knowledge
Graduate School in Social and Political Studies, University of Edinburgh, UK, Regional Editor, International Journal for Quality in Health Care
It seems evident that many researchers and practitioners, not to mention managers, involved in health care are rather circumspect, if not sceptical, about the value of social surveys and not without good reason. There has been, and continues to be, a lot of dubious, if not on occasion downright atrocious, examples of surveys that have been foisted upon patients, professionals and others that have helped sustain such a negative press. That is not to say that there have not existed shining exemplars of how surveys can and have added to the sum of human knowledge, but the general context of a questionable set of practices by some, fuelled by a view that they are simple to do and courting derision by those who see themselves at the top of an evidence-based tree, has done little to endear such practice to the clinical community.
Two recent large-scale reviews of questionnaires applied to health care, one a systematic review of the determinants of high response rates (292 RCTs) [1] and the other a large-scale (266 references) narrative review of best practice [2], are both revealing of two characteristics in particular. The first is the heterogeneity of results, for which selectivity bias was considered a possibility in the first study and in the second led the authors to declare that ‘there can be no universal recommendations on best practice in respect of questionnaire design and survey conduct’ (p. iv). The second is the nature of the selectivity bias in the ‘high level’ evidence. Few studies were drawn from the health care context [32% in Edwards et al. and a small (unspecified) number in McColl et al.]; the majority were studies conducted in the US, partly for reasons such as its large academic community and the sources largely located in English language journals; the use of ‘convenient’ but atypical samples, such as the heavy reliance on students, and the fact that many studies operate implicit or explicit restrictions on people on the basis of their age, perceived illness, ethnicity and sampling cost. Let me stress that this is not to claim that these studies were not interesting and revealing, but there is difficulty in being able to generalise on the basis of this selective and heterogeneous evidence, within which there are likely to be extensive interactions in the causal process. They also evade the knotty problem of cultural differences, both ethno-social and institutional. The appeal of experimentalism looks very shaky when placed within the social and political context of health care. Generalisation may be our common goal, but the question remains about where we draw the limiting lines in terms of culture, space and time.
If we hold surveys up to the same searching light of scrutiny then we get an equally problematic situation, partly for the same general measurement errors of social research, but also for a number of quite distinct reasons. It has the advantage of allowing observation of social phenomena as they naturally occur ‘in situ’, without the unethical or impractical problems of much experimental work. Its inability to randomise cases to different groups makes the process of establishing cause and effect more difficult, although not impossible, relying as it does on an iterative elaboration between theory and data, or using time ordered variables, often best done through panel surveys. However, it appears to be the quality of the survey development and implementation that is its greatest enemy. It is as though experiments are securely protected within the rigour of natural scientific endeavour, while surveys are seen as relatively simple applications of common sense that require little expertise. As a consequence, it appears that there is little attention to the educational or training needs of those who need to design surveys and insufficient resources to enable a good job to be done. Response rates are often a problem, although less so in health care research than many other fields, but this is not necessarily as serious as other more fundamental problems, such as poor questions, inappropriate response scales, exclusion of key variables, etc. It still seems ridiculous to come across patient surveys, for example, that claim to present patients’ views of quality that have never involved patients in their construction or evaluation; or, to take another common complaint, how few surveys analyse the non-response bias, by comparing those who respond with those who do not, or with the population profile. These and other basic questions, as listed by Kelly et al. [3], should perhaps be mandatory when reporting survey results.
Apart from reflecting practice ‘as is’ in the field, surveys can allow broader coverage of the population with less need for restricting who can take part. If we consider the history of surveys in health care over the last 50 years, it is evident that we have moved from virtually total absence to a state where nearly every service is committed to gauging patient experiences of care and increasingly staff, lay carers’, and citizens’ views as well. That research can take place outside a laboratory setting or with the direct involvement of those who are the subject of the study lays testimony to the more inclusive approach to understanding health care from a multi-stakeholder perspective. The question it raises is how we can raise its status from a methodology of questionable practices to a methodology that offers practicable questions for practical use. In a review of the use of patient surveys in the National Health Service (NHS) in Scotland [4], interviews with Directors for Public Involvement revealed a number of principles that were felt to be important in underpinning the effective use of surveys. These included ensuring that patients or the public should be involved in the planning and design of surveys, that there should be local ownership of surveys and that they should be adequately resourced. The evidence of how surveys had been developed there revealed little attention to questions of validity, methodological rigour, appropriateness and generalisability. It was clear that, while good examples existed, the majority of studies were carried out by people who, however well-intentioned, lacked survey research training and the resources to enable their efforts to be as useful and valid as they could have been for improving the quality of patient care.
The central question is how we can ensure that the survey instruments we develop are valid and reliable. As Bateson recognised, reliability is a necessary but insufficient condition of validity [5]. To show that a tool is reliable is not to demonstrate that it is appropriate or accurate in its prediction. While we can construct tests of the various forms of validity, in essence, as Steiner and Norman [6], as well as de Vaus [7] attest, validity is really about how well the instrument enables us to understand the object for which it has been developed. Without a ‘gold standard’ or with contested ideas about what should be included, ideas about criterion or content validity are hard to pin down. Construct validity is always subject to the strength and sophistication of the theory that exists at the time and it begs the question of whether departures from it are a sign of weakness or a genuine new understanding of reality. Bateson, in grappling with this conundrum of how to show that the outcome was valid, argued for a focus on attempting to ensure the best application of research knowledge to the process of doing surveys, for in this way it would help to maximise the probability of a successful outcome [5]. Here there are also some lessons we can learn from the more qualitative epistemological approaches to research, such as incorporating a reflexive position such that we stand back and critically assess the journey we have made in developing our survey, as well as asking others, such as our respondents, to reflect on their experiences.
Surveys need to be seen as an important partner with experiments in the pursuit of knowledge, but we can only expect them to be taken seriously if we pay much more attention to the educational and financial resources that underpin the improvement of quality of this form of evidence.
References
- Edwards P, Roberts I, Clarke M et al. Increasing response rates to postal questionnaires: systematic review. Br Med J 2002; 324: 1183–1191.
[Abstract/Free Full Text] - McColl E, Jacoby A, Thomas L et al. Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technol Assess 2001; 5.
- Kelley K, Clark B, Brown V, Sitzia J. Good practice in the conduct and reporting of survey research. Int J Qual Health Care 2003; 15: 261–266.
[Abstract/Free Full Text] - Scottish Consumer Council. Use of Patient Surveys in the NHS, Glasgow: SCC, 2001: http//www.scotconsumer.org.uk/reps01/reps12.pdf Accessed 21 January 2003.
- Bateson N. Data Construction in Social Surveys, London: George Allen and Unwin, 1984.
- Steiner DL, Norman GR. Health Measurement Scales: A Practical Guide to Their Development and Use, Oxford: Oxford University Press, 1995.
- de Vaus DA. Surveys in Social Research, 5th edn, London: Routledge, 2002.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||