Editorial |
Satisfaction surveys: do we really need new questionnaires?
Before introducing a new intervention in the health care setting, whether it is a technology, a new drug, or an evaluative tool, its validity must be fully tested as the final goal is to improve patients (quality of) care, as any change in patients care is potentially associated with benefits, risks, and costs.‘Validity’ is not a yes/no attribute, but the results of the accumulation of evidence were produced through a long process where hypotheses are formulated and then sequentially challenged with several empirical testings. A good example is the process of development and validation of new drugs. The regulatory agencies that are responsible for the first evaluation of the scientific and clinical value of new drugs, generally require information on the intrinsic quality and clinical safety and efficacy of drugs before approving their introduction in the market. After an exhaustive pre-clinical evaluation, phase I and II studies on small samples of patients give information about the toxicity and activity of the new compounds. Later phase III and IV studies provide data on safety and efficacy, either in an ideal and real setting. The cornerstone of this evaluation is a formal comparison of the new drug with an appropriate control. This approach derives from the experience that some drugs have been shown to have good intrinsic pharmacological properties in early phase studies but when they were tested in more carefully controlled research trials or in the clinical practice, results were less impressive or the benefit was largely overcome by negative effects. This complex and cumbersome evaluative process is implemented as drugs should be rapidly released for patients who need them but not at the expense of adequate knowledge about their real benefit [1].
During the past decades, a large number of patients questionnaires have become available to assess the outcome or the quality of health care of different conditions and settings. All these measures were recently grouped into the term patient-reported outcomes that refers to all concepts and measures about health and health care as perceived and reported by subjects (i.e. patients, citizens, consumers, etc.). Patient-reported outcomes represent the effects of treatment on subjects assessed entirely from the subjects’ perspective. Different questionnaires are developed and used to measure different concepts, such as quality of life, health-related quality of life, health perceptions, symptoms, satisfaction with treatment/care, etc [2]. In the field of health status assessment, after a period characterized by a rush to develop new and redundant instruments with scarce attention to their value, now there is a consensus on the methods to be adopted to develop/validate new instruments and researchers, and users have settled on a few well-documented tools. A ‘development and validation process’ has been well codified, and questionnaires are developed and validated through prospective multistep procedures based on explicit conceptual models, in which biological and clinical variables are linked in an priori hypothesized way with health-related quality of life outcomes, depending on the underlying health concepts they represent. Thus, a causal relationship pathway can be expected an then empirically tested. Questionnaires are developed and eventually evaluated in terms of validity, reliability, responsiveness, acceptability, and interpretability, using a combination of qualitative and quantitative empirical methods. Figure 1 shows a simplified way for questionnaire development and validation. It may be noted that a preliminary phase is identifiable (Conceptual framework and measurement model) that implies the item identification through focus group techniques, followed by a preliminary clinical testing in very selected sample(s) (Preliminary psychometric evaluation and measurement validation). Then, a further validation in independent and heterogeneous samples are carried out to confirm the psychometric attributes and add some information about the incremental value versus other well-established measures and/or more traditional outcomes and indicators (Clinical validation and comparison). Eventually, questionnaires are diffused with manuals and papers (Diffusion and monitoring) supporting their characteristics and facilitating the use. This approach is also generalizable and recommended to any other field where new instruments are developed to be used in the health care setting, such as diagnostic tests, guidelines and protocols, case-mix measures, etc [3,4].
|
The study reported by González et al., in this issue of the Journal, describes the process of development and validation of a new satisfaction with care questionnaire to be used on hospitalized Spanish-speaking patients. Methods used were standard multistep studies and evaluations that implied the review of literature, the identification of a multidimensional a priori conceptual and measurement model, several focus groups to identify candidate items, a first qualitative test to refine the questions and a further quantitative empirical validation in a convenience sample. The article is well organized, the methods are agreeable, data and findings are fairly reported and discussed. Nevertheless, something is missing: we do not know anything about the actual value of the new questionnaire in terms of its superiority when compared with other instruments. We do have only information about its psychometric intrinsic validity and on the correlation with a couple of variables that are simple mono-item questions about patients’ global perception of quality of care and self-perception of health improvement. This latter analysis can be considered either a sort of tautology or an inappropriate analysis as we do not have anything to support the validity, reliability, and meaning of the two external criteria adopted. On other words, the article ends where it is supposed to start. Preliminary phases (I and II) are just a demonstration of the good intrinsic psychometric validity of the scales, and the association with external criteria can assure us that the instrument is promising, but such evidence does not tell us anything about the true/incremental value of the new tool when compared with using nothing or something else. Data from preliminary phases are necessary but non-sufficient as the availability of confirmative and comparative data (from phase III) are crucial to have an idea of the actual value of the new technology. If we believe that we still need new language and country-specific satisfaction with care questionnaires to measure patient perceptions of health care quality, well pre-planned, multistep, sequential qualitative and quantitative studies should be conducted and, when completed, eventually published in peer-reviewed journals and then diffused to potential users. The very problems are at the beginning and at the very end of the entire process [1]: before starting a quite long and costly process to reinvent the ‘wheel’, it should be very well documented that there are no other alternatives, ready-to-be-used or suitable for linguistic or setting adaptation [2]; before publishing results, the validation process should be completed with final and definitive demonstration of incremental/added value versus the appropriate comparator, using outcomes that have meaning for decision-makers.
In this era where any decision taken in the health care setting may have implications in terms of benefits, risks and costs, what we need is evidence to support decision and not new tools not supported by good evidence.
Istituto di Ricerche Farmacologiche ‘Mario Negri’, Milan, Italy
References
- RedmondK. The US and European regulatory systems: a comparison. J Ambulatory Care Manage 2004; 27: 105–114.[Medline]
- WillkeRJ, Burke LB, Erickson P. Measuring treatment impact: a review of patient-reported outcomes and other efficacy endpoints in approved product labels. Control Clin Trials 2004; 25: 535–552.[CrossRef][Web of Science][Medline]
- WeingartenS. Practice guidelines and prediction rules should be subject to careful clinical testing. JAMA 1997; 277: 1977–1978.
[Abstract/Free Full Text] - ApoloneG. The state of research on multipurpose severity of illness scoring systems: are we on target? Intensive Care Med 2000; 26: 1727–1729.[CrossRef][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||