International Journal for Quality in Health Care 15:73-078 (2003)
© 2003 International Society for Quality in Health Care
Paper |
Quality of psychiatric care: validation of an instrument for measuring inpatient opinion
1Epidemiology and Biostatistics Unit, Istituto Superiore di Sanità, Rome
2Department of Psychiatry, S. Filippo Neri Hospital, Rome, Italy
Objectives. To validate a brief self-completed questionnaire for routinely assessing patients’ opinions on the quality of care in inpatient psychiatric wards (Rome Opinion Questionnaire for Psychiatric Wards).
Design. A preliminary version was assessed for face and content validity by eight psychiatrists and two patient focus groups. The final version was evaluated for acceptability, factor structure, internal consistency, and test-retest reliability.
Setting. An inpatient psychiatric ward in a general hospital in Rome.
Study participants. The questionnaire was administered to all consecutive inpatients admitted over a 6-month period (n = 169). Test-retest reliability was evaluated by administering the questionnaire for a second time to 27 inpatients.
Main outcome measures. Face and content validity: psychiatrists and focus groups’ opinions on relevance, importance, and clarity; acceptability: inpatients’ opinions on user-friendliness; factor analysis: principal component analysis; internal consistency: Cronbach’s alpha; test-retest reliability: Cohen’s weighted kappa coefficient, intraclass correlation coefficient.
Results. After evaluating face and content validity, the questionnaire was reduced to 10 items. Inpatients found the questionnaire to be acceptable. Factor analysis revealed that three factors—professional qualities of staff, information received, and physical environment—explained 67.2% of total variance. Cronbach’s alpha was 0.82 for the questionnaire and 0.61, 0.71, and 0.35 for the three factors, respectively. Test-retest reliability was good; weighted kappa higher than 0.9 for three items and 0.6–0.9 for seven items. The intraclass correlation coefficient was 0.80.
Conclusions. The questionnaire seems to be adequate for evaluating patients’ opinions on care in inpatient psychiatric wards. Because of its user-friendliness, it may be particularly suitable for routine use.
Keywords: patient opinion, mental health services, psychiatric ward, reliability, questionnaire, quality of care
Introduction
Patient satisfaction is an aspect of the quality of health care that should be taken into consideration when developing quality-improvement programs [1,2]. It is important to evaluate patient satisfaction not only as an outcome in itself but also because low satisfaction can lead to poor compliance with treatment and consequently to non-optimal health outcomes [3–5].
In recent years there has been an increase in the number of studies measuring patient satisfaction in a variety of mental-health care settings [6], yet the psychometric properties of the questionnaires used to evaluate satisfaction have not been adequately validated. In fact, the acceptability, validity and reliability are rarely, if ever, reported.
Some of these questionnaires, such as the Verona Service Satisfaction Scale [7] and the Client Satisfaction Questionnaire [8], have been thoroughly validated. However, the Verona Service Satisfaction Scale is relatively long, even in its shortest version [9], and the answers are worded in terms of ‘satisfaction’, which compared with scales worded in terms of ‘opinion’, generally elicit answers that tend to be less critical [10]. Furthermore, the questions do not always refer only to the time spent in the psychiatric ward (e.g. questions on recreational activities and psychotherapy refer to the previous year, regardless of whether the patient was in the psychiatric ward or elsewhere).
The Client Satisfaction Questionnaire, although brief and well worded, does not cover specific aspects of general psychiatric care (e.g. information received by the patient on his/her mental disorder) or on inpatient care (e.g. opinions about the layout and furniture of the ward, on the information about what to do after discharge, and on the psychological-group activities). Although it includes a question on the perceived efficacy of care, it does not allow the factors that influence this perception to be determined. Furthermore, the questionnaire makes use of a Likert answer scale with only four points, potentially rendering the questionnaire less sensitive to changes in patients’ opinions. Another commonly used tool is the questionnaire developed by Rosenheck and colleagues for use in Veterans Administration inpatient psychiatric units [11]; however, this questionnaire, which includes 73 questions, is also quite long.
An additional problem regarding the above-mentioned questionnaires is that none of them are specific to inpatient settings. In the literature, only three instruments specifically designed for measuring satisfaction in inpatient mental-health settings have been reported: (1) the Patient Opinion Survey [12] whose main disadvantage is the use of a dichotomous rating scale, which reduces sensitivity; this survey only has a certain degree of validation; (2) the instrument developed by Hansson et al. [13], which includes 48 items and, in our opinion, is too long for routine use; and (3) the Psychiatric Care Satisfaction Questionnaire [14], which has been tested on only a small sample of patients.
In light of the above, we developed a questionnaire for assessing patients’ opinions on the quality of inpatient psychiatric care. This questionnaire, which we have named the ‘Rome Opinion Questionnaire for Psychiatric Wards’ (ROQ-PW), was designed to have the following features: it is specific to inpatient psychiatric wards, which according to users’ opinion surveys, is a setting that has been neglected, may be self-administered, is acceptable for routine use (i.e. brief and clear), focuses on investigating opinions as opposed to satisfaction, and is sensitive to even minor complaints.
Objectives
The objective of this report is to describe the development, the main features and the validation of the ROQ-PW questionnaire.
Materials and methods
Preliminary version of the questionnaire
The preliminary version of the ROQ-PW questionnaire included 20 questions covering various aspects of psychiatric care that were deemed important based on a review of the literature and on an open-questionnaire survey conducted among 40 inpatients who were hospitalized in the psychiatric ward of the San Filippo Neri Hospital in Rome, Italy.
Face and content validity of the preliminary version
To assess content validity, we sent the preliminary version of the ROQ-PW questionnaire to all six of the psychiatrists working at the psychiatric ward of the San Filippo Neri Hospital and to two renowned clinical psychiatrists. These psychiatrists were asked to evaluate the preliminary version by providing their overall opinion and by listing the questions in order of relevance and importance. All eight psychiatrists felt that the questionnaire was a useful measure of patients’ opinions and that all of the questions were relevant and important. However, the level of importance varied among questions, as shown by the order in which they were placed.
To assess face validity of the preliminary version, we solicited two focus groups consisting of 15 inpatients in the hospital’s psychiatric ward (one group consisted of seven inpatients and the other group consisted of eight inpatients). These inpatients were asked for their opinions on the relevance, usefulness, and clarity of each question and to identify which questions they would eliminate if the questionnaire had to be shortened. Most of these inpatients commented that the questions were clear but that the questionnaire was too long. The focus groups also felt that there should have been questions on the quality of food and the severity of the side effects of drugs.
Final version of the questionnaire
Although the eight psychiatrists felt that all of the questions were relevant and important, we decided to include only 10 questions in the final version of the ROQ-PW questionnaire to maximize its acceptability and to increase the potential response rate. The 10 questions were chosen based on those questions the psychiatrists most often ranked highest in terms of the level of importance and on those questions, which could be eliminated according to the focus groups.
Study population and administration of the questionnaire
The final version of the ROQ-PW questionnaire was distributed to all inpatients consecutively admitted to the psychiatric ward over a 6-month period in the year 2000. The questionnaires were distributed 6–7 days after admission by two research assistants belonging to the clinical staff, who asked the inpatients to complete the questionnaire on their own and to put them in a box. No time limit was given to patients for filling-out the questionnaire, although the research assistants did record the amount of time that each patient took to complete the questionnaire. The questionnaires did not contain the inpatients’ names; instead, they contained individual codes recorded by the research assistants. The patients were assured that their responses would remain confidential.
Acceptability
To evaluate acceptability, the assistants asked the patients whether or not they found the questionnaire to be acceptable and if they felt it could be rapidly and easily filled-out.
Factor analysis
Exploratory principal component analysis was conducted and the Kaiser criteria [15] were used to determine the number of factors.
Internal consistency
The internal consistency was determined by calculating Cronbach’s alpha on the entire questionnaire and on each of the factors identified.
Test-retest reliability
To assess test-retest reliability, we asked 30 consecutive inpatients who had filled-out the questionnaire whether, after several days, they would be willing to fill it out for a second time. Test-retest reliability was measured by calculating the Cohen’s weighted kappa [16] for each of the 10 items on the questionnaire and the intraclass correlation for the entire questionnaire.
Inpatients’ opinions on the care provided
Once the questionnaires were filled-out, we calculated the mean and median values, the standard deviations (SD), and the ranges for the entire questionnaire and for each item.
Results
Final version of the questionnaire
As mentioned, we decided to include only 10 items on the final version of the questionnaire, all of which were ranked as relevant and important by both psychiatrists and patients. These items cover the following aspects of care: (1) suitability of care; (2) staff availability; (3) staff kindness and politeness; (4) treatment of agitated patients; (5) information received about health conditions; (6) information about drug treatment; (7) information about care after discharge; (8) physical environment; (9) recreational activities; and (10) psychological-group activities (each morning, a cognitive-behavioural psychotherapy group was held in the ward). Space was provided on the questionnaire for additional comments.
As an answer scale, we used a Likert-type scale with five points (1–5), with higher numbers indicating greater satisfaction. Specifically, for each question, there were five possible answers, which reflected increasing satisfaction with a specific aspect of care or increasing frequency of an event. For example, for the question, ‘To what point do you feel that the care received is suitable for treating your problem?’, the possible answers were: (1) ‘not very’; (2) ‘mildly’; (3) ‘sufficiently’; (4) ‘very, with some exceptions’; and (5) ‘very, with no complaints’.
After performing the validation of the questionnaire, we realised that an item on overall opinion was missing, making it impossible to assess the contribution of the individual items to overall opinion. Thus, we have since added an eleventh item to the questionnaire (as shown in the Appendix): ‘If a friend or a relative of yours had to be admitted to a psychiatric ward, to what extent would you recommended admission here?’ We also realised that, given the importance of the issue of patients’ empowerment, an item on patients’ involvement might have been desirable. We have thus added a twelfth question: ‘To what extent do you think you have had a say about your daily activities and your treatment here?’.
Study population and administration of the questionnaire
A total of 174 inpatients were consecutively admitted to the psychiatric ward over a 6-month period in the year 2000. Of these patients, 169 filled-out the questionnaire (response rate of 97%). Of these, 105 (62%) were males; the mean age was 36 years (range 20–68 years); 98 (58%) had more than 8 years of education (i.e. they had completed middle school); and 69 (41%) were employed. Overall, 103 (61%) of the 169 participating patients had an ICD-9 diagnosis of psychotic disorder. About one-third of the patients had been admitted compulsorily, and one-third had been admitted previously to the same or other psychiatric wards.
Acceptability
In general, the patients found the ROQ-PW questionnaire to be acceptable and quick and easy to fill-out. The only item that created some problems was the one regarding the information received on the care to be provided after discharge, because some patients said they had not yet been told that they were about to be discharged.
All but two of the patients completed the questionnaire on their own; the two remaining participants had the questionnaire read to them by the research assistants. The average time needed to complete the questionnaire was 12 minutes (range 5–25 minutes).
Factor analysis
The factor analysis of the items on the questionnaire yielded three separate factors, which accounted for 67.2% of the total variance (Table 1). These factors are referred to as: Factor 1, professional qualities of the staff (items 2, 3, 4, and 10); Factor 2, information received (items 1, 5, 6, and 7); and Factor 3, physical environment (items 8 and 9).
|
Internal consistency
The Cronbach’s alpha for the entire questionnaire was 0.82. The Cronbach’s alpha for Factors 1, 2, and 3 was 0.61, 0.71, and 0.35, respectively.
Test-retest reliability
Test-retest reliability was evaluated for 27 of the 30 patients asked to fill out the questionnaire for a second time (three patients were not available). The 27 patients filled out the questionnaire for the second time 4–6 days after the first time. Overall, the agreement between the first and the second set of results was good. For three items, the weighted kappa was higher than 0.9 and for seven items it ranged from 0.6 to 0.9. The intraclass correlation coefficient was 0.80.
Inpatients’ opinions on the care provided
Table 2 shows the mean, SD, median, and range of the answer scores for the questionnaire, by individual item. All items had a good spread (SD of approximately 1 on the scale of 1–5). Although this psychiatric ward is considered to be one of the best in Italy, satisfaction was relatively low (mean total score 28.4; median total score 28.5; range 12–49). Mean values of less than 3 and median values of less than 4 clearly indicate a need for improvement. The most negative opinions concerned the information received and the physical environment.
|
Discussion
We developed and evaluated a questionnaire for the routine assessment of patients’ opinions on the quality of care in inpatient psychiatric wards, with particular emphasis placed on aspects such as acceptability for routine use, investigating opinions as opposed to satisfaction, and sensitivity to even minor complaints. With regard to face and content validity, as mentioned, these were deemed to be acceptable by the eight psychiatrists and the two focus groups who evaluated the preliminary version of the questionnaire. However, the decision to limit the number of items to 10, in an attempt to maximise the response rate, may have been too extreme, and we feel that at least two items could be added to cover overall opinion and patients’ involvement. Questions on the quality of food and the severity of side effects of drugs could also be considered, as suggested by the focus groups, although the length of the questionnaire must always be kept in mind. Moreover, item 10 (i.e. ‘How useful is the psychological-group meeting?’) may not be pertinent to all settings and could be modified or omitted if psychological-group activities are not conducted. In interpreting the evaluation of face and content validity, it should be stressed that no quantitative data were collected, although it also bears mentioning that the eight psychiatrists involved in this evaluation included all six of the psychiatrists working in this specific ward, who are assumed to be extremely familiar with this particular setting.
With regard to acceptability, based on the opinions of the inpatients, this was deemed to be high, as also shown by the rapidity with which the questionnaire was filled-out (mean time of 12 minutes) and the high response rate (97%). In fact, we feel that one of the main advantages of the questionnaire is its brevity, user-friendliness, and the fact that it can be self-completed, even by psychiatric patients. As mentioned, the question on the information received on post-discharge care created some problems because some patients said they had not yet been told that they were going to be discharged. Ideally, the questionnaire should be administered at the time of discharge from the hospital. However, we decided to administer it earlier so as to facilitate the evaluation of test-retest reliability. In any case, because the length of stay in the psychiatric ward is brief (i.e. about 10 days), patients are generally provided with this information after several days. Moreover, the morning clinical group often addressed the issue of what would be done after discharge.
The results of the factor analysis were also satisfactory, in that the percentage of total variance explained by three factors was quite high (67.2%).
Regarding internal consistency, the Cronbach’s alpha for the entire questionnaire was 0.82, which is very satisfactory considering the limited number of items; it was also satisfactorily high for the three factors.
The level of test-retest reliability was also high, as shown by the Cohen’s weighted kappa, which exceeded 0.9 for three items and ranged from 0.6 to 0.9 for the remaining seven items, and by an intraclass correlation of 0.80. These results indicate a good level of repeatability, meaning that any changes in scores over time would reflect true changes in the level of satisfaction.
With regard to the opinions revealed by the questionnaire, although this is, strictly speaking, not part of the actual validation, it could provide indications as to the questionnaire’s sensitivity. Specifically, the finding that patient satisfaction was relatively low, despite the fact that this ward is considered to be one of the best psychiatric wards in Italy, could indicate that the questionnaire is more sensitive than others. In fact, the high levels of satisfaction revealed by other questionnaires have been the cause of concern, in that these levels suggest a lack of sensitivity [6], which is generally considered to be due to problems in the questionnaires’ construction. For example, with specific regard to the answer scale, those questionnaires using a scale with a range of responses wider than ‘satisfied/dissatisfied’ revealed that only 49% of clients were satisfied [17,18]. When developing the ROQ-PW questionnaire, we attempted to address this issue by using a five-point Likert scale and by including the answers ‘very, with some exceptions’ and ‘very, with no exceptions’. In this way, patients may state a positive opinion and at the same time point to specific needs for improvement, in that the question ‘Which exceptions?’ inevitably arises.
The overall results of this study seem to indicate that the ROQ-PW questionnaire is an adequate tool for evaluating patients’ opinions on the care provided in inpatient psychiatric wards. Although the questionnaire could be further validated by comparing it with another validated tool (i.e. concurrent validity), any differences would probably be quite difficult to interpret; future evaluations could also include analyses of predictive and discriminant validity. Despite the fact that the questionnaire was specifically developed for inpatient psychiatric wards, it could be slightly modified for use in other settings, such as day centres, residential facilities, and day hospitals.
Appendix
The Rome Opinion Questionnaire for Psychiatric Wards1
1. To what point do you feel that the care received is suitable for treating your problem?
(Scale: ranging from ‘not very’ to ‘very, with no complaints’.)
2. When asking the doctors, nurses, or other staff for help, how often did they meet your request?
(Scale: ranging from ‘never’ to ‘always, without exception’; additional answer: ‘I never asked for help’.)
3. To what extent have the doctors, nurses and other staff been kind and polite?
(Scale: ranging from ‘not very’ to ‘very, with no complaints’.)
4. How have you liked the way staff have dealt with agitated patients?
(Scale: ranging from ‘I did not like it very much’ to ‘I liked it very much, with no exceptions’; additional answers: ‘it never occurred’ and ‘no opinion’.)
5. How clear and complete was the information that the doctors and nurses provided on your health conditions?
(Scale: ranging from ‘not very’ to ‘very, with no complaints’; additional comment: ‘I never received any information’.)
6. How clear and complete was the information on the benefits and side effects of the drugs that you are taking?
(Scale: ranging from ‘not very’ to ‘very, with no complaints’; additional comment: ‘I never received any information’.)
7. How clear and complete was the information on what care will be provided after you are discharged?
(Scale: ranging from ‘not very’ to ‘very, with no complaints’; additional comment: ‘I never received any information’.)
8. How much do you like the layout and the furniture of this ward?
(Scale: ranging from ‘not much’ to ‘very much, with no complaints’.)
9. How often have recreational activities been made available (for example, television, cards, newspapers and magazines)?
(Scale: ranging from ‘rarely or never’ to ‘always, with no exceptions’; additional comment: ‘I was never in the mood’.)
102. How useful is the psychological-group meeting?
(Scale: ranging from ‘not very’ to ‘very, with no exceptions’; additional comment: ‘I have never participated’.)
113. If a friend or a relative of yours had to be admitted to a psychiatric ward, to what extent would you recommended admission here?
(Scale: ranging from ‘not at all’ to ‘very much’.)
123. To what extent do you think you have had a say about your daily activities and your treatment here?
(Scale: ranging from ‘none at all’ to ‘much, with no exceptions’.)
1The complete Questionnaire is available as Supplementary Data at IJQHC Online.
2As psychological-group activities are not conducted in all inpatient mental-health settings, this question could be omitted or modified.
3Questions added after the validation of the questionnaire.
This work was supported by grant no. 96/Q/T41 of the Italian National Mental Health Project, Istituto Superiore di Sanità, Project: Development and validation of tools for outcome evaluation of mental health services, Italy (Professor P. Morosini).
Address reprint requests to: Pierluigi Morosini, Laboratorio di Epidemiologia e Biostatistica, Istituto Superiore di Sanità, Viale Regina Elena 299, 00161 Rome, Italy. E-mail: morosini{at}iss.it 
Accepted for publication October 28, 2002.
References
- Cleary PD, McNeil BJ. Patient satisfaction as an indicator of quality care. Inquiry 1988; 25: 25–36.[Web of Science][Medline]
- Jackson JL, Kroenke K. Patient satisfaction and quality of care. Mil Med 1997; 162: 273–277.[Web of Science][Medline]
- Harris LE, Luft FC, Rudy DW et al. Correlates of health care satisfaction in inner-city patients with hypertension and chronic renal insufficiency. Soc Sci Med 1995; 41: 1639–1645.
- Keith RA. Patient satisfaction and rehabilitation services. Arch Phys Med Rehabil 1998; 79: 1122–1128.[CrossRef][Web of Science][Medline]
- Albrecht G, Hoogstraten J. Satisfaction as a determinant of compliance. Community Dent Oral Epidemiol 1998; 26: 139–146.[Web of Science][Medline]
- Ruggeri M. Patients’ and relatives’ satisfaction with psychiatric services: the state of the art of its measurement. Soc Psychiatry Psychiatr Epidemiol 1994; 29: 212–227.[CrossRef][Web of Science][Medline]
- Ruggeri M, Dall’Agnola R, Agostini C, Bisoffi G. Acceptability, sensitivity and content validity of the VECS and VSSS in measuring expectations and satisfaction in psychiatric patients and their relatives. Soc Psychiatry Psychiatr Epidemiol 1994; 29: 265–276.[CrossRef][Web of Science][Medline]
- Attkisson C, Zwick R. The Client Satisfaction Questionnaire: Psychometric properties and correlation with service utilisation and psychotherapy outcome. Eval Program Plann 1982; 5: 233–237.[CrossRef][Medline]
- Ruggeri M. Verona Service Satisfaction Scale (VSSS-32). Epidemiol Psichiatr Soc 1994; 3: 73–82.
- Ware JE, Hays RD. Methods for measuring patient satisfaction with specific medical encounters. Med Care 1988; 26: 393–402.[Web of Science][Medline]
- Rosenheck R, Wilson NJ and Meterko. Influence of patient and hospital factors on consumer satisfaction with inpatient mental health treatment. Psychiatr Serv 1997; 48: 1553–1561.
[Abstract/Free Full Text] - MacDonald L, Sibbald B, Hoare C. Measuring patient satisfaction with life in a log-stay psychiatric hospital. Int J Soc Psychiatry 1988; 34: 292–304.
- Hansson L, Bjorkman T, Berglund I. What is important in psychiatric care? Quality of care from the patient’s perspective. Qual Assur Health Care 1993; 5: 41–47.[Medline]
- Barker DA, Orrell MW. The Psychiatric Care Satisfaction Questionnaire: a reliability and validity study. Soc Psychiatry Psychiatr Epidemiol 1999; 34: 111–116.[CrossRef][Web of Science][Medline]
- Comrey AL, Lee HB. A First Course in Factor Analysis. Lawrence Erlbaum Associates, Hillsdale, 1992.
- Altman DG. Practical Statistics for Medical Research. London: Chapman and Hall, 1991.
- Lebow JL. Consumer satisfaction with mental health treatment. Psychol Bull 1982; 91: 244–259.[CrossRef][Web of Science][Medline]
- Lebow JL. Client satisfaction with mental health treatment: methodological consideration in assessment. Eval Rev 1983; 7: 729–752.
[Abstract/Free Full Text]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||