International Journal for Quality in Health Care Advance Access originally published online on July 19, 2006
International Journal for Quality in Health Care 2006 18(4):275-280; doi:10.1093/intqhc/mzl022
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The development and validation of the Concise Outpatient Department User Satisfaction Scale
Centre on Behavioral Health, University of Hong Kong, Hong Kong
Objectives. To develop and validate a concise scale for measuring outpatient satisfaction suitable across specialties and cultures.
Design & Setting. Item generation adopted a concept-driven approach, and 10 candidate items were administered together with a battery of validation items and scales in a cross-sectional survey at a government-aided Chinese medicine specialized outpatient department in Hong Kong.
Participants. About 344 consenting patients or their accompanying caregivers were recruited upon their first visit at the clinic and interviewed one month thereafter.
Results. The overall response rate was 79%. After deleting one item (physician’s manner and attitude) for its redundancy suggested by interitem correlations, exploratory factor analysis yielded two factors, General Service and Case Physician, explaining 75% of variance of the remaining nine items. The internal consistency coefficients of the whole scale and the two subscales were higher than 0.90. Criterion-related validity was supported by high correlations with three anchor items, overall satisfaction, intended future reutilization, and recommendation to others (r = 0.38–0.85). Significant correlations with compliance and negative affects provided preliminary evidence for construct validity.
Conclusion. The psychometric properties of the resulting 9-item scale supported its usefulness in measuring outpatient satisfaction. Further validation studies in various specialties and countries are suggested to make future cross-cultural comparisons possible.
Keywords: outpatient, scale development, user satisfaction, validation
Address reprint requests to S. M. Ng, Assistant Professor, Centre on Behavioral Health, The University of Hong Kong, G/F Pauline Chan Building, 10 Sassoon Road, Pokfulam, Hong Kong. E-mail: ngsiuman{at}hku.hk
Accepted for publication May 25, 2006.
In recent years, there has been an increasing demand of accountability and productivity by consumers. It is now a global trend in healthcare development toward integrating subjective user satisfaction into the evaluation of medical service quality [1,2]. There is evidence that patient satisfaction is associated with treatment outcomes [3,4] though the cause–effect relationship is not clear. It is possible that illness behavior plays a role in the mechanism through which satisfaction affects clinical outcomes. Ferris and the Health Services Research Group [5] reported that satisfied patients are more likely than their dissatisfied counterparts to show positive illness behaviors, e.g. complying with their medical regimens and disclosing important medical information to their physicians [6–8]. Therefore, knowledge of users’ satisfaction with the service can serve not only as a performance indicator but also to identify areas of improvement to provide better care and services for the betterment of the users’ health.
There are very few user satisfaction scales that are suitable for use in outpatient settings across different specialties. Among the existing ones, some scales have yet to complete the validation process [9]. Some reported satisfactory or good levels of psychometric properties, but very often they contain many items (typically 20–30 items) [10–12]. On the one hand, looking at individual aspects of service (at least the most important ones) is necessary, as most patients are not uniformly satisfied with all aspects even though they are generally satisfied with the service [13]. Also, it is conceivable that a longer scale more likely yields more reliable results. On the other hand, a user satisfaction measure with numerous items has its own drawbacks. A long scale can lead to low motivation of patients to complete the questions and subsequently a low response rate and biased sample as well as more missing data. It is also difficult for many outpatient settings to incorporate the administration of a long questionnaire into their busy daily operation, needless to mention that very often other questionnaires or assessments are administered together at the same time. Seeing this limitation, Perneger et al. [14] derived a 16-item outpatient satisfaction survey from previously published instruments. The scale was administered to a large sample (n = 1027) in Switzerland and the seven dimensions of the scale showed satisfactory to good internal consistency. An even briefer instrument (one item of overall satisfaction and eight items of different satisfaction aspects) was developed by Rubin et al. [15] to compare outpatient satisfaction with different kinds of practices in the United States. The authors successfully collected evaluation data from 17 671 outpatients and found that overall satisfaction was associated with patients’ staying with or leaving the physicians or systems in 6 months. However, the relationship between different satisfaction aspects and outcome was not examined. It would also have been helpful if the authors could provide detailed psychometric statistics of the questionnaire.
Another important issue of outpatient satisfaction assessment is that most scales are developed in the West. These scales often include items that are specific or applicable to only some cultures or regions, e.g. availability of free car parking space, multilingual information and translation, and managed care organizations or health insurance-related service. As such, the applicability of these scales in different cultures is severely limited. An outpatient clinic user satisfaction scale suitable for multinational studies is yet to be developed.
This study aimed to develop a user satisfaction scale for use in outpatient settings that could address the problems discussed above. The scale was expected to have the following characteristics: (i) short—ideally within ten items, so that it is administrable in a busy medical practice; (ii) items not specialty-specific so that the scale can be potentially applied in a wide range of outpatient settings; and (iii) items not culture- or region-specific so that the scale can be potentially applied in a wide range of regions of different cultures and socioeconomic development status.
 |
Method |
|---|
 Top Method ResultsDiscussionAppendixAcknowledgementsReferences |
|---|
Subjects
Subjects of this study were 344 participants of an evaluation study of a Chinese Medicine Specialist Outpatient Department of a government-aided hospital in Hong Kong. They were recruited through consecutive sampling of new cases presenting at the clinic during a 5-week period from September to October 2004. The recruitment of new patients only was to avoid or minimize possible systematic bias in sampling, as there is evidence that satisfaction varies with visit frequency [16,17]. Patients of all ages were included, but for those who were aged under 13 (the entry age of high school, at which the individual is commonly considered competent to understand and complete questionnaires) or mentally unfit to be interviewed, they had to be accompanied by a carer who could serve as the informant. Also, all subjects (patients or their informants) were able to communicate effectively in Cantonese or Putonghua. Written informed consent was sought from each subject before data collection. Subject’s confidentiality and interviewer’s independency were emphasized to avoid or minimize biased responses due to worries of loss of anonymity and negative impact on service to be received.
The Concise Outpatient Department User Satisfaction Scale
A scale was constructed to measure user satisfaction. The investigators adopted a concept-driven approach for generation of items that are considered generally applicable to outpatient settings of different specialties. The construct was conceptualized as having three domains, namely physical setup, clinic operation, and case physician. Specific items were then generated under each domain. Because the scale intends to be applicable across medical specialties, and hopefully across culture as well, great caution had been taken in avoiding items that are restricted to a certain disease group or cultural related lifestyle. The initial item pool was reviewed and critiqued by experienced health researchers at the Centre on Behavioral Health, The University of Hong Kong. Eventually, a three-domain, 10-item scale was constructed. The items were (i) physical environment, (ii) equipment and facilities, (iii) appointment arrangement, (iv) waiting time, (v) service of the dispensary, (vi) support staff, (vii) case physician’s professionalism, (viii) explanation given by the case physician, (ix) case physician’s attitude and manner, and (x) consultation time. Respondents need to answer each item in a response format of a 10-point anchored numerical scale, where ‘1’ indicates not satisfied at all and ‘10’ extremely satisfied.
Validation items/scales
Three additional items were created as the anchor items to examine the criterion-related validity of the scale: overall satisfaction of the clinic, intended future reutilization, and likelihood of recommending the clinic to others. They were also answered in the abovementioned 10-point response format. Because some studies reported that patient satisfaction is associated with clinical outcomes and compliance [3,4], the Chinese Affect Scale (CAS; a scale of 20 items that measure positive and negative affects common in and applicable to Chinese populations) [18] and two 10-point items of self-reported compliance with follow-up bookings and medical regimen were added to evaluate the construct validity of the scale.
Procedure
All new patients of the clinic were screened by the attending nurse staff at the assessment room before seeing their physicians. The attending nurse gave each new patient a printed reminder so that he/she would report to the independent interviewer in the specified consultation room after seeing his/her physician. At the same time, the interviewers could monitor prospective subjects’ whereabouts with a computer terminal in his/her consultation room. This enabled the interviewers to get touch with all new patients, even during the busiest hours. There were totally six interviewers in this study, each of whom was a retired registered nurse with over 20 years of experience in public hospitals in Hong Kong and had been trained by the investigators (S.M.N. and I.F.T.) on administering the questionnaires of the study, including the Concise Outpatient Department User Satisfaction Scale. There were two interviewers standing by at the clinic at any time during the recruitment period to recruit and interview subjects.
After receiving information of the study and giving informed consent, each subject was interviewed by an interviewer. Interview instead of self-completion was employed to minimize invalid or missing data [12] and to obtain more reliable responses by providing instant explanations of the instructions and items of the questionnaire to subjects when necessary. The subjects were followed up on telephone by their corresponding interviewers 1 month after first interview (i.e. from October to November 2004). If the subject could not be reached at the first attempt, then at least two more attempts were made before he/she was considered unreachable and dropout.
Statistical analysis
We used 1-month follow-up data for validation of the scale, as we reckoned that patients were not well informed upon their first visit at the clinic and their opinions might be premature and not very reliable. The Concise Outpatient Department User Satisfaction Scale was firstly evaluated for its factor structure with exploratory factor analysis (Principal Component Analysis with Varimax Rotation) with SPSS 11.5 for Windows. Then, the internal consistency of the refined scale was examined with split-half reliability and Cronbach’s alpha coefficient [19]. The scale’s criterion-related validity was examined by looking at the correlations between the total satisfaction score with the single-item overall satisfaction score, intended future reutilization, and likelihood of recommending the clinic to others. Construct validity was examined by correlating the total satisfaction score with compliance with follow-up arrangement, compliance with medical regimen, as well as positive and negative affects of the subjects.
|
Results |
|---|
|
TopMethodResultsDiscussionAppendixAcknowledgementsReferences |
|---|
Response rate
During subject recruitment, a total of 436 patients who satisfied the inclusion criteria were approached, out of which 367 gave consent to participate in the research, making the response rate of first interview 84.2%. All but 23 of the 367 subjects were successfully followed up 1 month later (response rate = 93.7%). The overall response rate of the study was therefore 84.2 x 93.7% = 78.9%.
Among the responders (n = 344), 98 (28.5%) were male and 246 (71.5%) were female. Their mean age was 44.0 years (SD = 17.1). Efforts were made to compare the basic demographic data (sex and ages were the only information that non-responders/dropouts agreed to disclose) of the responders (who were successfully followed up) and non-responders/dropouts of the study so as to check whether there was any systematic bias in the successfully recruited sample. Among the 92 non-responders/dropouts, 34 (37.0%) were male and 58 (63.0%) were female. Their mean age was 45.2 years (SD = 18.1). The responders and the non-responders/dropouts were not statistically different in terms of age (P = 0.593) and sex (P = 0.116).
Factor structure
Inter-item correlations were examined before proceeding to factor analysis. Item 9 (case physician’s attitude and manner) was removed from the scale because it was too highly correlated with items 7, 8, and 10 (r = 0.91, 0.83, and 0.88, respectively) and thus considered a redundant item.
The number of valid cases (i.e. those free of missing data) for exploratory factor analysis was 305. Using principal component analysis with criterion of eigenvalue > 1, two factors were extracted from the nine remaining items. Visual inspection of the Scree plot also supported a two-factor solution. The two factors explained totally 75.62% of variance. Factor loadings after Varimax Rotation of each item and total variance explained by the factors are presented in Table 1. With meaningful loadings set at 0.40, only item 1 (physical environment) had double loadings on the two factors. Names were given to the two factors according to the concept addressed by the items: factor 1 (items 1–6) was named general service; and factor 2 (items 7, 8, and 10) was named case physician.
|
Means and standard deviations of the total satisfaction score (averaged sum of the nine items) and the two subscales of the whole sample and by sex, age strata, and subsequent visit after first visit and before 1-month follow up are displayed in Table 2. T tests did not show any significant differences between the two sexes and the three age strata, but subjects who had subsequent visit(s) rated significantly higher on case physician (P = 0.048) but not total satisfaction (P = 0.14) and case physician (P = 0.37) when compared with those who had no subsequent visit. Regression analysis indicated that basic demographic variables including age, sex, marital status, and educational level did not account significantly for total satisfaction. The final version of the Concise Outpatient Department User Satisfaction Scale is provided in the Appendix.
|
Internal consistency
By odd–even split of the nine items, the Guttman split-half reliability was estimated to be 0.93 and the unequal-length Spearman–Brown reliability was 0.94. The coefficient alpha of the scale, which represents the average of all possible split-half reliability coefficients, was 0.91, falling into the range of ‘good’. The factor alphas were also in the good range: 0.90 for general service and 0.93 for case medical practitioner.
Criterion-related validity
Correlations between the satisfaction scores and validation items/scales are summarized in Table 3. Almost all individual items were significantly correlated with the three anchor items: overall satisfaction (r = 0.51–0.77), intended future reutilization (r = 0.13–0.50), and likelihood of recommending the clinic to others (r = 0.10–0.50). The total satisfaction score (averaged sum of all items) was also significantly correlated with the three anchor items (r = 0.85, 0.38, and 0.38, respectively, all P < 0.001), supporting the concurrent validity. The two subscale scores were also significantly correlated with the three anchor items. For general service, the correlation coefficients were 0.76, 0.26, and 0.23, respectively (all P < 0.001). For case physician, the correlation coefficients were 0.77, 0.49, and 0.51, respectively (all P < 0.001).
|
Construct validity
Significant but mild correlations were found between the total satisfaction score and compliance with follow-up bookings (r = 0.23, P < 0.001) and compliance with medical regimen (r = 0.18, P < 0.001). Although it was not associated with positive affect as measured by the CAS (r = 0.05, P > 0.05), it was negatively correlated with CAS negative affect (r = –0.26, P < 0.001) as expected. The two subscores showed similar patterns as the total satisfaction score. See Table 3 for details.
|
Discussion |
|---|
|
TopMethodResultsDiscussionAppendixAcknowledgementsReferences |
|---|
We successfully created a scale with a parsimonious number of items that is suitable for outpatient settings across specialties. The scale had high levels of internal consistency and validities as suggested by alpha coefficient and the scale’s correlations with other related measures. Factor analysis suggested that about 75% of the variance of the items could be explained by two latent variables, namely general service and case physician. Contrary to expectation, physical setup and clinic operation of an outpatient department could not be differentiated from one another; they contributed to user satisfaction as a single factor.
In this study, a 10-point response format in which the two ends are labeled respectively ‘not satisfied at all’ and ‘extremely satisfied’ was adopted. The decision was based on the rationale that giving ratings in a 10-point scale is intuitively more direct and thus easier than, say, 4-, 5-, or 6-point scales. This response format was also preferable to Likert scale for two reasons: respondents can give a rating without needing to read the description of each response choice as in Likert response format and it offers a more sensitive measure and avoids skewedness potentially caused by the choice of the number of response choices (e.g. 5- versus 6-point Likert scale) [20]. The high level of internal consistency (alpha > 0.90) and low rate of missing data (<2%) support that the scale was easy to understand and effective in obtaining reliable measures.
Interestingly, although the total satisfaction score was correlated with the single-item overall satisfaction more strongly than the two subscores did, the case physician subscore was a better predictor of intended reutilization, recommending to others, as well as compliance with follow-up arrangement and medical regimen. Because compliance with medical follow ups and regimen affects patients’ clinical outcomes, it is implicated that to help enhance patients’ clinical outcomes and patients’ satisfaction with their case physicians should be elevated through striving for and maintaining a high level of quality of physicians.
A limitation of the study is that the test–retest reliability of the scale has not been evaluated. Nevertheless, the ultimate goal of assessing patient satisfaction is to monitor the quality of service of the clinic as a whole and identify areas for improvement so as to improve patients’ treatment outcomes. The scale’s predictive power is worthy of further investigation. Follow-up studies may be conducted in the future to test whether satisfaction measured by this scale can predict patients’ illness behavior and clinical outcomes as indicated by objective as well as subjective measures. It is also advisable to test the scale on outpatient departments of other specialties to generalize its applicability and usefulness.
The Concise Outpatient Department User Satisfaction Scale is intended for assessing patient satisfaction in outpatient settings across specialties. The items included cover only main aspects conceived to be universal across specialties. This property of the scale enables comparisons of results from different specialties. The inevitable limitation is that aspects that might be of particular importance to particular specialties are not included, making the scale not suitable for in-depth evaluation of outpatient departments of a particular specialty. Similarly, this scale is intended to have potential of development and use in other regions or countries. Thus, in generating the item pool, precautions were taken to include only core satisfaction items and avoid lifestyle items specific to local context. Given its good psychometric properties as revealed in this study, further scale validation in other countries and cultures is worth pursuing for it would help conducting cross-regional or -cultural studies of health care service quality.
|
Appendix |
|---|
|
TopMethodResultsDiscussionAppendixAcknowledgementsReferences |
|---|
Concise Outpatient Department User Satisfaction Scale
Please indicate your satisfaction with each of the following aspects of this clinic by circling a number from 1 to 10, where ‘1’ means not satisfied at all and ‘10’ means extremely satisfied.
|
|
Acknowledgements |
|---|
|
TopMethodResultsDiscussionAppendixAcknowledgementsReferences |
|---|
This study was funded by the Tung Wah Group of Hospitals (TWGHs). We thank Dr K. F. Leung, Dr K. T. Tom, Ms Rita Li, Mr Philip Tam, Ms Suen, and Ms Kwan of TWGHs for their relentless support throughout this study. We gratefully thank the contributions of Ms Yun-ying Chang, Ms Sai-mei Lau, Ms Chi-lim Chan, Ms Siu-mui Wong, Ms Mei-ngan Sin, and Ms Sau-kam Lui in data collection. We also thank Ms Ada Yiu for her efficiency in data coding and entry and Ms Nora Cheng for carrying out quality assurance check of the data set.
|
References |
|---|
|
TopMethodResultsDiscussionAppendixAcknowledgementsReferences |
|---|
- Hudak PL, Wright JG. The characteristics of patient satisfaction measures. Spine 2000; 25: 3167–3177.[CrossRef][Web of Science][Medline]
- Urden LD. Patient satisfaction measurement: current issues and implications. Outcomes Manag 2002; 6: 125–131.[Medline]
- Holcomb WR, Parker JC, Leong GB, Thiele J, Higdon J. Customer satisfaction and self-reported treatment outcomes among psychiatric inpatients. Psychiatr Serv 1998; 49: 929–934.
[Abstract/Free Full Text] - Kelstrup A, Lund K, Lauritsen B, Bech P. Satisfaction with care reported by psychiatric inpatients: relationship to diagnosis and medical treatment. Acta Psychiatr Scand 1993; 87: 374–379.[Web of Science][Medline]
- Ferris LE, Health Services Research Group. A guide to direct measures of patient satisfaction in clinical practice. CMAJ 1992; 146: 1727–1731.[Medline]
- Aharony L, Strasser S. Patient satisfaction: what we know about and what we still need to explore. Med Care Rev 1993; 50: 49–79.[Medline]
- Carr-Hill RA. The measurement of patient satisfaction. J Public Health Med 1992; 14: 236–249.
[Abstract/Free Full Text] - Lochman JE. Factors related to patients’ satisfaction with their medical care. J Community Health 1983; 9: 91–109.[CrossRef][Medline]
- Dawn AG, Lee PP, Hall-Stone T, Gable W. Development of a patient satisfaction survey for outpatient care: a brief report. J Med Pract Manage 2003; 19: 166–169.[Medline]
- Gasquet I, Villeminot S, Estaquio C, Durieux P, Ravaud P, Falissard B. Construction of a questionnaire measuring outpatients’ opinion of quality of hospital consultation departments. Health Qual Life Outcomes 2004; 2: 43.[CrossRef][Medline]
- Lubeck DP, Litwin MS, Henning JM, Mathias SD, Bloor L, Carroll PR. An instrument to measure patient satisfaction with healthcare in an observational database: results of a validation study using data from CaPSURE. Am J Manag Care 2000; 6: 70–76.[Web of Science][Medline]
- Labarere J, Francois P, Bertrand D, Peyrin JC, Robert C, Fourny M. Outpatient satisfaction: validation of a French-language questionnaire: data quality and identification of associated factors. Clin Perform Qual Health Care 1999; 7: 63–69.[Medline]
- Coulter A, Cleary PD. Patients’ experiences with hospital care in five countries. Health Aff (Millwood) 2001; 20: 244–252.
[Abstract/Free Full Text] - Perneger TV, Etter JF, Raetzo MA, Schaller P, Stalder H. Comparison of patient satisfaction with ambulatory visits in competing health care delivery settings in Geneva, Switzerland. J Epidemiol Community Health 1996; 50: 463–468.
[Abstract/Free Full Text] - Rubin HR, Gandek B, Rogers WH, Kosinski M, McHorney CA, Ware JE Jr. Patients’ ratings of outpatient visits in different practice settings: results from the Medical Outcome Study. JAMA 1993; 270: 835–840.
[Abstract/Free Full Text] - Cho WH, Lee H, Kim C, Lee S, Choi KS. The impact of visit frequency on the relationship between service quality and outpatient satisfaction: a South Korean study. Health Serv Res 2004; 39 (1): 13–33.[CrossRef][Web of Science][Medline]
- Fitzpatrick R, Hopkins A. Problems in the conceptual framework of patient satisfaction research: an empirical exploration. Sociol Health Illn 1983; 5: 297–311.[CrossRef][Medline]
- Hamid PN, Cheng ST. The development and validation of an index of emotional disposition and mood state: the Chinese Affect Scale. Educ Psychol Meas 1996; 56: 995–1014.[Abstract]
- Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951; 16: 297–334.[CrossRef][Web of Science]
- Ware JE, Hays RD. Methods for measuring patient satisfaction with specific medical encounters. Med Care Rev 1988; 26: 393–402.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||