OUP user menu

A brief questionnaire for assessing patient healthcare experiences in low-income settings

Tashonna R. Webster , Jeannie Mantopoulos , Elizabeth Jackson , Heather Cole-Lewis , Lillian Kidane , Sosena Kebede , Yigeremu Abebe , Ruth Lawson , Elizabeth H. Bradley
DOI: http://dx.doi.org/10.1093/intqhc/mzr019 258-268 First published online: 30 April 2011


Objective The aim of this study was to develop and to assess the validity and reliability of two brief questionnaires for assessing patient experiences with hospital and outpatient care in a low-income setting.

Design Using literature review and data from focus groups (n = 14), we developed questionnaires to assess patient experiences with inpatient (I-PAHC) and with outpatient (O-PAHC) care in a low-income setting. Questionnaires were administered in person by trained interviewers. Construct validity was assessed with factor analysis; convergent validity was assessed by correlating summary scores for each scale with overall patient evaluations, and reliability was assessed with Cronbach's alpha coefficients.

Setting Eight health facilities in Ethiopia.

Participants Patients >18 years old who had a hospital stay >1 day (n = 230), and patients who received outpatient care (n = 486).

Main outcome measures Patient evaluations of health care experiences.

Results The factor analysis revealed 12 items that loaded on five factors for the I-PAHC questionnaire. The O-PAHC showed similar results with 13 items that loaded on four factors. Summary scores for nearly all factors were significantly associated (P-value < 0.05) with the patient's overall evaluation score. The measure of reliability, Cronbach's alpha coefficients, showed good to excellent internal consistency for all scales.

Conclusions The I-PAHC on O-PAHC questionnaires can be useful in assessing patients’ evaluations of care delivery in low-income settings. The questionnaires are brief and can be integrated into health systems strengthening efforts with the support of leadership at the health facility and the country levels.

  • health system strengthening
  • Ethiopia
  • quality improvement
  • validation
  • low-income country
  • primary care


Health systems strengthening is an important international priority for the World Health Organization (WHO) [1], the United States Agency for International Development (USAID) [2] and numerous donor organizations [3]. One of the WHO's six building blocks of health systems is the delivery of health services that are effective, safe and good quality for those who need them [1]. Strengthening health service delivery requires special attention to the experiences of patients as it is a key indicator of whether improvements in health care delivery have been made and where to focus future improvement efforts.

As part of the Ethiopian Hospital Management Initiative and broader health care reform efforts in Ethiopia, the Ethiopian Federal Ministry of Health sought to integrate ongoing measurement of patients’ experiences into its health system strengthening efforts; nevertheless few studies have been conducted to validate measures of patient experience in low-income countries, and none exists within Ethiopia. Although standardized patient surveys are widely used in countries such as the USA and UK, existing literature measuring patient experience in low-income countries is limited. Many studies have used the SERVQUAL instrument [48], which was originally designed for the retail sector and has been shown to have limited convergent and construct validity [9]. Other survey instruments for assessing patient experiences that have been validated in low-income settings have been designed for specific services, such as dental care [10], diabetes care [11], antiretroviral therapy services [12] or primary care [13, 14]. We could find no studies in low-income countries that utilized validated measures for hospital care, and those that assessed patient experiences in primary care [13, 14] were developed and tested in West Africa; we know of no published studies of an instrument that has been validated for use in Ethiopia or in East Africa.

Accordingly, we sought to develop and validate a brief measurement tool for assessing patient experiences with hospital and outpatient care in Ethiopia. Using focus group data to identify potentially important concepts in patient's evaluation of health care and multiple revisions and stakeholder pre-testing of survey items, we developed two surveys, which were then validated using data from five hospitals and three health centers. Information from this study can be useful for policy makers, clinicians and healthcare managers in low-income settings seeking to promote patient-centered care.


Questionnaire design

We developed separate questionnaires for patient assessment of healthcare for inpatient care (I-PAHC) and outpatient care (O-PAHC) in several steps. First, we conducted a thorough literature review to identify instruments that could be used to assess patient healthcare experiences in low-income countries. Although we examined multiple instruments [1519], we identified the Consumer Assessment of Healthcare Providers and Systems (CAHPS) questionnaires [2023] as the most widely used and survey with the most empirical research supporting its reliability and validity. Nevertheless, given that it is primarily used in the USA, we sought to evaluate the appropriateness of the CAHPS questions for use in Ethiopia and make appropriate modifications for use in Ethiopia. Therefore, as a second step, we conducted 14 focus groups in geographically diverse areas of Ethiopia to learn about aspects of care that were most salient to individuals in this context. Focus groups included 8–10 people and were homogeneous in gender with seven male and seven female groups. Based on focus group data, we identified additional potential domains that were viewed by individuals as important to their healthcare experience. Third, based on interviews with stakeholders in the Ministry of Health, physicians and hospital administrators in Ethiopia, we modified some items, eliminated items that were not relevant (e.g. responsiveness of nurse to call buttons, which are not used in Ethiopia) and added items that were expected to be important in the setting of Ethiopia (e.g. ease of finding way around the facility). Fourth, the survey was translated into Amharic and back-translated to check the validity of the translation. The survey was then pre-tested in one hospital in Addis Ababa with 50 patients purposefully sampled through a 2-week period to reflect different days of the week and times of day; with ∼10 of these patients, we conducted cognitive interviews [24] to identify questions that were unclear or confusing. Based on these data, the survey items were modified, and final questionnaire were developed for fielding. The final I-PAHC and O-PAHC questionnaires covered five domains of care: nurse communication, doctor communication, physical environment, pain management and medication and symptom communication. Items were scored using a 4-point Likert scale, ranging from 1 (never) to 4 (always) in the I-PAHC survey and 1(strongly disagree) to 4 (strongly agree) in the O-PAHC survey. In both questionnaires, we added items asking patients to provide an overall evaluation of care (scored 0–10) and asking patients if they would recommend this facility to friends and family (on a 4-point scale from definitely no to definitely yes). The final questionnaires in their validated form are shown in Appendices 1 and 2.

Sample and data collection

The validation study was conducted in five hospitals and three health centers, with the goal of recruiting 50 patients per facility to participate. All eight health facilities were located in urban or semi-urban areas in Addis Ababa or Amhara region of Ethiopia. We employed a non-random quota sampling technique to recruit patients for participation in the study to ensure a representative sample of demographic variables, such as age and sex. Data were collected using face-to-face interviews due to the prevalence of illiteracy in Ethiopia. Prior to survey completion, the interviewers, who were hired as data collectors and were not facility employees, explained the study to potential participants and obtained their consent to participate. For both surveys, interviewers conducted the surveys in-person on different weekdays and different times of day (i.e. morning, afternoons and evenings) seeking to recruit 25 adult patients (cat 16 years old) per week with an effort to get a representative sample. To be eligible for the I-PAHC survey, patients had to have a length of stay of at least one overnight stay. In order to be eligible for O-PAHC, the patient had to have received care at the health facility on the day the survey was conducted. The survey was conducted at the time of discharge, after the patient had been treated either as an inpatient or an outpatient. A total of 50 patients were recruited from each facility to ensure that there was sufficient power to detect a 1-point change in the overall 10-point patient evaluation rating. The research procedures were approved by the institutional review board at the Yale University School of Medicine.

Data analysis

We assessed the reliability as well as construct and convergent validity of the questionnaire. To evaluate reliability, we used a measure of internal consistency, Cronbach's alpha coefficient, with coefficients of 0.70 or higher interpreted as indicating good to excellent internal consistency, as recommended by experts [25, 26]. Construct validity assesses the degree to which items that are conceptually related (i.e. are meant to measure the same construct) are empirically associated as one would expect if the measurement is valid. We assessed construct validity of each questionnaire using factor analysis. Based on existing literature and focus groups, we anticipated specific items would load on specific factors, anticipating a total of 5 factors in each questionnaire. A factor is the latent concept that the questionnaire seeks to measure empirically. In order to avoid correlations between factors and to determine distinct domains, an orthogonal rotation was used. Consistent with the current literature using factor analyses [27], factor loadings ≥0.40 indicated that the survey items were correlated with the common factors that were determined a priori. Owing to structured missing data in the data set (i.e. skip pattern questions), we ran two sets of factor analyses for both PAHC questionnaires, one with and without skip pattern questions. The statistical procedure used deletes the respondents with missing data on any included items, which results in the analyses being limited to only the respondents who answered all of the questions, including those that would have been appropriately skipped by respondents who, for instance, did not have pain or did not have medications. Therefore, we conducted the factor analysis in two ways: both with and without the items related to the skip patterns, in order to ensure we used of all available data. We did not impute missing data because the vast majority of missing data were missing due to skip patterns. A total of n = 146 of 230 respondents for I-PAHC and n = 347 of 486 respondents for O-PAHC had missing data due to questionnaire skip patterns (i.e. these respondent had not experienced pain or had not been prescribed medication). Other missing data comprised <10% of the samples and hence were not imputed [28]. We created summary scores by summing item responses within each construct, creating a scale. These summary scores were used to assess convergent validity by examining the statistical Pearson correlation of the summary scores with responses to the overall patient evaluation item.


Demographic characteristics of study sample

A total of 230 of 242 patients approached participated in the inpatient survey (response rate of 95%) (See Appendix 1 for I-PAHC instrument); 486 of 512 patients approached participated in the outpatient survey (response rate of 95%) (See Appendix 2 for O-PAHC instrument) (Table 1). In the I-PAHC sample, the mean age of the participants was 37 years, with a range of 16 to 88 years. About half (52%) of the participants were female. Twenty-one percent of patients reported themselves as illiterate, and 18% reported that they had a diploma or more education. Many patients (57%) reported their health as fair or poor.

View this table:
Table 1

Demographic characteristics of patients

In the O-PAHC sample, the mean age of the participants was 38 years, with a range of 16 to 83 years. Fifty-one percent of the outpatient sample was female. Sixteen percent of the patients reported themselves as illiterate, and another sixteen percent reported having a diploma or more education. The majority of respondents (77%) reported their health as fair or poor.

All health facilities recruited a minimum of 50 patients to participate in each survey, except for one hospital which only recruited only 27 patients for the I-PAHC survey due to poor response rate (Table 2). Half of the health facilities had fewer than 100 beds and were non-teaching.

View this table:
Table 2

Hospital characteristics

Factor analysis

The factor analysis revealed 12 items in the I-PAHC that loaded on 1 of 5 factors (Table 3). The five-factor model generally corresponded with the constructs that were anticipated based on the literature and a priori hypotheses. Each of the items had a loading of 0.40 or greater on only 1 of the factors, except for the item that measured nurses’ courtesy and respect, which loaded on both the communication with nurses factor (factor 1, loading = 0.75) and the pain management factor (factor 4, loading = 0.48). This is likely as nurses are instrumental in pain management for patients. Because the item had a higher loading on the communication with nurses factor, the item remained grouped under that factor for all subsequent analyses. Three items regarding privacy (item 10), symptom recognition (item 17) and ease of finding way around the hospital (item 18) did not load on any of the five factors, and, therefore, were excluded from further analysis.

View this table:
Table 3

Factor analysis including skip pattern questions

The factor analysis for the O-PAHC survey generated similar results, with 13 items loading on four distinct factors (Table 3). A five factor model was determined a priori; however, items 10 and 11 loaded on the same factor as items 13 and 14. Items 10 and 11 were expected to measure health communication while items 13 and 14 were expected to measure medication communication, suggesting much of the communication influencing patients’ experiences may be about medications. Given that items 10 and 11 were thought to be measuring a different domain conceptually, these two items were dropped from the factor summary scores. Three items regarding the ability to distinguish between doctors/health officers and nurses (item 7), availability of medication in the dispensary (item 15) and ease of finding way around the health facility (item16) were excluded from further analyses due to loadings <0.40.

As an additional analysis, we re-ran the factor analyses excluding the skip pattern questions (Table 4), which existed in both surveys. As expected, this resulted in fewer factors (as skip questions pertained to medication use and pain management and were eliminated from this additional analysis). The factor analysis for the I-PAHC survey revealed three factors. All of the items loaded on the same factors as the previous analysis, except for one item. The item ‘nurses explain things in an understandable way’ loaded on two factors, communication with nurses (factor 1) and communication with doctors (factor 2), reflecting the correlation of patient views concerning physicians’ and nurses’ interactions with patients. Excluding the skip pattern questions from the O-PAHC factor analysis did not substantially change the factor loadings (Table 4).

View this table:
Table 4

Items factor analysis without skip pattern questions


Cronbach's alpha coefficients for all the scales for the I-PAHC and O-PAHC surveys exceeded 0.70 (except for physical environment scale, which had a Cronbach's alpha coefficient of 0.54 in I-PAHC and 0.59 in O-PAHC), suggesting good to excellent reliability for the scales pertaining to communication with nurses, communication with doctors, pain management and medication communication factors for both I-PAHC and O-PAHC surveys (Table 5).

View this table:
Table 5

I-PAHC ad O-PAHC reliability estimates for multi-item scales and items

Convergent validity

Nearly all associations between each of the summary scores and the item assessing patients’ overall evaluations of the health care experience for both I-PAHC and O-PAHC scales were statistically significant (Table 6). For instance, strong and statistically significant associations were apparent with the communication with nurses scale for I-PAHC (rho = 0.46, P < 0.05) and with the communication with doctors scale for O-PAHC (rho = 0.56, P < 0.05). Summary scores for the communication with doctors scale, physical environment scale and pain management scale showed moderate and statistically significant associations with the overall evaluation rating (rho coefficients = 0.40, 0.32 and 0.36, respectively, P-values < 0.05) for the I-PAHC survey. The medication communication scale summary score for the I-PAHC survey, however, did not correlate very highly (rho = −0.04, P-value = 0.71) with the overall evaluation rating. All of the correlations between the summary scores for scales and patients’ overall evaluation for the O-PAHC survey were >0.40 (P-values < 0.05), except for the correlation between the summary score for the physical environment scale and patients’ overall evaluation (r = 0.23, P < 0.05).

View this table:
Table 6

Correlations of scales and items with patients’ overall evaluation


This study reports on the development and validation of questionnaires to assess patients’ experience in health in a low-income country, Ethiopia. The I-PAHC questionnaire includes 25 questions that comprise 5 constructs, and the O-PAHC questionnaire includes 23 questions that comprise 4 constructs. The results indicate that the scales in both questionnaires have good to excellent reliability, and both questionnaires have good construct validity. Together, these results indicate that the items that were conceptually related were also highly correlated statistically, providing evidence that they were measured by the instruments appropriately. Additionally, we found the survey adminstration to be feasible and useful in detecting differences across hospitals.

Patient experiences are frequently assessed as part of quality improvement efforts in high-income settings; however, few validated questionnaires are available for use in low-income settings. The SERVQUAL, although used in other studies, is not designed for health care facilities and therefore may miss important contextual aspects of patient experiences, such as experiences with physicians and nurses, particularly in low-income settings. Additionally, the SERVQUAL assesses both individual's expectations and perceptions of actual performance, which are measured at the same time and can introduce bias, as well as result in a substantially longer survey instrument and increased respondent burden. Some of the other surveys that have been assessed focus on specific types of care, such as diabetes [11] or HIV treatment [12] and therefore are less applicable to general hospital and outpatient care. One additional survey, designed for use in primary outpatient care rather than in hospitals, has been shown to be valid for populations in Upper Guinea and in Burkina Faso [13, 14], but has not been validated in East Africa. Another survey was developed for assessing patient satisfaction with hospital care in Bangladesh [15]; however, the survey was specific to the Bangladesh context, again not necessarily reflecting the health care system issues in East Africa. The I-PAHC and O-PAHC, which are brief, valid and reliable questionnaires, may be particularly helpful in evaluating the impact of health systems strengthening efforts on patients’ experiences with facility-based care. These questionnaires will allow facilities to measure patient experiences at one point in time as well as monitor trends over time. Ongoing monitoring of such data can be used at the national level to set benchmarks for hospitals and health centers.

Despite the benefits of measuring patient experience, implementation of the surveys for routine use requires focused efforts and human resource capacity. First, interviewers must be identified, trained and supervised. Although we anticipated some patients may complete the surveys themselves, we learned that due to illiteracy rates, physical disabilities and consistency of completion, in-person adminstration by a trained interviewer was most effective for gathering high quality and consistently completed questionnaires. Nonetheless, training was important, particularly in the sampling and consent procedures. Training of interviewers was accomplished in a one-day session. In addition, data were compiled on site and entered using a pre-formatted MS Access programme, which then exported data to Excel to produce automatically summary tables and figures to track hospital performance. Completion of the interview and its data entry was estimated to require an average of 15 min per questionnaire for a total staff time of just more than 12 h for 50 questionnaires in the month of the survey.

The results of this study must be viewed in light of several limitations. First, the sample size is relatively small, although we had adequate power to detect the expected statistical associations. Second, although we tested internal consistency reliability, we were unable to assess the questionnaire's test–retest reliability due to the logistical challenges of having patients complete the survey twice. Third, the physical environment construct had relatively lower internal consistency reliability than the other scales in the instruments, which would suggest that the two items do not fully capture the physical environment construct. More items in this construct may be needed to improve internal consistency reliability of the physical environment scale. Finally, we conducted the study in one low-income country; results may differ in other low-income countries and evaluating questionnaires in the context where they will be used is important.

The international focus on strengthening health systems includes expanded efforts in quality improvement of delivery systems. Measuring impact of such efforts is challenging, particularly in resource-limited settings where medical records are often limited and data capture systems can be unreliable. In such contexts, the patient's experience can be an important indicator for comparing facilities and for evaluating efforts to enhance patient-centered, higher quality care. The PAHC questionnaires are brief and can easily be administered in health care settings with the support of leadership at the health facility level as well as the country level. Leadership support has been apparent in Ethiopia as the Federal Ministry of Health has endorsed the use of the PAHC questionnaires in its national reform guidelines, and hospital managers are beginning to use the questionnaire regularly. The presence of effective leadership at both ministry and facility levels is critical for facilitating the implementation process as well as providing the resources needed for ongoing data collection, analysis and action to enhance the quality of care and improve patients’ experiences.


This work was supported by a grant from the Clinton Health Access Initiative and the Centers for Disease Control (200-2010-3g226).


We are grateful to Heather Sipsma, PhD, Jeph Herrin, PhD and Paul Cleary, PhD for providing statistical guidance and reviews of earlier drafts.

Appendix 1: In-Patient Assessment of Health Care (I-PAHC) Survey

Embedded Image

Appendix 2: Out-Patient Assessment of Health Care (O-PAHC) Survey

Embedded Image


View Abstract