Family satisfaction with nursing facility care
RAND, Gerontology, Pittsburgh, PA, USA
Background. We present the psychometric properties of our nursing facility family satisfaction questionnaire (NF-FSQ), and the steps used to develop this instrument.
Methods. Family members from 14 nursing facilities in Pennsylvania were mailed the NF-FSQ. Factor analyses were used to test the extent to which the items in each domain represented the same underlying construct. To further report the applied psychometric properties of the instrument we used the completeness of data, score distributions, item–scale consistency and reliability of domain scores.
Results. Of the 550 surveys mailed, 387 usable surveys were returned (70% response rate). Factor analyses show that the items were representative of the underlying factors. The percentage of family members not providing responses for each question was low, and varied from 1.0% to 3.4%. The floor and ceiling effects of the responses for each of the 20 questions were low. The item–scale internal consistency analyses determined that the correlation of items within indexes were higher than those with other indexes. Cronbach’s alphas for the domains were all higher than usually recommended levels.
Conclusions. We believe we have produced a short, psychometrically sound family member satisfaction instrument for use in nursing homes. We also show that response rates from family members can be very high.
Keywords: patient satisfaction survey, satisfaction survey, surveys
Address reprint requests to Nicholas Castle, RAND, Gerontology, Pittsburgh, PA, USA. E-mail: Castle{at}RAND.org
Accepted for publication July 21, 2004.
Two recent reports [1,2] support the notion that few elders are adequately able to judge medical measures used as quality indicators of nursing facility care. However, more than 800 000 elders enter a nursing facility each year, creating a situation whereby consumers may choose inappropriate facilities because of this disconnection between the information available to make an informed choice and the ability of consumers to make that choice. In this context, if publicly released, satisfaction information may be important in helping consumers choose a nursing facility.
In addition, measuring and reporting satisfaction with nursing facility care is important in several other ways. Firstly, residents are more likely to follow medical advice when they rate their care as satisfactory [3]. Secondly, results from satisfaction surveys can be used as measures of accountability [4]. Thirdly, measuring and reporting satisfaction with nursing facility care may also be important in improving some aspects of quality [5].
Despite the substantial benefits to collecting satisfaction information, some barriers to effective implementation of these initiatives in nursing facilities exist. Several authors have noted potential difficulties in obtaining survey information from residents. These include problems with response rates [3], low cognition of residents [6], acquiescent response bias [7] and lack of response variability [8].
Compared with using resident satisfaction surveys, we propose that the barriers to obtaining sound satisfaction information on the quality of nursing facility care are substantially lower by using mail questionnaires to family members of residents. With proper administration response rates can be high and cognition is less problematic. Costs of mail surveys are also significantly lower than in-person interviews of nursing facility residents. Although, we should note that family members may have a different perspective on nursing facility quality than residents, and it is unknown whether family members are adequate proxies for residents, or whether they provide a different view of nursing facility quality [9].
The aim of this project was to develop and test a short and psychometrically sound family satisfaction survey instrument. We ultimately wanted the survey to be widely used, and considered it important to minimize the personnel time and costs associated with administration, data collection and data entry when using the survey instrument. To this end, we attempted to develop an instrument that was as short as possible, reflect the aspects of quality of care that are important to family members and provide broad representation of the various domains of quality of nursing facility care.
We present the psychometric properties of our nursing facility family satisfaction questionnaire (NF-FSQ), and the steps we took to develop this instrument. One approach often used in instrument validation is factor analysis. However, such tests do not address the validity of the instrument when in use. Given that interest in using satisfaction instruments in nursing facilities is growing, in addition to factor analyses, we also present applied tests of usefulness for the NF-FSQ.
 |
Methods |
|---|
 Top Methods ResultsDiscussionReferences |
|---|
Review of existing instruments
Our first task was to determine whether an appropriate instrument currently existed. We searched the MEDLINE, CINAHL (Cumulative Index for Nursing and Allied Health Literature) and Psychological Abstracts databases. With the exception of relatively few studies [10–13] questionnaires identified addressed resident satisfaction with care. A review of many of these resident satisfaction instruments is provided by Kruzich and Cohen-Mansfield [14]. None of the family satisfaction instruments identified in our literature review met all of the criteria described above.
We also sought information from long-term care associations, state governments and private vendors. Similar to a recent publication in this area [15], we found several important resident and family satisfaction initiatives [e.g. the Ohio Department of Aging Family Satisfaction Survey (ODA-FSS)] [16]. None of the family satisfaction instruments met all of the criteria described above.
In addition, during the past 3 years we have collected satisfaction instruments used by nursing facilities in New Jersey, Pennsylvania and Maryland (n = 447 survey instruments). For the most part these instruments developed by facilities were short, but no details were available regarding psychometric properties. A further limitation of these instruments was that questions often addressed administrative and marketing issues, and not necessarily resident quality-of-care issues.
Questionnaire development
Domains
Our first consideration was to determine what areas of concern the questions should address. These general areas of concern are often referred to as ‘domains’. There seems to be little agreement in the literature concerning which domains are most important to measure [17–19], and most of this literature does not address the perspective of family members. Therefore, we asked family members to rate the general areas of nursing facility quality they believed were most important.
Using a sample of convenience consisting of four local nursing facilities, 375 surveys were mailed to family members of residents. A total of 292 surveys were returned, giving a response rate of 78%. Over a 3-month period 
90 surveys were mailed by each facility, following the same protocol and exclusions described below in the section describing our sample selection. One open-ended item asked family members to describe areas of nursing facility quality they believed were most important. Responses were transcribed into a text database, and areas of nursing facility quality were collapsed into domains. This process was guided by Patton’s cross-case approach to content analysis [20].
A list of 15 domains was identified and additional family members were asked to rate the top five areas. Again using a sample of convenience consisting of four local facilities (not including any of the four facilities used above), 370 surveys were mailed to family members of residents. A total of 279 surveys were returned, giving a response rate of 75%. Over a 3-month period 90 surveys were mailed by each facility, following the same protocol and exclusions described below in the sample selection section. A list of 15 domains was given, and family members were asked to identify the five areas they believed were most important. The results are shown in Table 1. We found a marked drop in the overall rating for the eighth domain, so based on this result we considered seven domains to be most important.
|
Item development
To develop items for the pilot satisfaction instrument we examined questions from published and commercially available sources (described above), and surveys previously collected from nursing facilities. These survey questions (1237 in all excluding duplicates) were separated into each of the domains listed in Table 1. Items were then evaluated for possible use in the survey instrument. Relevant questions were defined to be short and address just one topic. For the seven domains of interest, this reduced candidate items to 212.
These items were then rewritten to conform to the scaling requirements of the survey (described below) and to achieve a Flesch–Kinkaid scale score of 7, or lower. A Flesch–Kinkaid scale score of 7 implies that a respondent with a seventh-grade education should correctly understand the question being asked. This level was chosen because in the two mail surveys, described above [n = 571 (279 + 292)], 90% of respondents met or exceeded this level of education.
We then had a five-panel research team, consisting of experts and practitioners in gerontology, geriatrics and long-term care, choose their candidate items. These experts were asked to pick three questions in each satisfaction domain that they thought best captured the information, but also did not overlap in content with each other within the domain. The three most highly rated questions in each domain were used in the pilot instrument.
Response scale
A factor common to the results of many satisfaction surveys is a lack of response variability—in the case of elders’ use of the upper end of a response scale [8]. Using a total of 2450 questionnaires and examining five different response scales, Castle [21] found that a visual analogue rating scale from 1 to 10 was least prone to this bias. Therefore, this response scale was used in the NF-FSQ. Nevertheless, this scale performed best in an evaluation of only five different response scales [21], and the possibility exists that a different scale may be more advantageous.
Pilot instrument
In summary, the final pilot instrument consisted of seven domains, with three questions in each domain. In addition, two global questions were included. The global items were included, firstly, because of face validity reasons as almost all questionnaires use global items, secondly, because this gives a nice way of ending the instrument, and thirdly, because analytically one can examine possible respondent inconsistencies (i.e. low scores for the majority of individual items and high scores for the global items). The same methods described above were used to choose and modify the global items.
Data collection
Sampling frame
The sampling frame included all nursing facilities in Pennsylvania (PA), excluding the eight facilities used to develop the instrument questions described above. We examined facilities from this state because of convenience. The On-line Survey, Certification And Recording (OSCAR) data were used to generate a list of names and addresses of Medicare/Medicaid certified nursing facilities. This resulted in a total pool of 847 facilities with 72 000 beds.
Sample selection
To produce psychometric statistics and validate our questionnaire we estimated that a total sample size of 400 respondents was needed. Based on sampling strategies used in previous studies of satisfaction in nursing facilities, statistical consultation and expert panel opinion, we planned to survey 30 family members per nursing facility. Thus, we required participation from 14 nursing facilities to achieve our overall sample goal.
A random sample of 70 nursing facility administrators was contacted by letter describing the study. This was followed by a telephone call asking if they were willing to participate. Fifty-one calls were made to achieve our sample of 14 participating facilities. This resulted in a 34% participation rate for facilities. Reasons for non-participation were varied, but the most common reason was the need for chain facilities to get approval from corporate headquarters. Indeed, analyses (not shown) using the OSCAR show our sample to be representative of PA nursing facilities in terms of ownership, size and census, but to be under-representative of facilities belonging to chains.
Administrators in each facility were given 40 of our surveys with postage-paid envelopes. They were instructed to complete the mailing information for family members of residents and send out the questionnaires. We gave administrators 40 surveys because we anticipated a 75% response rate – giving us approximately 30 complete surveys per facility. Questionnaires had facility identification codes but no family member identification codes. Our purpose was to survey the general long-term care adult population. Therefore, administrators were instructed to exclude family members of residents who were <65 years of age, hospice care residents and family members of residents with lengths-of-stay of <30 days.
As we did not have access to resident records, we could not randomly pick family members for the questionnaire mailing. Rather, administrators were asked to randomly pick residents family members. This approach is somewhat limited because we have no way of knowing whether administrators conducted random sampling, or whether they chose family members likely to give more favorable responses. We had three nursing facilities of greater than 200 beds in our sample, for which this bias could exist because of the presumably large number of potential respondents available. Although, for the majority of facilities we believe this bias was unlikely to occur because to mail 40 surveys almost all eligible family members would need to be included.
It is also worth noting that three nursing facilities in the sample had less than 70 beds. These facilities were instructed to send out the questionnaires to all eligible family members, and if they did not achieve the target of 40 mailings to repeat this again 1 month later (with unduplicated family members).
Analyses
Our first objective was to obtain information about the factor structure of the pilot questionnaire and the performance of individual items. Our second objective was to report the applied psychometric properties of the instrument. Factor analyses were used to test the extent to which the items in each domain appeared to represent the same underlying construct. That is, to measure the degree of congruence between the domains of interest and the questions used to measure these attributes.
To further report the ‘applied’ psychometric properties of the instrument we followed the work of McHorney et al. [22]. The percentage of family members not providing responses for each question was determined. This is important because a score for each scale cannot be confidently computed if a high number of individual items comprising that scale are missing [22]. Score distributions include floor and ceiling effects; although, for satisfaction scores ceiling effects are usually of most interest. In our case, these are calculated by reporting the percentage of responses with a rating of 1 (floor) and 10 (ceiling). Item–scale internal consistency was determined, and represents the degree to which items correlate within each domain (corrected for item overlap). To examine the internal consistency we calculated Cronbach’s alpha for each domain.
|
Results |
|---|
|
TopMethodsResultsDiscussionReferences |
|---|
Of the 14 nursing facilities in our sample, 13 mailed all 40 questionnaires and one mailed 30 questionnaires. Thus, a total of 550 questionnaires was mailed from the 14 nursing facilities. An average of 29 questionnaires per facility was received, giving an average response rate of 73%. We excluded from the analyses family members with less than four visits, giving us a final sample size of 387 and usable average response rate of 70%. Excluded family members were evenly distributed across facilities, and the range of individual facility response rates was relatively uniform (64–77%).
Table 2 presents descriptive statistics and psychometric properties of the NF-FSQ. The primary factor loadings from the factor analyses are shown in the first column. All loadings exceeded the minimum cut-off of 0.30, indicating that the items were representative of the underlying factors. In addition, the groupings of items were the same as those proposed in the pilot instrument, as shown by the eigenvalues indicating a single factor solution (i.e. values greater than 1) for each domain (not shown).
|
The percentage of family members not providing responses for each question was low, and varied from 1.0% to 3.4%. The means and standard deviations show that in all cases the distributions are skewed. Family members tend to report favorable satisfaction, although it should be noted that for all questions the full range of scores was used (results not shown). The floor effects are negligible, whereas, as with most satisfaction surveys the ceiling effects are moderately high. The lowest ceiling effect score was 18.4% and the highest 29.2%.
The item–scale internal consistency analyses (corrected for overlap) show that the correlation of items within indexes were in all cases higher than those with other indexes (not shown). McHorney et al. [22] use a measure of correlation of >0.40 in item–scale internal consistency analyses. In all cases each of our items achieved this level. Cronbach’s alpha for all the domains was higher than the usually recommended level of 0.70 [22].
|
Discussion |
|---|
|
TopMethodsResultsDiscussionReferences |
|---|
Our satisfaction instrument is clearly very simple and brief. The number of responses and the response scale are highly amenable to facilities collecting their own data and entering the data into a simple database. While we recognize that other more sophisticated means of conveying satisfaction results are available, including web-based systems, we believe that for some nursing facilities these systems may actually be a barrier to facilitating use of the information, as they require both web access and some skill in navigating the sites. Some facilities have such access and skills, but others do not. We do not mean to disparage the nursing facility industry, but it should be noted that research has indicated that many nursing facility providers are ill prepared to analyze and interpret complex statistical reports, including quality performance data. This is important, because the development of a valid satisfaction survey only represents the first step in quality improvement—use of this instrument is a critical subsequent step. Thus, a means for measuring provider performance and presenting that information in an easily interpretable manner may not only be useful in enhancing consumer choice, but also allow for meaningful performance improvement feedback for providers. Although, even with our simple and short questionnaire, some facilities may still find it useful to receive instructions and/or training on data collection, use and reporting of satisfaction data.
It is worth noting that short instruments are also not without limitations. A short instrument is unlikely to be specific in identifying areas for quality improvement. In this respect longer instruments, such as the ODA-FSS, are advantageous as they provide more detailed information. An alternative approach to using either longer or shorter instruments, that could be examined in the future, is using a brief family satisfaction instrument followed by more specific instrument(s). The brief instrument could be used by states and accreditation bodies to assess aggregate levels of satisfaction (for example), but could also indicate to the facility potential areas for quality improvement. Other more specific satisfaction tools (or modules) could then be used by the facility to further assess potential problems identified by the short instrument.
Our experience shows that a typical respondent will take less than 7 minutes to answer the survey questions. For providers, data entry takes less than 3 minutes for each returned survey. Another advantage of the instrument we present is that it is not highly prone to ceiling effects. We do not mean to imply, however, that the instrument is not influenced at all by these effects. This is important, because ceiling effects bias results, and ultimately are not useful for quality improvement initiatives. Not all publications in this area present analyses of ceiling effects. Therefore, it is somewhat difficult to precisely determine how our questionnaire performs relative to others. However, in cases where an analysis of ceiling effects is given, our instrument seems to perform well in this area compared with other instruments (e.g. Gasquet and associates [9]).
We excluded from the analyses family members with less than four visits. This exclusion was not based on any theoretical or statistical information, but other studies have used similar exclusion rules [23]. Although, in retrospect, four visits for some family members may not be sufficient to adequately gauge facility quality. In our case this probably added little bias to the analyses, as the modal number of visits made by family members was 13. Nevertheless, future work should determine whether a more appropriate and statistically sound cut-off score for the number of visits could be used.
We should note that it is unclear as to whether proxies, such as family members, provide valid information. Some authors believe ‘the experiences and perceptions of a health care representative are simply not equivalent to those of a resident immersed in the day-to-day environment of the nursing home’ [24]. A further recent publication also found that proxy satisfaction scores were dissimilar from residents’ scores [9], but in this case for several satisfaction domains resident–proxy responses were highly correlated [9]. More work is needed in this area to determine whether family members are adequate proxies for residents, or whether they provide an orthogonal view of quality of care.
The issue of whether family members provide valid information is important. As we assert above, collecting satisfaction information from family members has fewer potential barriers than collecting equivalent information from residents, and as such represents an attractive option. However, collecting only family member satisfaction information risks censoring residents’ opinions if differences in resident and family opinions exist (and of course vice versa). Both residents and family are important customers for nursing facilities. Thus, collecting satisfaction information from both parties probably represents the optimal approach for truly understanding satisfaction with a nursing facility.
A potential bias to satisfaction surveys involves recall bias [25]. That is, over time family members may have difficulty in accurately responding to some questions. Our survey administration process ensured that family members received a questionnaire only when an elder was in the nursing facility. This may have decreased bias due to potential time lags. Other biases may still exist. For example, family members may have a social desirability tendency [26]. That is, they may respond to the satisfaction questions as they think they are expected to respond. Family members may also have had fear of reprisal, even though the anonymity of respondents was guaranteed; the fact remains that they did receive the survey directly from the facility. An identification code (facility code) was included on each questionnaire that we later used for data matching. Family members may have mistaken this for an individual identification code.
Despite these limitations, we believe the NF-FSQ provides a sound satisfaction instrument for use in nursing facilities. Firstly, we believe we have produced a short, psychometrically sound family member satisfaction instrument for use in nursing facilities. This is important because we were unable to find in the published literature a short instrument with well-defined domains and available psychometric properties for use in this long-term care setting. Secondly, with a response rate of 70%, we show that response rates from family members can be high. This is important because low response rates are likely to produce biased results, and are an endemic problem with resident satisfaction surveys.
|
References |
|---|
|
TopMethodsResultsDiscussionReferences |
|---|
- AARP. The Costs of Long-term Care: Public Perceptions versus Reality. Washington, DC: Health and LTC Research Center, 2001.
- Kaiser Family Foundation. State Budgets Under Stress: How Are States Planning to Reduce the Growth in Medicaid Costs? Washington, DC: Kaiser Commission on Medicaid and the Uninsured, 2002.
- El-Guebaly N, Toews J, Leckie A, Harper D. On evaluating patient satisfaction: Methodological issues. Can J Psych 1983; 28: 24–29.
- Cohen-Mansfield J, Ejaz F, Werner P. Satisfaction Surveys in Long-Term Care. New York: Springer, 2000.
- Fisher C. Sleuthing for clues: Providers use satisfaction surveys to uncover activities and services residents need and want. Provider 1997; 23: 30–36, 38.
- Simmons SF, Schnelle JF, Uman GC, Kulvicki AD, Lee KO, Ouslander JG. Selecting nursing home residents for satisfaction surveys. Gerontologist 1997; 374: 543–550.
- Ross C, Steward C, Sinacore JM. A comparative study of seven measures of patient satisfaction. Med Care 1995; 33: 392–406.[CrossRef][ISI][Medline]
- Pascoe G, Attkisson CC. The evaluation ranking scale: A new methodology for assessing satisfaction. Eval Prog Plann 1983; 6: 335–347.
- Gasquet I, Dehe S, Gaudebout P, Falissard B. Regular visitors are not good substitutes for assessment of elderly patient satisfaction with nursing home care and services. J Gerontol A Biol Sci Med Sci 2003; 58: 1036–1041.
- Kleinsorge IK, Koenig HF. The silent customers: Measuring customer satisfaction in nursing homes. J Health Care Mark 1991; 11: 2–13.[Medline]
- Soberman LR, Murray M, Norton PG, van Maris B, Tasa K. Satisfaction of residents and families in long-term care: III. Dissemination of results. Qual Manag Health Care 1997; 5: 63–71.[Medline]
- Mitchell JM, Kemp BJ. Quality of life in assisted living homes: A multidimensional analysis. J Gerontol Psychol Sci Soc Sci 2000; 55: 117–127.
- Ejaz FK, Straker JK, Fox K, Swami S. Developing a satisfaction survey for families of Ohio’s nursing home residents. Gerontologist 2003; 43: 447–458.
[Abstract/Free Full Text] - Kruzich J, Cohen-Mansfield J. Assessment of satisfaction: Summary of published scales. In Cohen-Mansfield J, Ejaz F, Werner P, eds. Satisfaction Surveys in Long-Term Care. New York: Springer, 2000.
- Lowe T, Lucas J, Castle N, Robinson J, Crystal S. State satisfaction initiatives. Gerontologist 2003; 43: 883–896.
[Abstract/Free Full Text] - Straker JK, Ehrichs L, Ejaz FK, Fox K. Implementation of the 2002 Ohio Nursing home Family Satisfaction Survey. Columbus, OH: Ohio Department of Aging, 2003.
- Kurzich J, Clinton J, Kelber S. Personal and environmental influences on nursing home satisfaction. Gerontologist 1992; 32: 342–350.[Abstract]
- Harel Z. Quality of care, congruence and well-being among institutionalized aged. Gerontologist 1981; 21: 523–531.[ISI][Medline]
- Rantz MJ, Vogelsmeier A, Manion P et al. Statewide strategy to improve quality of care in nursing facilities. Gerontologist 1999; 43: 248–258.
- Patton MQ. Qualitative Research and Evaluation Methods. Thousand Oaks, CA: Sage, 2002.
- Castle NG, Engberg J. Response formats and satisfaction for elders. Gerontologist 2004; 44: 358–367.
[Abstract/Free Full Text] - McHorney CA, Ware JE, Lu R, Sherbourne CD. The MOS 36-item short-form health survey (SF-36): III Tests of data quality, scaling assumptions, and reliability across diverse patient groups. Med Care 1994; 32: 40–66.[ISI][Medline]
- Ejaz FK, Noelker LS, Schur D, Whitlatch CJ, Looman WJ. Family satisfaction with nursing home care for relatives with dementia. J Applied Geron 2002; 21: 368–384.
- Davis MA, Sebastian JG, Tschetter J. Measuring quality of nursing home service: residents’ perspective. Psychol Reports 1997; 81: 531–542.[ISI][Medline]
- Applebaum RA, Straker JK, Geron SM. Assessing Satisfaction in Health and Long-term Care: Practical Approaches to Hearing the Voices of Consumers. New York: Springer, 2000.
- Krowinski WJ, Steiber SR. Measuring and Managing Patient Satisfaction. Chicago, IL: American Hospital Association, 1996.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||