International Journal for Quality in Health Care Advance Access originally published online on July 22, 2005
International Journal for Quality in Health Care 2005 17(6):511-519; doi:10.1093/intqhc/mzi064
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Using goal attainment scaling to improve the quality of long-term care: a group-randomized trial
1 Sherbrooke University Geriatric Institute, Research Centre on Aging, 2 University of Sherbrooke, Community Health Sciences, and 3 University of Sherbrooke, Family Medicine, Sherbrooke, Québec, Canada
Objective. To assess the effect of a 6-month interdisciplinary customized intervention based on goal attainment scaling and aimed at improving the quality of care provided by long-term care facilities to frail older adults.
Design. A pair-matched group-randomized trial with quality of care measurements taken before and after the intervention and 6 months later.
Settings. Forty unregulated small-to-medium sized long-term care facilities delivering inadequate care to at least one resident. Facilities were matched on baseline quality of care, health district, and size. One member of each pair was then randomly allocated to the intervention, the other acting as a control.
Study participants. The trial involved 201 frail older adults randomly selected from the 40 participating facilities.
Intervention. The intervention was tailored to the main quality problems identified at baseline in the facility. The first task of the intervention team was to set weighted quality improvement goals with the facility manager, which were then recorded on a goal attainment follow-up guide. Subsequent monthly on-site visits, interspersed with frequent telephone calls, were meant to assist the manager and staff to implement permanent changes in the areas of care targeted for improvement.
Main outcome measure. Quality of care measured with the Quality of Care (QUALCARE) Scale, a multidimensional instrument that uses a 5-point scale to assess six subdimensions of care: environmental, physical, medical management, psychosocial, human rights, and financial. A score greater than 2 is considered indicative of inadequate care.
Results. The intervention effect on the overall quality of care was neither clinically nor statistically significant. Change from baseline to the end of the intervention averaged .21 and .22 in the experimental and control conditions, respectively (P = 0.86).
Conclusion. Attainment of preset quality improvement objectives did not translate into detectable improvements in the care provided to residents.
Keywords: aged, goal attainment scaling, group-randomized trial, improvement, long-term care setting, quality of care
Address reprint requests to Gina Bravo, Research Centre on Aging, Sherbrooke University Geriatric Institute, 1036 Belvedere South, Sherbrooke, Québec, Canada J1H 4C4. E-mail: gina.bravo{at}usherbrooke.ca
Accepted for publication June 18, 2005.
Concerns about the quality of care provided in the long-term care setting persist despite some evidence of improvement [1]. Hence innovative quality improvement strategies still need to be developed, implemented in settings delivering suboptimal care, and tested for their efficacy [2,3]. Assessing the efficacy of such interventions is challenging for health services researchers because of the heterogeneity of care recipients and the diversity of quality problems encountered. A facility that mostly houses cognitively impaired older adults may have difficulty caring for those who exhibit disruptive behavior or tend to wander. Another facility that mainly serves clients no longer able to accomplish basic activities of daily living may need to improve the provision of personal care. Still another facility that lacks an effective monitoring system of the residents’ state of health may be unable to react promptly and adequately in emergency situations.
These examples suggest that a one-size-fits-all approach may not be appropriate in this context. They underscore the need to design and implement customized interventions that are tailored to the more pressing quality problems faced by the particular facility. In turn, this requires choosing facility-centered measures of the impact of proposed interventions that accommodate the multiplicity and diversity of quality improvement objectives.
Goal attainment scaling is an attractive approach for designing and assessing health care interventions that vary in content and objective [4]. Lewis et al. [5] describe it as ‘a technique for evaluating program effectiveness on the basis of the extent to which individualized goals, established at intake, have been achieved’. Goal attainment scaling thus appears suitable for interdisciplinary teams and problems that warrant an individualized approach to intervention planning and outcome measurement. Developed in the 1960s as a means to assess community-based mental health services, goal attainment scaling has since been used in a variety of health settings and client populations. Moreover, it has been shown to be a reliable, valid, and responsive method to quantify program-induced changes relative to expected outcomes [6–11].
We recently applied goal attainment scaling in 20 residential care facilities providing suboptimal care to frail older adults. These facilities had been allocated to the experimental arm of a group-randomized trial designed to assess the value of this approach in improving the quality of long-term care. The 6-month intervention has been described in detail elsewhere [12]. Briefly, it was conducted by three interdisciplinary teams of health professionals experienced in caring for frail older adults and formally trained in using goal attainment scaling in long-term care institutions. Applying the technique in that context involved the following steps. Firstly, areas for improvement were selected from the main quality problems identified at study entry. In consensus with the manager, a numerical weight was then assigned to each goal to reflect its relative importance. Thirdly, each goal was formally expressed in measurable, observable terms as a statement of the outcome to be achieved by the end of the intervention. Achieving this result would constitute a successful outcome for the facility. Next, weighted goals and expected outcomes were transcribed onto a follow-up guide similar to that shown in the Appendix[12]. Other potential outcomes were then identified to complete the graded 5-point scale, using the expected outcome (scored as 0) as the benchmark. Twenty different follow-up guides were constructed, one for each experimental site.
Through monthly visits to the facility and frequent telephone calls, the intervention team then assisted the manager and staff to implement permanent changes in the areas of care targeted for improvement. Assistance included informing facility managers of community-based agencies that could be called upon for social and health care services, providing staff with clinical practice guidelines that had been developed at our institution, purchasing small pieces of equipment to improve the safety of the premises, and training staff in caring for difficult clienteles such as that with behavioral problems.
At the end of the experimental period, the intervention team assessed the status of the facility against the goal attainment levels. An aggregate measure of change—the goal attainment score—was then calculated for each experimental site as follows:
 |
Aggregating data across facilities showed that goal attainment scores had increased significantly between pre- and post-intervention, from 29.44 ± 6.69 to 51.51 ± 12.35 (one-sample t test = 7.72, P < 0.01). In addition, the distribution of final scores did not differ significantly from a normal distribution with a mean of 50 and a standard deviation of 10 (Kolmogorov–Smirnov statistic = 0.20, P > 0.25), as expected when preset objectives have been achieved.
From these findings, we concluded that quality improvement objectives that are established with the manager following a thorough evaluation of the quality of care provided in the facility can, on average, be achieved with the help of experienced health professionals. This conclusion, however, was based on analyses conducted at the facility level. It remained to be seen whether achieving these goals translated into measurable improvements in the quality of care provided to residents. In the event that care did improve following the intervention, it should then be ascertained whether improvements were sustained after the research team left the experimental setting [13]. This article complements our previous article by reporting the results of assessing the impact of the intervention on residents’ quality of care, versus usual care, at the conclusion of the intervention and 6 months later.
 |
Methods |
|---|
 Top Methods ResultsDiscussionAppendixReferences |
|---|
We conducted a pair-matched group-randomized trial based on a repeated-measures design [14–16]. The study was approved by the Research Ethics Board of the Sherbrooke University Geriatric Institute. Written informed consent was obtained from all facility managers and capable residents involved in the study. In the case of mentally incapacitated residents, consent was provided by their legal or designated guardians.
Study population
The study population has a two-level hierarchical structure with residents nested within facilities. Facilities eligible for the study met the following inclusion criteria: (i) they were unregulated; (ii) they housed between 4 and 45 elderly residents; and (iii) they provided inadequate care to at least one older adult. The first two criteria follow from prior findings showing that unregulated small-to-medium sized facilities were those most likely to deliver inadequate care [17,18]. Whether a facility satisfied the third criterion was determined before randomization, on the basis of an assessment of the quality of care provided to a random sample of residents. Homes caring primarily for younger adults with developmental disabilities, physical or mental handicap were excluded.
Residents eligible for the study were 65 years or older. They had difficulties with at least two activities of daily life and had been living in the facility for at least 3 months. Any resident waiting for a transfer to another long-term care institution or whose health condition warranted immediate attention from a health professional was excluded.
Sampling
The study took place in two regional health districts of the province of Québec, Canada. A list of facilities satisfying the first two inclusion criteria was provided by the two Regional Boards. Facility managers were informed of the study through a personalized letter, then contacted by phone to initiate the recruitment process. Verification of the third inclusion criterion required going to the facility and assessing a sample of residents. A validated case-finding telephone questionnaire was used to reduce the number of homes that had to be visited [19]. Answers to the questionnaire are used to predict the quality of care provided. Facilities whose predicted score lay above the empirically derived cut-off point were labeled positive and considered potential candidates for the trial. Starting with the facility expected to provide the most inadequate care (i.e. had the worst predicted score), and gradually proceeding down the ordered list, we asked each manager for authorization to assess the quality of care provided to frail older adults (between 4 and 6 depending on the size of the facility). Consenting managers provided a list of all residents who met the study eligibility criteria.
The residents randomly selected from these lists were contacted to confirm their eligibility and willingness to collaborate. Eligible residents who agreed to be assessed underwent an evaluation of the quality of their care using the Quality of Care (QUALCARE) Scale described below. A facility was eligible for randomization if one or more of the assessed residents received a quality rating above 2, reflecting inadequate care. Eligible facilities were then pair-matched according to three factors: average quality of care provided to the sampled residents (below or above the median), health district, and size (± 15 beds). Lastly, one member of each pair was randomly assigned to the intervention condition whereas the other acted as a control. No intervention was conducted in the control homes, which continued to provide the usual care to their residents. Figure 1 shows the recruitment process.
|
Data collection
Baseline characteristics of the experimental and control facilities were gathered through the case-finding questionnaire and a short interview with the manager. Sociodemographic information was collected on the residents selected for the trial as well as data regarding their functional and cognitive disabilities. Functional disabilities were assessed with the revised version of the functional autonomy measurement system (SMAF) [20]. Total scores range from 0 (complete autonomy) to 87 (total dependency). Cognitive deficits were measured with the Modified Mini-Mental State (3MS) Examination [21], a revised version of Folstein’s Mini-Mental State Examination (MMSE). Total scores range from 0 (worst) to 100 (best).
Evidence of the efficacy of the intervention was based on two measures: final goal attainment scores [12] (a facility-level variable, only available in the experimental sites and considered an intermediate outcome) and quality of care scores (the primary outcome, measured at the resident level). Care quality was assessed at three occasions: during the recruitment phase (baseline), at the end of the intervention (first posttest), and 6 months later (second posttest). Neither data collectors nor residents were told which facilities had been allocated to the intervention and control conditions. To perform intent-to-treat analyses, we planned to remeasure all surviving residents at both posttests, even in the facilities that dropped out of the trial.
Quality was assessed with the QUALCARE Scale, a validated multidimensional instrument comprising 54 items that assess care in six key areas: environmental, physical, medical management, psychosocial, human rights, and financial [22–24]. Each item is scored on a 5-point scale from one (best possible care) to five (worst possible care). Ratings are assigned after spending time in the facility, directly observing and interacting with the residents and their care providers. A resident’s quality of care score is derived by averaging the ratings he/she received on each item of the scale.
Statistical analyses
Analyses of the main effect of the intervention were based on general linear mixed models [14]. A matched mixed-model analysis of covariance (ANCOVA) was performed when restricting the analysis to data collected at the first posttest and a matched random-coefficients analysis when also including data from the second posttest. Analyses were done with SAS PROC MIXED, which employs the resident as the unit of analysis, accounts for the nested design of the trial and accommodates missing values [25].
Sample size
We used an equation given in Murray [14] to determine level-specific sample sizes. Setting the two-sided type 1 error at 0.05, we estimated that an analysis involving 20 matched sets would have 80% power to detect an adjusted difference of 0.5 between the intervention and control conditions in the quality of care provided. This estimate assumed that three residents were sampled from each small-size facility (4–9 beds) and five from each medium-sized facility (10–45 beds). To account for an anticipated 20% loss to follow-up at the resident level, the number of residents sampled from each small-size facility was raised to four and that from each medium-sized facility to six. Allowing for an extra 15% loss to follow-up at the facility level, the number of matched sets to recruit was increased from 20 to 24. We thus planned to enroll 240 impaired older adults from 48 facilities delivering inadequate care.
|
Results |
|---|
|
TopMethodsResultsDiscussionAppendixReferences |
|---|
As shown in Figure 1, 40 facilities were randomized to the experimental or control condition instead of 48 as planned initially. Identifying these facilities was a lengthy process that took over a year and involved assessing the quality of care provided to 336 frail older adults from 70 facilities. Characteristics of the 40 facilities are summarized in Table 1. Despite the small sample, randomization led to two groups of facilities sharing similar characteristics. Residents from experimental facilities were also comparable to those from their control counterparts (cf. Table 2). In particular, there was no statistically or clinically significant difference between conditions in the dimension-specific and overall quality of care provided to the residents.
|
|
The number of facilities and residents involved in subsequent stages of the trial are shown in Figure 2. Two experimental sites closed during the intervention period, resulting in the relocation of 12 subjects. The number of residents lost to follow-up at the first posttest was much larger than expected: 41.2 and 37.4% in the experimental and control conditions, respectively (P = 0.58). Reasons for not being reassessed post-intervention did not vary across conditions: death (11 versus 17, P = 0.19), transfer (28 versus 17, P = 0.08), hospitalization (1 versus 3, P = 0.30), and refusal (2 versus 0, P = 0.26). On average, the 79 residents lost by the end of the intervention were less educated (P = 0.03), more dependent (P = 0.03), and had been judged, at baseline, to receive lower quality care in the medical management domain (P = 0.04) than the 122 residents who were reassessed. However, the 42 losses to follow-up from the experimental arm were comparable to the 37 losses from the control arm (smallest P-value = 0.11). Hence, although attrition was much greater than anticipated, it was non-differential.
|
Table 3 summarizes the results of testing the short-term effect of the intervention on quality of care. No statistically significant difference was observed between conditions at the end of the intervention, neither in the overall quality of care provided to residents (P = 0.86) nor in any of the subdimensions of care (smallest P-value = 0.26). To ascertain whether the intervention effect could have been delayed, analyses were extended to the second posttest. The two condition mean slopes that model change over time in care quality were not statistically different (P = 0.62). Non-significant results were also obtained when replicating this analysis on each of the six subdimensions of care (smallest P-value = 0.12). Statistically significant improvements (P < 0.01) were observed on 2 sub-scales of the QUALCARE Scale, for both the experimental and control conditions: environmental and medical management.
|
|
Discussion |
|---|
|
TopMethodsResultsDiscussionAppendixReferences |
|---|
In this study, attainment of quality improvement objectives, chosen in conjunction with the facility manager following a comprehensive evaluation of the care delivered, failed to translate into positive changes in the quality of care provided to frail elderly residents. Facilities that refused to participate in the study might have benefited from the intervention. In addition, our decision to include facilities that provided inadequate care to at least one resident might have rendered the intervention effect difficult to detect. Other possible explanations for the negative result of this trial include insufficient statistical power, methodological weaknesses, or a weak intervention.
Insufficient statistical power might have been suspected if there had been a trend toward an effect, but no such trend was observed. Quality of care tended to improve over the course of the trial, but to the same degree in the experimental and control conditions, which precludes attributing change to the intervention. Improvement in the quality of care delivered in the control facilities was unexpected and may reflect a contamination bias. The mere fact of involving these facilities in a quality improvement trial, and repeatedly measuring the quality of care they provided, may have induced positive changes in care practices. Instrument bias that results from modifying the way the outcome is measured over time is another possible explanation. Although the quality assessors were trained in using the QUALCARE Scale, it is possible that they gradually became more lenient in their assessments.
Methodological weaknesses might also explain the lack of evidence regarding the effect of the intervention. Group-randomized trials are challenging to design and analyse. This trial, however, satisfies the 10 recommendations made by Ukoumunne et al. [16] for designing and analyzing interventions delivered to clusters of individuals. One of their recommendations involves choosing among two designs for follow-up assessments: the cohort design, in which the same subjects are used at each measurement time, and the repeated cross-sectional design, in which a fresh sample of subjects is drawn from the clusters at each measurement time. In designing the study, we chose to reassess the same residents, knowing that the cohort design is potentially more powerful than the repeated cross-sectional design. In light of the high proportion of losses to follow-up, selecting a new set of subjects at each measurement time may have proven more powerful.
Lastly, the intervention may not have been strong enough for attainment of quality improvement objectives to induce positive changes in care quality, as measured by the QUALCARE Scale. The intervention was designed to assist facility staff in improving care, but it had no intended effect on staff size. Like a significant proportion of nursing homes, unregulated facilities are severely understaffed, at least in the province of Québec where this trial was conducted [27]. Several studies have found positive relationships between care quality measures and staffing levels, up to some threshold [3,28]. Like inadequate staffing, turnover is another significant problem in long-term care settings [3] that would tend to reduce the effect of any staff-focused intervention.
Exploratory analyses of clinical data suggest a fourth explanation for the intervention’s lack of effect. Residents’ cognitive abilities were measured at baseline and also at the end of the intervention to capture changes in their mental status over the intervention period. A mixed-model ANCOVA conducted on the 122 residents assessed at both time points shows a mean decrease in total 3MS scores among the experimental subjects (P < 0.01) whereas no change was observed in the controls (P = 0.50). The significance of this result follows from prior analyses showing that cognitive impairment was the strongest determinant of the quality of care provided to a resident [18]. Hence, the positive effect of the intervention on care quality, if any, may have been obscured by increased needs of the residents in the experimental facilities.
In short, this study failed to show improvement in the provision of higher quality care following the intervention. However, this does not exclude the possibility that the intervention improved other outcomes of interest, such as survival time, rate of preventable hospitalizations, transfers to higher care level institutions, satisfaction with care, and staff burden or turnover [29]. In light of the deleterious effects of suboptimal care on resident outcomes [30], variables that may be more sensitive to changes induced by the intervention should be investigated before discarding goal attainment scaling as a useful quality improvement strategy.
|
Appendix |
|---|
|
TopMethodsResultsDiscussionAppendixReferences |
|---|
|
The study was funded by an unrestricted grant from the Canadian Institutes of Health Research (#199909MCT-77944-CT). The first author holds a National Researcher’s Award from the Fonds de la recherche en santé du Québec. The authors express their deepest appreciation to the facility managers and residents involved in the study, as well as to the numerous research team members, including research coordinator Lise Messier, consultant Roger Laplante, members of the intervention teams, and the data collectors. Thanks are also extended to Dr. Allan Donner for his advice on how to obtain unbiased estimates of standard deviations in cluster-randomized trials.
|
References |
|---|
|
TopMethodsResultsDiscussionAppendixReferences |
|---|
- HarringtonC, Carillo H, Wellin V. Nursing facilities, staffing, residents, and facility deficiencies 1996 through. San Francisco: University of California, 2003.
- WeinerJM. An assessment of strategies for improving quality of care in nursing homes. Gerontologist 2003; 43: 19–27.
[Abstract/Free Full Text] - RantzMJ, Hicks L, Grando V et al. Nursing home quality, cost, staffing, and staff mix. Gerontologist 2004; 44: 24–38.
[Abstract/Free Full Text] - KiresukTJ, Smith A, Cardillo JE (eds.) Goal Attainment Scaling: Application, Theory, and Measurement. Hillsdale, NJ: Lawrence Erlbaum Associates, 1994.
- LewisAB, Spencer JH, Haas GL, DiVittis A. Goal attainment scaling: relevance and replicability in follow-up of inpatients. J Nerv Ment Dis 1987; 175: 408–418.[CrossRef][ISI][Medline]
- RockwoodK, Joyce B, Stolee P. Use of goal attainment scaling in measuring clinically important change in cognitive rehabilitation patients. J Clin Epidemiol 1997; 50: 581–588.[CrossRef][ISI][Medline]
- EvansDJ, Oakey S, Almdahl S, Davoren B. Goal attainment scaling in a geriatric day hospital. Team and program benefits. Can Fam Physician 1999; 45: 954–960.[ISI][Medline]
- GordonJE, Powell C, Rockwood K. Goal attainment scaling as a measure of clinically important change in nursing-home patients. Age Ageing 1999; 28: 275–281.
[Abstract/Free Full Text] - StoleeP, Zaza C, Pedlar A, Myers AM. Clinical experience with goal attainment scaling in geriatric care. J Aging Health 1999; 11: 96–124.
[Abstract/Free Full Text] - RockwoodK, Graham JE, Fay S, ACADIE Investigators. Goal setting and attainment in Alzheimer’s disease patients treated with donepezil. J Neurol Neurosurg Psychiatry 2002; 73: 500–507.
[Abstract/Free Full Text] - RockwoodK, Howlett S, Stadnyk K, Carver D, Powell C, Stolee P. Responsiveness of goal attainment scaling in a randomized controlled trial of comprehensive geriatric assessment. J Clin Epidemiol 2003; 56: 736–743.[CrossRef][ISI][Medline]
- BravoG, Dubois MF, Roy PM. Improving the quality of residential care using goal attainment scaling. J Am Med Dir Assoc 2005; 6: 173–180.[CrossRef][Medline]
- SchnelleJF, Cruise PA, Rahman A, Ouslander JG. Developing rehabilitative behavioral interventions for long-term care: technology transfer, acceptance, and maintenance issues. J Am Geriatr Soc 1998; 46: 771–777.[ISI][Medline]
- MurrayDM. Design and Analysis of Group-Randomized Trials. Oxford: Oxford University Press, 1998.
- ElbourneDR, Campbell MK. Extending the CONSORT statement to cluster randomization trials: for discussion. Stat Med 2001; 20: 489–496.[CrossRef][ISI][Medline]
- UkoumunneOC, Gulliford MC, Chinn S, Sterne JAC, Burney PGJ, Donner A. Evaluation of health interventions at area and organisation level. BMJ 1999; 319: 376–379.
[Free Full Text] - BravoG, Dubois MF, Charpentier M, De Wals P, Émond A. Quality of care in unlicensed homes for the aged in the Eastern Townships of Quebec. Can Med Assoc J 1999; 160: 1441–1445.[Abstract]
- BravoG, De Wals P, Dubois MF, Charpentier M. Correlates of care quality in long-term care facilities: a multilevel analysis. J Gerontol B Psychol Sci Soc Sci 1999; 54: 180–188.
- DuboisMF, Bravo G, Charpentier M. Which residential care facilities are delivering inadequate care? A simple case-finding questionnaire. Can J Aging 2001; 20: 339–355.
- DesrosiersJ, Bravo G, Hébert R, Dubuc N. Reliability of the functional autonomy measurement system (SMAF) revised for epidemiological study. Age Ageing 1995; 24: 402–406.
[Abstract/Free Full Text] - TengEL, Chui HC. The Modified Mini-Mental State (3MS) examination. J Clin Psychiatry 1987; 48: 314–318.[ISI][Medline]
- PhillipsLR, Morrison EF, Chae YM. The QUALCARE Scale: developing an instrument to measure quality of care. Int J Nurs Stud 1990; 27: 61–75.[CrossRef][ISI][Medline]
- PhillipsLR, Morrison EF, Chae YM. The QUALCARE Scale: testing of a measurement instrument for clinical practice. Int J Nurs Stud 1990; 27: 77–91.[CrossRef][ISI][Medline]
- BravoG, Girouard D, Gosselin S, Archambault C, Dubois MF. Further validation of the QUALCARE Scale. J Elder Abuse Negl 1995; 7: 29–48.[Medline]
- LittellRC, Milliken GA, Stroup WW, Wolfinger RD. SAS System for Mixed Models. Cary, NC: SAS Institute, 1996.
- WhiteIR, Thomas J. Standardized mean differences in individually-randomized and cluster-randomized trials, with application to meta-analysis. Clin Trials 2005; 2: 141–151.
[Abstract/Free Full Text] - BravoG, Charpentier M, Dubois MF, De Wals P, Émond A. Profile of residents in unlicensed homes for the aged in the Eastern Townships of Quebec. Can Med Assoc J 1998; 159: 143–148.[Abstract]
- SchnelleJF. Determining the relationship between staffing and quality. Gerontologist 2004; 44: 10–12.
[Free Full Text] - KaneRL, Flood S, Bershadsky B, Keckhafer G. Effect of an innovative medicare managed care program on the quality of care for nursing home residents. Gerontologist 2004; 44: 95–103.
[Abstract/Free Full Text] - BravoG, Dubois MF, De Wals P, Hébert R, Messier L. Relationship between regulatory status, quality of care and 3-year mortality in Canadian residential care facilities: a longitudinal study. Health Serv Res 2002; 37: 1181–1195.[CrossRef][ISI][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  T. Perneger Ten reasons to conduct a randomized study in quality improvement Int. J. Qual. Health Care, December 1, 2006; 18(6): 395 - 396. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||