OUP user menu

Studies pertaining to the ACOVE quality criteria: a systematic review

Marjan Askari, Peter C. Wierenga, Saied Eslami, Stephanie Medlock, Sophia E. De Rooij, Ameen Abu-Hanna
DOI: http://dx.doi.org/10.1093/intqhc/mzr071 80-87 First published online: 2 December 2011


Purpose To identify and uniformly describe studies employing the Assessing Care Of Vulnerable Elders (ACOVE) quality indicators within a comprehensive thematic model that reflects how the indicators were used.

Data sources A systematic search of MEDLINE, EMBASE and CINAHL was conducted.

Study selection English-language studies meeting our criteria published prior to January 2010.

Data extraction Included studies were analyzed and described by two independent researchers.

Results of data synthesis A total of 41 articles met our selection criteria. Studies were classified into the themes ‘Application of indicators’ (32 studies) and ‘Analysis and development of indicators’ (13 studies). ‘Application’ studies included assessing quality of care, influencing behavior of health professionals and examining the association of quality of care with other factors. ‘Analysis and development’ included studies developing new indicator sets, and those adapting and validating the original quality indicators to new settings.

Conclusions The indicators were used in a wide range of applications with two main foci: the assessment of quality of care for elderly patients, and investigating the feasibility of similar indicators and their adaptation to new settings. Very few of the studies published to date have addressed the goal of care improvement. We foresee an important role for application of indicators that proactively help health-care professionals to deliver the right care at the right time, for example by resorting to decision support systems.

  • quality indicators
  • quality of care
  • elderly
  • vulnerable elders
  • clinical decision support systems


In recent years many studies have been dedicated to the care of elderly patients. The effects of multimorbidity, polypharmacy and the overall quality of care have been investigated [15]. Care for elderly patients is complex and not yet well understood [6]. Not only are elderly patients often excluded from clinical trials, but also due to their multimorbidity a multitude of possibly conflicting guidelines are contemporaneously applicable to them [7].

Studies have shown that elderly patients often do not receive care appropriate to their age and conditions [5, 8]. The vulnerable elders, defined as the group of persons 65 years of age or older who are at high risk of death or functional decline, form an important subgroup for investigation [9].

To improve care for elderly patients there is a need to know where, when and for which conditions deficits exist, calling for reliable and comprehensive methods for the assessment of quality of care that considers both medical and geriatric conditions [1012]. Many of the current methods are not intended to be comprehensive, but focus on a specific process of care or on the assessment of the quality of a treatment for one condition [11, 1315]. In addition, many methods tend to be subjective, meaning that they depend to a large extent on the implicit knowledge and experience of the assessor, thus jeopardizing inter-rater reliability. Unlike subjective methods, objective methods consist of explicitly specified assessment instruments and are often based on literature review and expert consensus, and are therefore more reliable [12]. However, most explicit methods for the assessment of the quality of care of elderly people are not comprehensive.

In 2000, researchers at research and development (RAND) and university of California, Los Angeles (UCLA) developed the Assessing Care Of Vulnerable Elders (ACOVE) quality indicator  set [9, 16]. This set consists of explicitly phrased IF–THEN clinical rules with comprehensive coverage of general medical and geriatric conditions. They are intended to evaluate, by means of gauging adherence to the rules, whether the care being delivered at the level of the health-care system meets pre-specified standards of quality. Assessment is meant to inform and, in consequence, to facilitate quality improvement efforts [9, 16]. The rules are based on evidence and expert opinion, and describe process rather than outcome measures. The rules also specifically address undertreatment that is often overlooked in the elderly patient population. Due to these properties, the ACOVE quality indicator set has a unique place amidst screening and assessment methods for measuring the quality of care of elders, especially the vulnerable ones.

This paper reviews the decade of research pertaining to the ACOVE quality indicators. The objective was to identify and summarize all studies published after the introduction of the ACOVE quality indicator sets in the literature. The studies are described in a thematic conceptual model meant to understand the different ways in which the ACOVE quality indicators have been used and to expose areas of promising future research.


Data sources

Relevant English-language articles published between the presentation of the ACOVE-1 initiative in 2001 and the end of January 2010 were searched in multiple databases (MEDLINE (using Scopus and PubMed), Ebsco-CINAHL and Ovid-EMBASE) by using the query ‘ACOVE OR (‘assessing care’ AND (vulnerable OR frail*))’.

Study selection

Articles were included if they used the original ACOVE quality indicators or adaptations, updates or extensions thereof. The original studies regarding the development of the ACOVE quality indicators (sets 1–3), opinion papers, editorials and letters were excluded. Congress abstracts were also excluded because they often provide limited details.

Two reviewers independently examined the collected studies in two rounds. The first round consisted of critically reading the title, keywords and abstract. In the second round both reviewers independently assessed the full text of the articles selected in the first round. One investigator screened citations to identify additional possible candidate articles.

Disagreements in each round between the two reviewers were resolved by consensus. In the cases when the reviewers were unable to reach consensus a third reviewer was involved to make a final decision. Inter-rater agreement has been calculated using Cohen's kappa.

Data extraction

From the selected studies, the two reviewers independently used a structured form for abstraction to obtain the study characteristics, objectives, methods, affiliation of the authors and research group, and the number and focus of the quality indicators. Based on the ACOVE project's intended objectives [9], the studies were provisionally organized into two main themes: ‘Implementation’ (how and to what extent were quality indicators applied for assessment and improvement of care) and ‘Development’ (adaptation and extension of quality indicators). Based on a bottom-up analysis of the study objectives and quality indicator application, sub-categories were identified. These sub-categories were then organized into larger categories and put into a thematic conceptual model.

Results of data synthesis

Figure 1 shows the article selection flow diagram. From the 347 papers, 50 were selected for full text screening. A total of 41 articles met our selection criteria. Inter-rater agreement was high (kappa: 0.73, only five papers necessitated the involvement of the third reviewer). Analyzing the studies’ objectives, methods, contents and types of quality indicator application, we arrived at five categories within the main themes ‘Application of quality indicators’, and ‘Analysis and development of quality indicators’. This is shown in the thematic conceptual model (Fig. 2). To get insight into the nature of these studies, we describe the most important findings per category.

Figure 1

Article selection flow diagram.

Figure 2

Thematic conceptual model of studies pertaining to the ACOVE quality indicators. There are five categories in the conceptual model pertaining to the ACOVE quality indicators (QIs): Cat A, studies that develop a new set of quality indicators; Cat B, studies that adapt the original ACOVE quality indicator set to a new setting; Cat C, studies that assess the quality of care; Cat D, studies that examine the association between the quality of care and other factors (such as health-care outcomes, patient opinion and patient characteristics); and Cat E, studies that aim to influence the behavior of or educate health-care professionals. Note: Some studies had two or more goals, and could therefore be assigned to more than one category. There were 41 distinct studies, 13 ‘Analysis and development’ studies and 28 studies that applied the quality indicators. *Outcome measures: quality indicator pass rates. **Examples of investigated associations or used outcome measures: patient's observed vs. expected overall quality score; effect of quality of care on survival; effect of number of morbidities on pass rates. ***Outcome measures: scores of knowledge before and after receiving the interventions, quality indicator pass rates.

Application of quality indicators

Assessing the quality of care

Eighteen studies [5, 1733] in this category assessed the quality of different types of care. Eight articles pertained to the assessment of care for a specific condition: management and detection of pain [18, 29], falls and instability [20], congestive heart failure care [21], osteoarthritis [24], pressure ulcer care [26] and urinary incontinence [17, 28]. Ten studies focused on specific domain(s) of care or overall quality: pharmacologic care [27, 30] and appropriateness of prescribing/underuse [22], quality of hospital care [31], geriatric care [19, 33] and overall quality of care [5, 23, 25, 32].

From these 18 studies, 4 studies focused on nursing home residents [17, 18, 26, 32], 5 on managed care plans [5, 19, 2830], 2 on hospitalized patients [22, 31] and 5 on primary care patients [20, 23, 25, 27, 33]. Two studies had mixed settings [21, 24]. The patient population in these studies mainly consisted of patients aged 65 or older (defined by a threshold age range of minimal 50 to maximum 75 years). Only seven studies explicitly mentioned that the population consisted of vulnerable elderly patients, all of which used the Vulnerable Elders Survey (VES-13) to identify vulnerability.

Two studies in this category explicitly mentioned the inclusion of patients aged 50 years or older.

Overall, between 3 and 207 quality indicators were used in the 18 studies. When viewed per condition, there were between 1 and 43 quality indicators used. While 8 studies used the original ACOVE quality indicators [5, 20, 22, 24, 2830, 33], 10 studies used adapted ACOVE quality indicators or newly developed ACOVE-like quality indicators [1719, 21, 23, 2527, 31, 32].

The source of data used to evaluate the quality indicators was in most cases a combination of medical record data and interviews with caregivers and patients. One study used a combination of medical records, direct observation and electronic measurement [26]. Two studies assessed the quality of care with interviews only [24, 25], two studies by medical records only [21, 31] and three studies utilized administrative data [23, 27, 32]. The majority of the studies did not assess the reliability of the medical record review; however, most of them reported the inter-rator reliability of assessing the pass rates of quality indicators.

In Supplementary material, Appendix B, all studies and their main findings are summarized and in Table 1 a brief summary of characteristics of the included studies is provided.

View this table:
Table 1

Brief summary of characteristics of the included studies

CharacteristicsNumber of studies (percentage)
Settings (or intended setting)
 Primary care14 (34)
 Nursing home9 (22)
 Hospitals4 (10)
 Managed care5 (12)
 Others9 (22)
 VE11 (27)
 ≥754 (10)
 ≥6518 (44)
 ≥553 (7)
 Others5 (12)
Condition focus
 Single condition28 (68)
 Multiple conditions13 (32)
  • We only categorized populations as VE, if the article explicitly mentioned the term ‘vulnerable elderly/elders’ in the text. VE: vulnerable elderly.

Association between quality of care and other factors

Twelve studies fell into this category [3445]. The association between quality of care and the following factors were studied: multimorbidity [39, 41], survival [34], functional decline after discharge [35], time consumption of care processes [40], osteoarthritis severity [38] and race [36, 37].

Four studies had a more indirect approach to examine an association with the quality of care: three studies assessed whether care improvement on a specific care type targeted by the quality indicators also influenced the quality of non-targeted types of care [4244] and the fourth study examined the effect of an intervention that improved care for falls, incontinence and cognitive impairment in community-dwelling patients on nine non-targeted conditions [45].

In the 12 above-mentioned studies between 9 and 236 quality indicators were used. Four studies used adapted ACOVE quality indicators [35, 38, 43, 44], five studies analyzed existing data collected with the original ACOVE quality indicators in the Wenger et al. study [34, 3942], and three studies utilized ACOVE-2 indicators [36, 37, 45].

In most cases, the combination of medical record data and interviews with caregivers and patients was necessary for scoring the quality indicators [3436, 3942]. Two studies used a combination of medical records, direct observation, interviews and electronic measurement [43, 44]. Three studies assessed the quality of care using only medical records [37, 38, 45].

The patient population in eight studies consisted of patients aged 65 or older [34, 35, 3944] of which six focused on vulnerable elderly [34, 35, 3942], one on patients of 75 years and older [45], two on 60 years and older [36, 37] and one on 55 years and older [38] (see also Table 1).

Two studies focused on nursing home residents, seven on community-dwelling patients [34, 36, 37, 3942], one on hospitalized patients [35] and two studies focused on patients in primary care [38, 45]. All study characteristics are shown in Supplementary material, Appendix B.

Influence the behavior of or educate health-care professionals

This category contains two studies. The first study implemented a pharmacotherapist-led educational intervention that consisted of a theoretical presentation and a knowledge test, both based on 30 pharmacology-related quality indicators [46]. The second study used a practice-based intervention based on ACOVE quality indicators in primary care that included case finding, physician education and practice efforts to improve the quality of care for falls and urinary incontinence [33]. In this study 18 quality indicators were used for the assessment of the quality of care.

Analysis and development of quality indicators

Developing new ACOVE quality indicators

One study by McGory et al. [47] described the development of a completely new set of 76 quality indicators for vulnerable older patients undergoing abdominal operations. An expert panel rated and discussed the indicators using a modification of the RAND/UCLA method, which was used to develop the original ACOVE quality indicators.

Adaptation of quality indicators to a new setting and validation

Twelve studies were classified into this category [19, 26, 31, 4856]. Three studies reported on the translation of ACOVE quality indicators to another country. Steel et al. [49] successfully translated ACOVE quality indicators to the UK in order to assess the quality of primary and secondary care for elders using patient surveys. Kroger et al. [54] adapted a selection of 82 quality indicators for use in Canada to assess the quality of care of elderly people with cognitive impairment or dementia. van der Ploeg et al. [55] reported on the adaptation and validation of ACOVE-3 quality indicators in the Netherlands for use in general practice care quality assessment for vulnerable elders.

The remaining studies adapted and validated quality indicators within the same country to another health-care setting, to other patient populations or to other conditions.

Eight studies adapted the ACOVE quality indicators for the following purposes: assessment of quality of care for geriatric conditions [50], residential care [51], general medical conditions [53] and pressure ulcer care [26] in nursing homes, for home-based primary care [48], for geriatric and general hospital care [31], for care for osteoarthritis, rheumatoid arthritis and analgesics use [52] and for community-dwelling patients with advanced dementia and poor prognosis [19].

The final study classified into this category reported the validation, and not the adaptation of a quality indicator set. The quality indicator set, which had already been adapted to the nursing home setting, was validated in terms of measurement feasibility utilizing two data sources (medical record data and administrative data) [56].

Of the 11 studies that adapted a quality indicator set, 9 studies used the modified RAND/UCLA (Delphi) method with the help of content experts [19, 4851, 5356]. Two studies used a different approach: one used an expert panel to select quality indicators but did not mention how this selection process was conducted (e.g. whether it was based on Delphi rounds or not) [31], and the other study was not explicit on the quality indicator adaptation method or the professionals concerned [26].

The reasons for discarding or adapting the quality indicators for use in a new setting or country were varied, for example being inapplicable to that country or setting due to other guidelines, disagreement in the reported evidence, shortening or extending the follow up period or continuity of care, difference in recommended treatment and changing the medication options.

Involvement of ACOVE study group members

Thirty-one studies (76%) of the included 41 were conducted in collaboration with one or more representatives of the original ACOVE study group. Ten studies (24%) were performed without (mentioned) support of the ACOVE group.


A strong increase in the number of ACOVE-related studies was exhibited in recent years. In this systematic review we identified and summarized 41 relevant research papers pertaining to the ACOVE quality indicators. The studies were organized in a conceptual model containing five main categories providing a better understanding of where and how ACOVE quality indicators have been applied since 2001. Most research originated from the ACOVE group itself but there are some translational efforts to other countries. The efforts to collect data in order to assess care are substantial and there is paucity, in studies addressing quality-of-care improvement.

Our systematic literature search was designed to give a complete overview of the studies pertaining to the ACOVE quality indicators. Although our conceptual model for categorizing studies was based on the original goals of the ACOVE initiative and on a bottom-up analysis of the articles that were found, it is possible that other researchers in a comparable process would define other categories. We hope that the organization chosen will prove useful for researchers to identify studies relevant to them and to put them in perspective. A future study is needed to report on the formal quality of the included studies, and the overall quality of care as assessed using the ACOVE quality indicators.

The ACOVE quality indicators were used in several care settings (ranging from primary care to hospital care, from pharmacologic care to residential care), for multiple conditions (from all ACOVE conditions to specific conditions like osteoarthritis or focusing on general medication use), and in several different elderly patient populations (from community-dwelling patients to nursing home residents). Our results showed that the concept of using ACOVE-like quality indicator has been extrapolated only to a limited extent to other patient populations than the elderly. Although the ACOVE set was developed for vulnerable elderly patients, the majority of the studies did not distinguish between the vulnerable elders and the general elderly population. This could be due to the difficulty of identifying who is vulnerable (the VES-13 considers age, self-rated health, limitations in physical function and functional disabilities).

In most studies, the combination of patient record review and interviews was used to extract the data. Only few studies used automated data extraction methods, because the required data are often unavailable, hard to access or difficult to standardize. Therefore, electronic capture of ACOVE-related data elements and facilitating their extraction forms important future work. Since evaluating the quality indicators imply laborious data collection activities, future work for care assessment and improvement will be considerably facilitated once the measurement systems are in place.

The distribution of studies over the model's categories showed that quality indicators were mainly used in two categories: (i) those that applied the quality indicators for the assessment of the quality of care and (ii) those that examined whether an association existed between the quality of care and other factors (such as health-care outcomes, patient opinion and patient characteristics). Only one study in our review addressed the (positive) association between quality indicator performance and survival among community-dwelling vulnerable older adults. This association has also been addressed in a very recent study, not reviewed here, in which better quality indicator performance was associated with lower likelihood of death 1 year after discharge in hospitalized seniors [57]. Furthermore, our model showed that various studies were aimed at adapting the original ACOVE quality indicator set to a new setting.

Our results suggest two opportunities for additional work on quality indicator application. First, only one study developed a completely original indicator set that consisted of quality indicators with content not based on ACOVE indicators. This may be related to the difficulty of developing a new quality indicator set. Simply translating and adapting the comprehensive ACOVE quality indicators may prove sufficient in the same patient group. Translation between the USA and other countries has been shown to be possible, which theoretically facilitates the comparison of results between different countries [49, 54, 55]. Second, in contrast to the abundance of studies that used the quality indicators for retrospective quality assessment by screening medical records, only two studies applied the quality indicators in a proactive manner to directly improve the delivered care [33, 46]. Our results suggest that the ACOVE framework has mainly been used to assess care, rather than to achieve the ultimate goal of the quality indicators, that of improving the quality of care, although it is possible that such initiatives are not always published. This assessment takes place after care has been delivered and often forms a painstaking, costly process typically requiring the examination of patient charts and interviewing patients or health-care workers. We believe that by decoupling assessment from improvement, crucial opportunities to improve care at the right time and the right place may be missed. Important future work consists of attempting to switch from using quality indicators solely as an assessment instrument, to using quality indicators as the basis for change of care delivery by providing timely and proactive feedback to the care givers. Computerized clinical decision support systems may play an important role in this vision.

Until now most research originated from the ACOVE group itself and specifically from the USA. We found six studies (five in Europe and one in Canada) that used the ACOVE indicators suggesting that they may be increasingly used outside of the USA. In sum, the ACOVE quality indicator set has formed the basis for many studies inspiring various applications and it holds promise of forming a common ground for aligning diverse research efforts. However, the gap between published studies on measuring and improving quality of care is still large.


This work was supported by ZonMw (The Netherlands Organization for Health Research and Development) by grants for the PROFIT (#300020010) and ICOVE (#311020302) projects.


A.A.-H., M.A. and S.d.R. conceived the preliminary study design. A.A.-H. supervised and integrated the whole work. M.A. and P.W. carried out the literature search, and performed acquisition, examination and analysis of the data. S.d.R. and A.A.-H. validated the data acquisition. M.A. and P.W. drafted the manuscript. All authors participated in the design of the paper, the regular discussions, and read and approved the final manuscript. No other persons contributed to this project. The authors declare that they have no conflicts of interest.


View Abstract