OUP user menu

Performance measurement and improvement frameworks in health, education and social services systems: a systematic review

Anne Klassen, Anton Miller, Nancy Anderson, Jane Shen, Veronica Schiariti, Maureen O'Donnell
DOI: http://dx.doi.org/10.1093/intqhc/mzp057 44-69 First published online: 1 December 2009

Abstract

Purpose To perform a systematic review, supplemented by a targeted grey literature scan, for performance measurement and improvement frameworks within and across the health, education and social service systems. The intended outcome was the creation of a foundation of evidence to inform the development of cross-sectoral quality improvement frameworks.

Data sources MEDLINE, CINAHL, PsycINFO, ERIC, EMBASE, Social Services Abstracts, Social Work Abstracts and Education Index Full Text were searched up to April/May 2007. In addition, 26 governmental and 27 organizational websites were searched.

Study selection English language material with a publication date of 1986 or more recent that described a health, education or social services multidimensional framework for performance measurement and improvement.

Data extraction The framework name; administrative sector; level of application; setting; population of interest; categories of quality described within the framework; country of application; and citations to other performance measurement and improvement frameworks were extracted from each article.

Results In total, 111 frameworks were identified. Most frameworks (n = 97) were developed in or for the health sector. A concept sorting exercise identified 16 quality concepts applicable across many settings, sectors and levels of application.

Conclusion This systematic review of quality domains will be relevant and useful to those who are developing and using performance measurement and improvement frameworks for adult and child populations within or across the health, social service or education sectors.

  • quality measurement
  • quality indicators
  • healthcare system
  • child
  • special needs
  • chronic disease

Background

The development and implementation of quality improvement frameworks and performance indicators for services and systems that contribute to the health of individuals and populations is of enormous interest to decision-makers and policy-makers in the present era of accountability, cost-effectiveness and quality improvement [1]. Substantial progress has been made towards developing frameworks and indicators for people requiring medical and nursing services delivered in institutional settings such as hospitals. Adults and children with complex, chronic health conditions, on the other hand, may receive much of their care in community settings, and from an array of service providers from various sectors and agencies. For children with special needs and their families, for example, such services usually go beyond the traditional health services to include early intervention and (re)habilitation, specialized therapies and equipment, special education and other supportive measures provided in homes, schools and community facilities, as well as family support services, with responsibility for service provision falling under different administrative sectors.

As a foundation for the development of a cross-sectoral framework for quality improvement for persons with special healthcare needs and complex chronic health conditions, we performed a systematic review of peer-reviewed literature, and a targeted scan of grey literature, to identify frameworks developed for quality improvement in three service sectors, i.e. health, education and social services. The search was intentionally international and comprehensive in order to identify frameworks in all three sectors for different levels of application (e.g. program, organization, system), different settings (e.g. public health, women's health system) and different populations (e.g. adults, children). The aim of this paper is to present findings from the comprehensive literature review of frameworks and a concept sorting exercise completed to determine the quality categories embedded within them.

Methods

Systematic review

Given the diffuse nature of our topic, and the fact that quality improvement terminology differs across service sectors, key articles, websites, databases and thesauri relevant to each sector were reviewed, and six concepts identified for in-depth searching (see Table 1). The following databases were then searched up to April/May 2007: MEDLINE, CINAHL, PsycINFO, ERIC, EMBASE, Social Services Abstracts, Social Work Abstracts and Education Index Full Text. The five larger databases (i.e. MEDLINE, CINAHL, PsycINFO, ERIC and EMBASE) were searched using thesaurus terms and keywords. The remaining databases were searched by keyword (subject headings proved too broad to be useful). The search strategies used are available upon request. Reference lists of included studies were also reviewed to identify citations to performance measurement and improvement frameworks.

View this table:
Table 1

Search concepts and example of search terms

Search conceptExample of search terms
Known quality improvement type frameworksBalanced scorecard; dashboard; report card; value compass; strategy maps; performance prism; European Foundation for Quality Management; plan-do-check-act; Baldrige; total quality management; continuous quality improvement; six sigma
Quality improvement termsBest practice; quality dimension; quality assessment; quality framework; performance measure; quality tool; quality of service; quality of care; organizational quality
Performance indicators termsQuality indicator; process indicator; outcome indicator; indicator-based performance
Multi-agency coordination termsContinuity of care; medical home; integrated service delivery; multi-agency integration; key worker; care coordination
Terms relevant to children and youth with special healthcare needsDisabled children; chronic disease; rehabilitation; special needs; special educational needs; special healthcare needs; complex needs
Service sector termsEducational model; social care; social support

Preliminary searching of the six concepts led to high recall but low precision. To increase precision, a refined search strategy was developed whereby an article had to include at least two of the six search concepts. While this proved to be an effective strategy for the smaller databases, the recall was still high for the large databases. An additional filter was therefore developed for the larger databases to include only articles that also contained specified terms for frameworks or indicators. Because this filter may have excluded some relevant articles, we reviewed citation lists of included articles to identify frameworks that may have been overlooked.

Screening was performed by two screeners who worked independently. A review coordinator resolved any conflicts. Title, abstract and full text of papers were screened and included articles were those published in English with a publication date of 1986 or more recent that described, with sufficient detail to permit data extraction, a health, education or social services framework with at least two categories of quality (see Table 2 for glossary of common terms used in this paper, such as framework and category). Because much variation was found in what authors refer to as quality improvement type frameworks, we more specifically defined such frameworks as a model that specifies elements that can and should be measured and monitored in order to ensure a systematic process of improving quality of services. This conception was incorporated into our screening criteria by asking ‘Does the article clearly focus on and is it likely to provide details of a framework (system of measurement, scorecard, etc.) for measuring, evaluating or tracking performance, quality of care/services?’ Data from included papers were extracted by one reviewer and the extracted information was checked by a second reviewer.

View this table:
Table 2

Glossary of terms

TermDefinition
Quality improvementA systematic data-guided activity designed to bring about improvements in organizational processes or products. In the present context, this refers to improvements in the delivery of services in particular settings.
FrameworkA conceptual device to provide structure to a set of ideas, values or conceptual entities, and which usually contains two or more domains or groupings of items.
Performance measurement and improvement frameworkA model that specifies elements that can and should be measured and monitored in order to ensure a systematic process of improving quality of services.
Quality domain or categoryTerms used to represent what we call a category, i.e. a part of a quality improvement framework under which indicators to measure performance or quality can be grouped.
Quality or performance indicatorAn explicitly defined and measurable item that can be used to evaluate and monitor the status of organizational or system processes or performance.
Children and youth with special needs (CYSN)Children and youth with disabilities and/or complex chronic health conditions.

Targeted grey literature scan

The systematic review results revealed that most identified frameworks focused on the health sector. The grey literature search was therefore weighted towards identification of additional frameworks specifically developed for education and social services (see Appendix 1 for list of websites). Websites that contained frameworks already obtained through the systematic review, and the review of citations, were excluded. For grey literature, we included any material published after 1999 describing a performance measurement and improvement framework that had not already been captured in the systematic review. For websites that offered a search capability, the following terms were used: balanced scorecard (included because of its prominence [2]); quality framework; conceptual framework; performance framework; quality model; performance indicator; quality indicator; performance measurement and quality of care. For sites that did not have a search capability, all relevant sections of the website were examined. For government sites, we searched ministries for health, education and social services separately. Twenty-six governmental and 27 organizational websites, identified by the research team, were searched. The full text of potentially relevant articles was examined by two reviewers. Conflicts were resolved by consensus.

Data extraction and concept sorting exercise

Appendix 2 describes the information extracted about each framework identified through both the bibliographic and grey literature searches.

To organize the information about each framework's component parts, we performed a concept sorting exercise. This procedure involved the organization of brief verbal statements, using a group approach [3, 4]. The inclusion of multiple perspectives (our multidisciplinary team included six members) and use of a consensus approach were methods used to ensure rigour. The concept sorting exercise involved several steps. First, for each primary framework, we extracted and transcribed onto cards information on all explicitly named or implied categories of quality. Second, over the course of several meetings, the team reviewed and considered the content of each card, and, by a process of consensus, placed the card into one (or more if appropriate) concept piles. After all cards were sorted, an iterative process that involved checking and rechecking cards assigned to each concept grouping (including those set aside because they did not appear to represent a quality category or did not provide enough information to categorize) ensured that all cards were correctly assigned. At the end of the process, the publications from which the cards were created were re-read, to validate the interpretation of the meaning of each category. Finally, a descriptor and a definition were agreed upon for each concept pile, summarizing the concept contained in that grouping.

Results

Our search results appear in Fig. 1. From the 26 853 citations retrieved and reviewed, the full text of 594 articles was examined of which 64 articles described a multidimensional framework that contained sufficient details for data extraction. A review of citations led to the identification of 19 additional articles. The grey literature search added an additional 27 articles. From 110 included articles, we identified 111 frameworks. These frameworks varied in complexity, from simple frameworks designed to measure only a few categories of quality, to complex models in which characteristics of health system performance (e.g. safety, effectiveness, efficiency) were placed within a broader context that included measures of health status, non-medical determinants of health, community and health system characteristics (e.g. Canadian health indicator framework [5]).

Most frameworks were developed by borrowing from existing frameworks or approaches to quality improvement. The Institute of Medicine two-dimensional framework [6], for example, which measures healthcare quality and reflects consumer perspectives on healthcare needs or reasons for seeking care, appeared in seven frameworks [713]. The most common identified framework was the BSC. While most BSCs were applied in a healthcare setting, we found one BSC designed for education [14]. BSC frameworks were found at all levels of application, and applicable to a broad range of topics. We found 32 examples of the BSC framework [1444], 9 [1422] of which used Kaplan and Norton scorecard's [45] traditional quadrants—financial performance, customer satisfaction, internal processes and learning and growth. Ten frameworks within our review used the triad of structure, process and outcome, initially expostulated by Donabedian [46], as an organizational feature [4756].

Our search identified 97 frameworks developed in (or for) the health sector [58, 1013, 1544, 4750, 52105]; eight for education [14, 51, 106111]; one for social services [112] and five [9, 113116] that were multi-sector frameworks applied to both health and social services. We found 54 frameworks that were applicable at the system level [513, 21, 22, 32, 3439, 43, 44, 47, 50, 5254, 5658, 60, 63, 69, 70, 75, 77, 79, 86, 88, 89, 9295, 9799, 104, 108, 110, 111, 113116], 33 that were organizational/institutional level [19, 20, 2431, 33, 41, 42, 48, 59, 62, 64, 67, 68, 71, 73, 76, 80, 83, 84, 87, 91, 100, 102, 105, 107, 109, 112] and 24 that were individual program/unit/department level [1418, 23, 40, 49, 51, 55, 61, 65, 66, 72, 74, 78, 81, 82, 85, 90, 96, 101, 103, 106]. For two sectors (health and education) there were multiple frameworks at each level of application. The sole social services framework was an institutional level framework for residential care for people with intellectual disability [112]. Of the five multi-sector frameworks [9, 113116], all were system level frameworks and four [113116] were relevant to people with disabilities.

The 111 frameworks covered a broad range of settings. Settings where more than five frameworks were identified included health systems (n = 33) [58, 1013, 21, 32, 3436, 47, 50, 5658, 60, 63, 69, 70, 75, 77, 79, 86, 88, 9395, 97, 98, 116], hospital or health organizations (n = 17) [19, 20, 2628, 33, 48, 59, 62, 67, 73, 76, 80, 84, 91, 100, 105], population, public health or community health (n = 10) [9, 22, 25, 30, 44, 52, 53, 99, 114, 115], mental health or substance abuse addictions (n = 9) [37, 43, 61, 64, 68, 71, 89, 96, 113], education, training or daycare (n = 8) [14, 51, 106111] and women's health (n = 7) [18, 39, 55, 72, 74, 81, 90]. Frameworks were found that were relevant to a variety of different populations, including women, veteran or the elderly. Fourteen frameworks were applicable to children and youth [11, 12, 14, 47, 51, 66, 100, 106111, 115].

Concepts represented in the frameworks

The most notable finding in our study was that across a heterogeneous listing of frameworks developed for different purposes and applied in different sectors, levels of application, settings and populations of interest, we were able to identify, through our concept sorting exercise, a set of common quality concepts. A total of 1488 cards were sorted into 16 unique concept groups, which we used to describe salient aspects of quality from the 88 primary frameworks. The 16 concept groups could be categorized under five thematic headings, which we termed: collaboration; learning and innovation; management perspective; service provision; and outcome. Table 3 shows the thematic headings, each quality concept, its definition and proportion of frameworks at each level that included each concept. It can be noted from the definitions that each quality concept was composed of a number of subthemes that our team determined, through the concept sorting exercise, grouped together conceptually. For example, the quality concept availability/accessibility had the following subthemes: geographic accessibility, physical accessibility, financial accessibility (affordability), availability and wait times/timeliness.

View this table:
Table 3

Quality concept groupings and their frequency by level of application

DefinitionaLevel 1b(n = 16)Level 2b(n = 24)Level 3b(n = 48)TOTAL (n = 88)
Collaboration
 Collaboration: Linkages and partnerships that are established among different service delivery systems, networks and providers to enable effective planning and the implementation of a high-quality integrated service.38382531
Learning and innovation
 Creating and using knowledge: Extent of commitment to a continuous learning environment that supports research, the development, dissemination and exchange of information and knowledge, and the creation and application of evidence-based practice guidelines and standards.38382933
 Quality improvement processes: System of processes and activities that promote continuous quality improvement, including those related to program evaluation, performance monitoring, accountability and accreditation.44503842
 Workforce development/support: Degree to which a service delivery system manages and develops its workforce through a variety of processes (e.g. provision of a supportive and safe work environment, providing opportunities for learning and growth, valuing staff satisfaction).56502538
Management perspective
 Leadership/governance: How a service or organization engages in strategic planning processes to develop and facilitate the achievement of its values, mission and vision, to be responsive to change, and to achieve sustainable success.19542532
 Infrastructural capacity: A service or organization infrastructure, e.g. physical facilities, equipment, fiscal resources, human resources and information technology.50715458
 Business/financial management: Extent to which business and financial management plans are developed, implemented and monitored, and address issues such as efficiency, resource allocation, stability and process management.63838178
Service provision
 Equity/fairness: Provision of services that are of equal quality and that are distributed fairly across populations, regardless of client characteristics (e.g. gender, ethnicity, impairment, geographic location, socioeconomic status).1943825
 Availability/accessibility: Extent to which services are available and accessible (geographical location, physical environment, affordability) in a timely manner.69507567
 Comprehensiveness: Extent to which there is a range of services provided that is broad enough in scope and depth to meet client needs.00106
 Appropriateness: Services represent a ‘best fit’ with client needs, and are based on established standards and evidence.44254036
 Client centredness: Extent to which the planning and delivery of services involves clients, provides them with information to support their decision-making, and is positive, acceptable and responsive to their needs and expectations, and respectful of privacy, confidentiality and differences (e.g. cultural, socioeconomic).75586766
 Coordination: Provision of services that are continuous, integrated and organized around the client, both within and across service settings and over time.56213535
 Client safety: Any actual or potential harm to the client, through the provision of a service, that is prevented, avoided, reduced or minimized through integrated risk management activities.13464640
Outcome
 Effectiveness: Extent to which a service achieves the desired result(s) or outcome(s), at the client, population or organizational level.44638572
 Client perspective: Extent to which the client perspective and experience of a service is measured and valued as an outcome of service delivery.50463842
  • Values are represented by percentages.

  • aEach definition represents a synthesis of the varying definitions from all cards sorted into each of the 16 unique concept groups.

  • bDefinitions of levels: Level 1: frameworks for specific services, program, units or departments (that usually exist within an organization or institution such as a department within a hospital or a team within a school district), that represent direct interface with clients. Level 2: frameworks for organizations or institutions that comprise individual programs, units or departments (e.g. hospitals; school districts). Level 3: frameworks for systems that comprise a number of organizations/institutions (e.g. national health system; provincial education system).

We explored the relationships between the 16 quality concepts and level of application to identify whether certain aspects of quality were more or less often included (see Table 3). With the exception of ‘comprehensiveness’, which was evident only in system level frameworks, the remaining 15 concept groups were present at all levels of application, albeit unevenly. While some quality concepts were included in more than 50% of frameworks at all three levels of application (i.e. infrastructural capacity; business/financial management; availability/accessibility; client centredness), the inclusion of other aspects of quality appeared to depend to some extent on the level of application. The concept group ‘equity/fairness’, for example, was included in 38% of system level frameworks, but only 4% of organizational level frameworks. Similarly, ‘coordination’ was included in 56% of individual program/unit/department level frameworks, but only 21% of organization/institution level frameworks. Finally, ‘leadership/governance’ was included in 54% of organization/institution level frameworks, but only 19% of individual program/unit/department level frameworks.

Table 4 shows the presence or absence of each concept group for each of the 88 primary frameworks. While there are clearly some prominent performance measurement and improvement frameworks, such as the BSC, our review also identified many miscellaneous frameworks.

View this table:
Table 4

Quality concept groupings by type of framework (n = 88)

FrameworkQuality concept groupings
CollaborationLearning and innovationManagement perspectiveService provisionOutcome
CollaborationCreating and using knowledgeQuality improvement processesWorkforce development/supportLeadership/governanceInfrastructural capacityBusiness/financial managementEquity/fairnessAvailability/accessibilityComprehensivenessAppropriatenessClient centrednessCoordinationClient safetyEffectivenessClient perspective
BSC: traditional four-quadrant framework
 BSC (traditional)a [14]XXXX
BSC: adaptation of traditional framework
 BSC (variation) [23]XXX
 BSC (variation) [24]XXXXXXXXXXX
 BSC (variation) [25]XXX
 BSC (variation) [26]XXXXXXXXXX
 BSC (variation) [27]XXXXXXXX
 BSC (variation) [28]XXXXXXX
 BSC (variation) [29]XXXX
 BSC (variation) [30]XXX
 BSC (variation) [31]XXXXX
 BSC (variation) [32]XXXXXX
 BSC (variation) (subset of broader context) [34]XXXX
 BSC (variation) [35]XXX
 BSC (variation) (subset of broader context) [36]XXXXXXXXXXXX
 BSC (variation) [37]XXXX
 BSC (variation) [38]XXXXX
 BSC (variation) [39]XXXXXXXXXXX
BSC—hospital reports: adaptation of traditional framework
 Hospital report: BSC (variation) [42]XXXXXXXX
 Hospital report: BSC (variation) (subset of broader context) [44]XXX
Baldrige framework
 Baldrige education criteria frameworka [107]XXXXXXXXX
Bruce framework
 Model for quality of prenatal care [55]XXXXX
European Foundation for Quality Management (EFQM)
 EFQM Excellence Model [67]XXXXXXXX
 European foundation for quality management—healthcare adapted framework [73]XXXXXXXX
Institute of Medicine matrix
 Alberta quality matrix for health [13]XXXXXXX
 Health care quality indicators framework (OECD) (subset of broader context) [7]XXXXXXXX
 National healthcare quality conceptual framework [6]XXXXXX
 US health system performance scorecard [10]XXXXXXXX
 National healthcare disparities framework [8]XXXXXXX
 Quality of health care for children and adolescents conceptual framework [11]XXXXXXX
 Pediatric data quality system framework [12]XXXXXX
 Medicaid home and community-based services quality matrixc [9]XXXXXXXXXXXX
Maxwell dimensions of quality
 Conceptual framework for day care provisiona [51]XXXXXX
 Multidimensional quality evaluation model for learning disability health services [65]XXXXXX
 Health services utilization and research commission framework for system performance indicators [63]XXXXXX
National health performance framework
 Performance framework for ambulance services [49]XXXXXXXXXXX
 Mental health performance indicator framework [71]XXXXXXXXXXXX
 National health performance framework (subset of broader context) [69]XXXXXXXXXX
National health service performance framework
 National health service performance framework [70]XXXXXXX
New Zealand health and disability system quality dimensions
 Patient safety and quality domains [57]XXXXXXXX
 New Zealand health and disability system quality dimensions [58]XXXXXXX
Rosenbluth model
 Model for evaluating collaborative healthcare programs for children with special needs [66]XXXXXXX
Total quality management model
 Framework for total quality management [93]XXXX
Veterans health administration framework
 Veteran health administration quality management accountability framework [92]XXXXXXXX
World Health Organization
 Framework for measuring health system quality [86]XXXXXXXXXX
Miscellaneous primary frameworks
 Framework for practice organization of general practice [85]XXXXXXXXX
 Quality indicators for end-of-life-care [82]XXXXX
 Intensive care unit evaluation framework [101]XXXXXXXXX
 Emergency Department Performance Framework [103]XXXXXX
 Quality framework for addiction treatment programs [96]XXXXXXX
 Quality of care framework (subset of broader context) [81]XXXX
 Quality of care in maternity services framework [90]XXXXXXXXX
 Quality and outcomes framework [78]XXXXXX
 10-Rights framework [48]XXXXX
 Performance wheel [91]XXX
 Model of nursing home quality [87]XXXXX
 Performance assessment Tool (PATH) for quality improvement in hospitals [105]XXXXXXXXXX
 Framework for quality management in healthcare [84]XXXXXXXXX
 Model for evaluating the delivery of pediatric primary care services [100]XXXXX
 Commission on accreditation of rehabilitation facilities performance framework [83]XXXX
 Framework for assessing quality of mental healthcare [64]XXXXXXXXXXX
 Quality performance roadmap [80]XXXXXXXXX
 Western Australian clinical governance framework (subset of broader context) [76]XXXXXX
 Public Health Department Program Evaluation Model [99]XXXXXX
 Cancer Quality Council of Ontario Cancer System Strategy Map [88]XXXXX
 Quality measures for mental healthcare [89]XXXX
 Framework for measuring the quality of preschool child health surveillance [47]XXXXXXXXX
 Framework for local public health system quality assessment [52]XXXXXXXX
 Public health system performance framework (subset of broader context) [53]XXXXX
 Quality of care conceptual framework [50]XXXXXXX
 Developmental performance framework for primary care [98]XXXXX
 Framework for evaluating the impact of vertically integrated rural health networks [94]XX
 Framework of quality of life of dying persons [54]XXXXXXXX
 Health systems performance framework [95]XXX
 Framework for managing the quality of health services in New South Wales (subset of broader context) [77]XXXXXXXXX
 Canadian Institute for Health Information Health Indicator Framework (subset of broader context) [5]XXXXXXXXX
 Conceptual framework for the strategic quality plan and strategic initiatives [75]XXXXXXXX
 Ontario Health Quality Council Reporting Framework [97]XXXXXXXXX
 Health quality council quality matrix [56]XXXXXXX
 Saskatchewan framework for health service and outcome indicators [79]XXXXXXXXXX
 National quality standards in gifted and talented educationa [106]XXXXXXXXXXX
 Systemic change frameworka [110]XXXXXXXXXXX
 OECD education indicator frameworka [111]XXXXXXX
 Quality and effectiveness of education and training systems in the EUa [108]XXXXXXXXX
 Model for quality evaluation in residential care for people with intellectual disabilityb [112]XXXX
 Model of quality for people with disabilitiesc [116]XXXXXXXXX
 Framework for studying community system performance (subset of broader context)c [115]XXXXXX
 Mental health, developmental disabilities and substance abuse services performance domainsc [113]XXXXXXX
 Medicaid home and community-based services quality frameworkc [114]XXXXXXXXX
  • aEducation sector. bSocial services sector. cMulti-sector. X=Presence of the concept group.

Discussion

The results of our comprehensive search highlight the volume and breadth of work focused on the development of frameworks for performance or quality improvement purposes. We identified a large number of frameworks, many of which represent applications of a preexisting framework.

The frameworks identified were heterogeneous; however, they could be organized by their level of application, setting and target population. An important finding was that, although there was a preponderance of frameworks developed in and intended for the health sector, very few frameworks from or for the education and social services sectors were identified. This finding appears to reflect the increasing desire for, and acceptance of, a framework as a basis for quality improvement in healthcare in the USA, Canada, Europe and elsewhere [2, 117]. The results of this search suggest that such frameworks have been less readily embraced, or perhaps are less familiar, in other human services sectors. However, measuring and tracking performance is relevant to policy-makers, planners, managers and front-line workers in all services sectors who seek to understand the outcomes of the services provided to their particular population and there is clearly, at present, a lack of cross-sectoral frameworks available. With an increasing emphasis being placed of working together across agencies to achieve common goals [118], we would argue for the development of more joint frameworks that are cross sectoral and that reflect the context and needs of all settings involved. The fact that quality improvement has been promoted as a core value in the healthcare field more than in education or social services means that the development of cross-sectoral quality improvement initiatives may be met with barriers and challenges that will need to be dealt with through dialogue and education. The existence of a clearly defined and commonly agreed nomenclature set would allow researchers, policy-makers and others to talk to each other and plan effectively.

A number of challenges were encountered in performing this review. First, the diffuse nature of the topic posed challenges in terms of identifying and reviewing the literature. By beginning with an initial scoping review by an experienced librarian, we were able to identify a number of key concept areas to focus our search. Second was the need to create a search strategy that achieved an appropriate balance between recall and precision. While it was recognized that the application of the filter was necessary and may have resulted in the exclusion of some relevant articles, it was also anticipated that this would be mitigated by our citation review of included papers. Third, the most common framework found in our review was the BSC. While this finding may relate to the inclusion of ‘balanced scorecard’ as a search term, we also searched specifically for other known frameworks (e.g. European Foundation for Quality Management, Baldrige framework). The prolific use of this particular framework does suggest a significant contribution to the area of performance measurement.

We acknowledge that there is a degree of subjectivity in differentiating eligible from ineligible frameworks for our review and that our definition of a performance measurement and improvement framework (i.e. a model that specifies elements that can and should be measured and monitored in order to ensure a systematic process of improving quality of services) may have resulted in the exclusion of a few frameworks touted by others as being a ‘quality improvement framework’. For example, Maxwell [119] and Donabedian [120] are cited frequently for their significant contributions to quality measurement, but according to our criteria, their publications describe a list of quality domains rather than a framework for performance measurement and improvement. We acknowledge that the decision about when a ‘list’ or ‘groups of indicators’ becomes a ‘framework’ is inherently subjective, and more importantly, the application of too rigid criteria runs the risk of excluding significant frameworks. Similarly, Wagner's chronic disease model [121], which has been described as ‘providing a conceptual framework for transforming health care for patients with chronic conditions [122]’ was not included as it did not conform to our criteria for a performance measurement and improvement framework. However, given that our approach to identifying quality concepts was qualitative, and we had a large amount of data to work with to develop the quality concepts, we do not believe our findings would differ if the ‘borderline’ frameworks that were not included had been.

Despite these challenges, across the heterogeneous set of frameworks found, a comprehensive set of quality concepts that were applicable in many settings, sectors and levels of application, were identifiable. Frameworks, irrespective of where they originate, tended to be composed of similar elements, although sources sometimes label the quality categories differently by referring to them as domains, dimensions or quadrants; or by using different terms to describe thematically similar constructs. The concept sorting exercise helped to identify the commonalities in underlying structure and meaning across frameworks. The consistency with which the main quality categories were found led us to propose the 16 quality concepts as a core set, usable and adaptable by almost any quality frameworks.

Planning a performance measurement and improvement framework will require choosing the most appropriate quality concepts, keeping in mind that particular quality domains may be more or less relevant depending, for example, on the intended level of application or the number of sectors working together. One quality concept particularly relevant to evaluating integrated service delivery within multi-sector frameworks, for example, is the concept collaboration. Given the broad scope of our search, and the diversity of frameworks identified, we anticipate that our findings, observations and analyses will be of use to a wide spectrum of policy-makers, researchers and managers who are at the early stages of planning to use or develop a framework, and who wish to consult and build a framework using the research evidence.

Funding

This study was funded by a grant from the Canadian Institutes of Health Research. Anne Klassen is a recipient of Canadian Institute of Health Research career award. Veronica Schiariti is the recipient of salary support from the Sunny Hill Foundation for Children through a grant from the British Columbia Ministry of Education.

Appendix 1

SiteURL (accessed April to June 2008)
Grey literature websites: organizations
 Agency for Healthcare Policy and Research (Agency for Healthcare Research and Quality) (AHRQ) inclusive of Effective HealthCarehttp://www.ahrq.gov/
 Improving chronic illness carehttp://www.improvingchroniccare.org/index.php?p=Regional_Framework&s=50
 Commonwealth fundhttp://www.cmwf.org/
 Council for Disabled Children (National Children's Bureau) (UK)http://www.ncb.org.uk/Page.asp?sve=785
 Every child matters (National Evaluation of Children's Trusts) (managed by the UK Department for Children, Schools and Families (DCSF))http://www.everychildmatters.gov.uk/
 National Initiative for Children's Healthcare Qualityhttp://www.nichq.org/nichq
 Commission on Accreditation of Rehabilitation Facilities (CARF)http://www.carf.org/
 National Dissemination Center for Children with Disabilitieshttp://www.nichcy.org/
 Beach Center on Disability (University of Kansas)http://www.beachcenter.org/
 CanChildhttp://www.canchild.ca/
 Centre for Research on Educational and Community Services (CRECS) (University of Ottawa)http://www.socialsciences.uottawa.ca/crcs/eng/
 Health Quality Council of Albertahttp://www.hqca.ca/
 New England SERVEhttp://www.neserve.org/index.html
 Canadian Association for Community Livinghttp://www.cacl.ca/
 Canadian Evaluation Societyhttp://www.evaluationcanada.ca/site.cgi?s=1
 Centre for Quality Assessment & Improvement in Mental Health (CQAIMH)http://www.cqaimh.org/
 Council of Ministers of Education, Canadahttp://www.cmec.ca/
 Performance Measurement Associationhttp://www.som.cranfield.ac.uk/som/research2/centres/cbp/pma/
 Campbell Collaborationhttp://www.campbellcollaboration.org/
 National Association for Healthcare Quality (US)http://www.nahq.org/
 National Association of Children's Hospitals and Related Institutions (NACHRI)http://www.childrenshospitals.net/AM/Template.cfm?Section=Homepage&Template=/customSource/homepage/homepage.cfm
 RAND Corporationhttp://rand.org/
 Southern Alberta Child and Youth Health Networkhttp://www.sacyhn.ca/
 University of York (UK)http://www.york.ac.uk/
 Canadian Council on Health Services Accreditationhttp://www.cchsa.ca/default.aspx?page=276
 Health Quality Council of Saskatchewanhttp://www.hqc.sk.ca/portal.jsp?KfoYhitywMAvRhvh6/qyPDBIzBf0QfLQkUwK4QBZaJs5RT805SdC1fur5/Qw7//Ekx8VmhwEmSQ=
 Ontario Health Quality Councilhttp://www.ohqc.ca/en/index.php
Grey literature websites: government
 Government of USA
  Department of Health and Human Services (HHS)http://www.hhs.gov/
  Department of Education (ED)http://www.ed.gov/index.jhtml
 Government of Canada
  Health Canadahttp://www.hc-sc.gc.ca/index_e.html
  Public Health Agency of Canadahttp://www.phac-aspc.gc.ca/
 Government of UK
  Department of Healthhttp://www.dh.gov.uk/en/index.htm
  Department for Children, Schools and Familieshttp://www.dcsf.gov.uk/
  Office for Disability Issueshttp://www.officefordisability.gov.uk/
 Government of Scotland
  Health and Community Carehttp://www.scotland.gov.uk/Topics/Health
  Education and Traininghttp://www.scotland.gov.uk/Topics/Education
 Government of New Zealand
  Ministry of Social Development (inclusive of Child, Youth and Family's)http://www.msd.govt.nz/
  Ministry of Healthhttp://www.moh.govt.nz/moh.nsf
 Government of Australia
  Department for Health and Aginghttp://www.health.gov.au/
  Department of Families, Housing, Community Services and Indigenous Affairs (FaHCSIA)http://www.facs.gov.au/
 Government of Sweden
  Ministry of Education and Researchhttp://www.sweden.gov.se/sb/d/2063
  Ministry of Health and Social Affairshttp://www.sweden.gov.se/sb/d/2061
 Government of Norway
  Ministry of Children and Equalityhttp://www.regjeringen.no/en/ministries/bld.html?id=298
  Ministry of Health and Care Serviceshttp://www.regjeringen.no/en/ministries/hod.html?id=421
  Ministry of Education and Researchhttp://www.regjeringen.no/en/ministries/kd.html?id=586
 Government of Netherlands
  Ministry of Education, Culture and Sciencehttp://www.minocw.nl/english/
  Ministry of Health, Welfare and Sporthttp://www.minvws.nl/en/
  Ministry for Youth and Familieshttp://www.jeugdengezin.nl/english/
 Government of North Carolina
  Department of Health and Human Serviceshttp://www.ncdhhs.gov/
  North Carolina Public Healthhttp://www.ncpublichealth.com/
 Government of California
  Department of Social Serviceshttp://www.dss.cahwnet.gov/cdssweb/default.htm
  Department of Educationhttp://www.cde.ca.gov/
  Department of Developmental Serviceshttp://www.dds.cahwnet.gov/
  California Children & Families Commissionhttp://www.ccfc.ca.gov/
 Government of Western Australia
  Department of Communitieshttp://www.communities.wa.gov.au/DFC
  Department of Education and Traininghttp://www.det.wa.edu.au/
  Department of Education Serviceshttp://www.des.wa.gov.au/pages/index.php
  Department of Healthhttp://www.health.wa.gov.au/home/
 Government of Alberta
  Health and Wellnesshttp://www.health.gov.ab.ca/
  Children's Services (inclusive of Children and Youth Initiative)http://www.child.gov.ab.ca/
  Educationhttp://www.education.gov.ab.ca/
 Government of British Columbia
  Children and Family Developmenthttp://www.gov.bc.ca/mcf/
  Educationhttp://www.gov.bc.ca/bced/
  Healthhttp://www.gov.bc.ca/health/
 Government of Nunavut
  Community and Government Serviceshttp://cgs.gov.nu.ca/en/services-communities
  Educationhttp://www.gov.nu.ca/education/eng/index.htm
  Health and Social Serviceshttp://www.gov.nu.ca/health/
 Government of Manitoba
  Disability Issues Officehttp://www.gov.mb.ca/dio/index.html
  Education, Citizenship and Youthhttp://www.edu.gov.mb.ca/
  Family Services and Housinghttp://www.gov.mb.ca/fs/
  Healthhttp://www.gov.mb.ca/health/
  Healthy Childhttp://www.gov.mb.ca/healthychild/index.html
 Government of New Brunswick
  Individuals and Familieshttp://app.infoaa.7700.gnb.ca/gnb/Pub/EServices/ListServicesBySector.asp?AreaID1=2&SectorID1=0
 Government of Newfoundland and Labrador
  Educationhttp://www.ed.gov.nl.ca/edu/
  Health and Community Serviceshttp://www.gov.nf.ca/health/
 Government of Nova Scotia
  Community Serviceshttp://www.gov.ns.ca/coms/
  Educationhttp://www.ednet.ns.ca/
  Healthhttp://www.gov.ns.ca/health/
 Government of Ontario
  Children and Youth Serviceshttp://www.children.gov.on.ca/CS/en/default.htm
  Training, Colleges and Universities (Education)http://www.edu.gov.on.ca/eng/welcome.html
  Health and Long-Term Carehttp://www.gov.on.ca/health/
 Government of Prince Edward Island
  Education and Learninghttp://www.gov.pe.ca/infopei/index.php3?number=13755
  Health and Wellnesshttp://www.gov.pe.ca/infopei/index.php3?number=13758
 Government of Quebec
  Education, Leisure and Sporthttp://www.mels.gouv.qc.ca/GR-PUB/m_englis.htm
  Family and Seniorshttp://www.mfa.gouv.qc.ca/index_en.asp
  Health and Social Serviceshttp://www.msss.gouv.qc.ca/en/index.php
 Government of Saskatchewan
  Social Serviceshttp://www.socialservices.gov.sk.ca/
  Healthhttp://www.health.gov.sk.ca/
  Educationhttp://www.learning.gov.sk.ca/
 Government of the Northwest Territories
  Education, Culture and Employmenthttp://www.ece.gov.nt.ca/
  Health and Social Programshttp://www.hlthss.gov.nt.ca/english/default.htm
 Government of Yukon
  Community Serviceshttp://www.community.gov.yk.ca/
  Educationhttp://www.education.gov.yk.ca/
  Health and Social Serviceshttp://www.hss.gov.yk.ca/
 Government of Los Angeles County
  Children and Family Services, Department ofhttp://dcfs.co.la.ca.us/
  Public Health, Department ofhttp://www.lapublichealth.org/

Grey literature websites searched

Appendix 2

Framework name and source (some frameworks were implied and not named)
Sector
Administrative sector(s) in which the framework was developed or used:
  • Health

  • Education (preschool to high school)

  • Social services

Level of application: frameworks were applied in a range of levels of quality measurement from those relevant at the provider–client interface to much higher levels of aggregation. We categorized all frameworks as follows:
  • Level 1: frameworks for specific services, program, units or departments (that usually exist within an organization or institution such as a department within a hospital or a team within a school district), that represent direct interface with clients

  • Level 2: frameworks for organizations or institutions that comprise individual programs, units or departments (e.g. hospitals; school districts)

  • Level 3: frameworks for systems that comprise a number of organizations/institutions (e.g. national health system; provincial education system)

Framework setting (e.g. public health; school district)
Population of interest (e.g. children)
Primary vs. application frameworks: Primary frameworks were defined as one-of-a-kind frameworks or involved substantive changes or adaptations to existing framework(s) (e.g. two or more frameworks merged to create a new framework). Application frameworks were those describing a specific or further application of the primary framework.
Categories used in frameworks to denote subheadings for various domains, dimensions, elements or quadrants of quality and under which quality or performance indicators can be grouped (see definition in Table 2).
Country of application
Citations to other quality improvement frameworks
Source: Systematic review or the targeted grey literature scan

Data extracted from each framework

References

View Abstract