OUP user menu

Identifying population-level indicators to measure the quality of cancer care for women

Monika K. Krzyzanowska , Lisa Barbera , Laurie Elit , Asma Razzaq , Refik Saskin , Naira Yeritsyan , Arlene S. Bierman
DOI: http://dx.doi.org/10.1093/intqhc/mzr043 554-564 First published online: 26 July 2011


Objective Although there is interest in measuring the quality of cancer care, there has been limited effort to evaluate quality for specific subpopulations such as women or to examine differences in performance among women associated with sociodemographic characteristics. The objective of this study was to identify a comprehensive set of quality indicators for evaluation of the quality of cancer care received by women using administrative data.

Design A conceptual measurement framework developed by the study investigators was used to guide literature review to identify existing quality indicators. The list of potential indicators from the literature was first reviewed by the study investigators with respect to importance and feasibility to determine a set of indicators to present to an expert panel who used a modified Delphi process to select indicators for inclusion using predetermined explicit criteria.

Setting The Project for an Ontario Women's Health Evidence-Based Report Card.

Participants A multidisciplinary expert panel consisting of clinicians, researchers and administrators with expertise in cancer, quality of care and/or health services research.

Main Outcome Measure Set of quality indicators evaluable from administrative data.

Results The initial literature search identified 427 indicators, of which 46 were rated as important and feasible by the study investigators. Following two rounds of ratings and an in-person meeting, the expert panel recommended 31 indicators for inclusion in the final set spanning the following areas: general indicators (three indicators), cancer screening (six), colorectal cancer (four), lung cancer (three), breast cancer (five), gynecologic cancers (five), and end-of-life care (five).

Conclusions A comprehensive set of 31 indicators was identified to evaluate the quality of cancer care received by women that also allows assessment of gender and socioeconomic disparities in cancer care.

  • quality indicators
  • cancer
  • women's health
  • equity
  • modified Delphi panel


Cancer is a leading cause of morbidity and mortality worldwide. There has been significant and increasing interest in measuring the quality of cancer care as a mechanism to identify gaps in performance and to support quality improvement efforts [1]. To date, the majority of these efforts have focused on measuring regional performance [2], performance specific to a single cancer [35] or treatment modality [6, 7]. Although a large body of evidence has found that health-care performance can vary by gender and sociodemographic factors [815], there has been limited work in developing indicators from the perspective of potentially unique or vulnerable subpopulations such as women, socially disadvantaged groups or the elderly whose quality of care may be different from that of the overall population. There is growing recognition of the need to incorporate an equity lens into performance measurement and reporting. Quality improvement interventions may narrow, maintain or widen disparities depending on the relative magnitude of improvement among different population subgroups [16]. In fact, recent US legislation has encouraged data collection and reporting of quality indicators by gender, ethnicity and socioeconomic status (Section 185, Medicare Improvements for Patients and Providers Act, 2008 [17]).

Because women and men often have very different experiences with the health-care system and different social contexts that can influence access, quality and outcomes of care, it is important to assess gender differences in performance [18]. There are biological as well as social reasons why incorporating a gender and equity perspective into quality measurement is important especially when it comes to cancer. The burden of disease as measured by cancer incidence varies between men and women [19] and there is evidence that women may have difference susceptibility to carcinogens such as tobacco and may respond to treatment differently [20, 21]. In contrast to other areas of medicine such as cardiovascular diseases where differences in the quality of care between men and women have been documented [14], much less is known about sex-based differences in the quality of cancer care. A 2006 report on the quality of cancer services among women in Ontario commissioned by the Ontario Women's Health Council which analyzed the results of the Ontario Cancer System Quality Index, a web-based annual report on the quality of cancer services in Ontario by sex, found some differences in the use of end-of-life services between women and men but minimal sex-based differences in cancer treatment [10]. However, this report did not look at differences in performance among women associated with socioeconomic position or geography. While few studies have specifically evaluated differences in the care received by women and men with cancer, there have been numerous studies that have documented differences in care among subgroups of cancer patients, including subgroups of women with respect to sociodemographic factors such as age, income, ethnicity and where one lives [2225]. For example, the majority of studies which have evaluated age or income-based differences in cancer care have found that elderly patients and patients of lower socioeconomic status have consistently different patterns of care than their younger or wealthier counterparts. Given the systematic differences in care between subgroups of cancer patients, it is only natural that any quality improvement efforts incorporate this perspective in order to succeed.

In Ontario, the Project for an Ontario Women's Health Evidence-based Report (POWER) study was designed to help policy-makers and providers to improve the health of and reduce inequities among women in Ontario by providing actionable data on women's health and health system performance in the province. As a leading cause of morbidity and mortality both in Canada and worldwide, cancer was included as one of the key diseases of interest in the POWER study. The indicators were also selected for the purpose of assessing differences in performance between women and men, and among women associated with income and geography. The perspective of the study was unique as the existing reports do not routinely examine gender differences in performance, nor do they look at differences between subgroups of women associated with socioeconomic status, ethnicity or geography. Hence, the heterogeneity of women's experiences and needs are not reflected in these reports. Furthermore, many reports focus on particular health sectors and not the continuum of care, and they rarely assess both population health and clinical care. The object of the POWER study was to address these limitations [18, 26].

In this article we present the process we undertook to identify a comprehensive set of quality indicators for evaluating the quality of cancer care at the population level with a unique focus on women and on equity. Given the lack of empiric data on differences in quality of cancer care between men and women, the final report on the selected indicators will both contribute to the evidence base in this area and serve as an example of how to incorporate an equity lens into performance measurement on a routine basis.


Identification of potential indicators

The overarching goal of the project was to identify a comprehensive set of measures to evaluate the quality of cancer care received by women in Ontario at the population level using locally available administrative health-care data. An interdisciplinary working group of 10 investigators with interest and expertise in performance measurement and cancer health services research was convened to carry out the project. The process began with two in-person meetings of the group to agree on a conceptual framework for measurement, which was then used to guide the literature review to identify candidate quality indicators. The aim of the project was to identify existing measures from the literature rather than develop novel indicators.

Based on discussions among members of the working group, it was decided that the Cancer chapter of the POWER study would look at indicators spanning the entire spectrum of the cancer disease trajectory from screening, through treatment and survivorship to end-of-life care with a specific focus on key causes of cancer morbidity and mortality in women, including cancers that occur both in men and women (colorectal and lung cancer) as well women-specific tumors such as breast cancer and gynecologic cancers of the cervix, ovary and uterus. Indicators relating to behavioral risk factors were not included in the cancer chapter as they were evaluated as part of another chapter of the POWER study focusing on population health [27].

The literature search included both the peer-reviewed literature as well as a targeted scan of the gray literature. Sources of peer-reviewed literature that were included in the search were Medline, Pubmed, Embase, Cinahl, Psychinfo and the Cochrane Database of Systematic Reviews for reviews of indicators. Gray literature sources included quality indicator gateway sites such as the National Quality Forum and National Quality Measures Clearinghouse and the Agency for Healthcare Research and Quality as well as more cancer-relevant sites such as the Cancer System Quality Index (http://csqi.cancercare.on.ca/), the American Society of Clinical Oncology (http://www.asco.org/), the National Cancer Comprehensive Network (http://www.nccn.org/index.asp) and Cancer Control Planet (http://cancercontrolplanet.cancer.gov/). This initial list was reviewed by a research coordinator with expertise in epidemiology, health services research and performance measurement to delete any duplicate measures and compile a final list of unique quality indicators to present to the working group for further deliberation.

Review process

The above search resulted in a master list of several hundred potential indicators. This list was divided into the following areas: cancer screening, colorectal cancer, lung cancer, breast cancer, gynecologic cancers and end-of-life care. At least two members from the working group with content expertise in the given area reviewed the theme-specific list and ranked each indicator according to importance and feasibility of measurement using administrative health-care data in Ontario, Canada. Indicators were felt to be important if they were able to provide key information on women's health issues, identify disparities between men and women or among women and were actionable. Indicators that were rated as both important and feasible by both working group members were retained to be presented to the external expert panel for the final selection process.

The modified Delphi technique was used for subsequent prioritization and selection of the candidate quality indicators. An expert panel was convened based on nominations from investigators in the POWER study as well as stakeholder advisory groups. In order to ensure a range of perspectives on the panel, members from the full range of the continuum of care were invited from primary care to tertiary care and across different oncologic specialties including surgical, medical and radiation oncology and palliative care. Finally, to ensure that the panel was interdisciplinary, a range of professionals were invited including not only physicians, but also health services researchers and administrators. The final panel consisted of 11 participants including three surgical oncologists, three medical oncologists, one radiation oncologist, two palliative care physicians and two cancer health services researchers. For the first round of ratings, each member of the panel was sent an electronic copy of the Cancer Indicator Reference Manual, which described each of the potential indicators in detail and was then asked to complete an internet-based survey to evaluate and rate potential indicators. The panel members were asked to rank the potential indicators with respect to the specified indicator selection criteria: important, amenable to action, valid and reliable, interpretable and comparable and relevant to equity issues which are summarized in Table 1. These criteria were developed from a comprehensive review of indicator selection criteria used by other reporting bodies and projects to meet project objectives, with specific acknowledgment of the indicator selection criteria developed by the National Committee for Quality Assurance (http://www.ncqa.org/tabid/415/Default.aspx) and those used by the Ontario ministry of health and long-term care (MOHLTC) Health System Score Card [18]. Importance was defined as reflecting an important health issue or aspect of health system functioning. Amenable to action meant that the results from the indicator could be used to inform policy or alter behavior of health services providers. Validity, reliability and interpretability reflected the scientific soundness of the indicator. Indicators were felt to be relevant to equity if previous studies suggested that there were or may be significant differences with respect to gender or socioeconomic status. This set of attributes was developed and approved by the overall group of investigators involved in the POWER study and was used by other working groups (such as cardiovascular health and depression working groups) in indicator selection to ensure consistency in indicator selection across disease areas. The panel members were also explicitly asked whether the potential quality indicator should or should not be included in the report card. Panel members were also asked to suggest other potential indicators.

View this table:
Table 1

POWER study indicator selection criteriaa (18)

Results of the initial survey were summarized using pre-defined decision rules and presented to the panel during an in-person meeting. Panel members' responses to the question ‘Should this indicator be included in the POWER study report card?’ were used to classify the indicators into those where there was consensus to either include or eliminate the indicator and those that needed further discussion. Results and comments from the web-based survey were distributed to inform discussion. Indicators which were recommended for inclusion by more than two-thirds of the panel (66.7%) were included in the final list, whereas those which less than one-third (33.3%) of the panel endorsed for inclusion were eliminated. At an in-person meeting, panel members were given the opportunity to recommend further discussion of these indicators if they felt a strong case could be made for their inclusion or exclusion. Then, the rating results pertaining to the remaining indicators where there was no consensus either way were presented to the panel, following which the panel members were asked to re-rate the indicators with only those recommended by the panel included in the final list.


A summary of the indicator selection process is depicted in Fig. 1. The initial literature review identified 427 unique quality indicators which were divided into the following areas: cancer screening, colorectal cancer, lung cancer, breast cancer, gynecologic cancers and end-of-life care. Indicators that were pertinent to more than one cancer site (i.e. cancer incidence) were put into a ‘general’ category. The working group ranked 46 of these as important and feasible. These were subsequently presented to an external expert panel for prioritization. The panel selected 31 indicators for evaluation as part of the Cancer chapter of the POWER study.

Figure 1

Process for the selection of quality indicators for measuring quality of cancer care in women.

Among the approximately 300 indicators excluded during the preliminary review by the working group, approximately 94 were felt to be important but not feasible with currently available administrative health-care data in Ontario. The majority of these indicators were not feasible because they required detailed pathologic reports that include stage and other pathologic data (i.e. hormone receptor status in breast tumors) for evaluation which are currently not routinely available in Ontario administrative data. Most of the indicators assessing receipt of chemotherapy or hormonal therapy were excluded for this reason as were several quality indicators assessing quality of surgery or quality of pathologic reporting. The third group of not feasible indicators pertained to delays in diagnosis and management of symptoms in the advanced cancer population.

Following the first ranking of the 46 indicators by the expert panel, 20 indicators were included and none were excluded. The remaining 26 indicators were discussed during the in-person meeting and of these an additional 10 were selected by the panel for inclusion in the final set of indicators. The panel also suggested one additional indicator pertaining to evaluation of chemotherapy use in small cell lung cancer to be added to the final set. Details of the final set of 31 quality indicators selected by the expert panel are summarized in Table 2. Of the 31 indicators that were selected for inclusion in the Cancer chapter of the POWER report card, the vast majority (29/31) were process indicators. The general indicators group was composed of indicators that were applicable to multiple cancer sites such as incidence of cancer or wait times for surgery. Five indicators were selected to address breast, cervix and colorectal cancer screening including two indicators that addressed follow-up of abnormal screen results among women receiving cervical cancer screening. The disease-specific indicators focused predominantly on different treatment modalities such as surgery, radiation therapy or chemotherapy usually for early stage disease, although there were a few surveillance indicators for cancer survivors. There were very few indicators pertaining to receipt of systemic therapy as these types of indicators were generally considered not feasible with the currently available data in Ontario, specifically because of lack of detailed cancer stage data. Five indicators pertaining to quality of end-of-life care among patients who died of cancer were also chosen which included indicators that assessed both the use of hospital-based services as well as homecare services. The 16 excluded indicators (Table 3) spanned across the entire disease continuum from diagnosis through end-of-life care.

View this table:
Table 2

Description of the 31 quality indicators selected by the expert panel for evaluation

View this table:
Table 3

List of eliminated indicators


In this manuscript, we describe the methods and outcome of the process we used for identifying a set of population-level performance measures aimed at measuring the quality of cancer care in women, gender differences in performance and disparities in quality associated with income and geography among women. Although the potential quality indicators were chosen from among existing measures in the literature, the perspective and framework used for selection are unique as the majority of quality measurement efforts to date both within and outside cancer have been disease or procedure focused. Equity in health, which has been defined as ‘the absence of systematic and potentially remediable differences in one or more aspects of health across populations or population groups defined socially, economically, demographically or geographically’ [28], has been recognized by the Institute of Medicine as one of the six core dimensions of quality [29]. Incorporating this perspective into quality measurement by developing indicators for specific vulnerable subgroups of the population and by analyzing indicators by factors such as sex or income is very important to minimize the risk that quality improvement initiatives do not actually widen the gaps in quality of care [30].

The most unique aspect of our study is the perspective from which our indicators were selected. Cancer is a leading cause of morbidity and mortality in women. Furthermore, the diagnosis and treatment of cancer, even if the outcome is not fatal, have a significant impact on quality of life, morbidity and health-care utilization. Although relatively little is known about the differences in cancer care between women and men, gender-based differences in care have been documented in other areas of medicine. More importantly, the available literature does point to significant differences in cancer screening and care between subgroups of women with respect to factors such as age [23], socioeconomic status [31], race [32] and immigration status [33] highlighting the need for incorporating these perspectives into quality measurement. By using a gender and equity lens in measuring quality, we can identify specific subgroups of individuals whose quality of cancer care can be improved greatly. In addition, the types of interventions required to improve care are likely to differ depending on sociodemographic factors. We stratified existing performance indicators by sex and then assessed potential differences in performance among subgroups of women, as well as including gender-specific indicators for cancers that occur only in women. In the future, should biological differences in cancer associated with sex result in gender-specific differences in guidelines for cancers occurring in both men and women, there may be a need to develop gender-specific indicators that reflect these differences.

Another unique aspect of our study is the comprehensive nature of our indicator set. Our indicators span the entire spectrum of cancer from screening through end-of-life care and across several different types of cancers all of which are important causes of morbidity and mortality in women specifically. The comprehensive nature of the indicator set will allow us to look at differences in care in different sectors of the health-care system including primary care (cancer screening), tertiary care (most treatment indicators) as well as community care (end-of-life indicators). This will allow us to determine how the different sociodemographic factors affect quality of care in the different health-care sectors.

There are several limitations to our study. First, we focused on existing indicators in the literature and therefore there may be areas of cancer care received by women, which are very important to evaluate but for which indicators do not yet exist. Furthermore, the majority of our indicators were process indicators, which also reflects the current state of the performance measurement field where few validated outcome quality measures exist. Secondly, because of lack of reliable stage and limited clinical data in the available administrative health-care data in Ontario, a number of indicators felt to be important by the working group members were not feasible and therefore eliminated early in the review process. The majority of these pertained to indicators for which detailed stage and/or pathologic data were needed for measurement such as use of systemic therapy for different types of cancer or quality of pathologic diagnosis. This group of indicators could be feasible in other jurisdictions with more detailed pathology reporting. A number of cancer organizations such as the American Society of Clinical Oncology have developed indicators that could fill this gap in jurisdictions where such data are available [34]. A number of important but not feasible indicators had to do with delays in diagnosis and symptom management and are unlikely to be feasible in any jurisdiction currently but may be feasible in the future as the quality of administrative data improves and electronic health record data becomes more widely available. Thirdly, a few indicators such as cancer incidence or cancer survival are not directly actionable and therefore of limited value if used on their own as a quality measure. However, population-based interventions to reduce risk are available, and when included in a comprehensive set of measures such as those selected as part of this study they provide the context in terms of burden of disease and impact on morbidity and mortality which are crucial from a population health perspective.

In summary, this study used a systematic approach to identify a comprehensive set of indicators to assess the quality of cancer care for women in Ontario from a population health perspective. This approach included a comprehensive literature search, a cohesive working group with a range of expertise and a modified Delphi panel that represented the range of the continuum of care and various disciplines within cancer care. We have since used this indicator set to evaluate the quality of cancer care in women of Ontario and identified a number of gaps in care that are currently being addressed [35]. We hope that our conceptual framework and indicators will be of use to other jurisdictions interested in incorporating equity into performance measurement.


This study was funded as part of the POWER Study whose funding is from Echo: Improving Women's Health in Ontario, an agency of the Ontario Ministry of Health and Long-Term Care (MOHLTC). This study was supported by the Institute for Clinical Evaluative Sciences (ICES), which is funded by an annual grant from the MOHLTC. The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by ICES, Echo or the Ontario MOHLTC is intended or should be inferred.


View Abstract