International Journal for Quality in Health Care 14:521-523 (2002)
© 2002 International Society for Quality in Health Care
Letter to the Editor |
Design effects and intraclass correlation coefficients from a health facility cluster survey in Benin
Centers for Disease Control and Prevention, Atlanta, GA, USA
Africare, Porto Novo, Benin
Direction Départementale de la Santé Publique de l’Ouémé, Porto Novo, Benin
Centers for Disease Control and Prevention, Atlanta, GA, USA
To the Editor: For at least two decades, disease control programs in developing countries have used clinical guidelines for health workers to improve the management of ill children and prevent childhood deaths. To monitor health worker performance and evaluate the implementation of such guidelines, a common method has been to conduct health facility cluster surveys in which health workers are observed during consultations [1,2]; a cluster is usually defined as all, or a sample of, consultations occurring at a given health facility on a given day. Data from these surveys tend to be correlated because of the similarity in case management quality for children seen at the same health facility (often all children are seen by only one or two health workers).
When planning such a survey, the sample size of consultations must account for potential correlation in the data. A common approach is to estimate an ‘effective’ sample size, n, with a formula that does not account for correlation, and then to calculate a final sample size by multiplying n by an estimate of the design effect. The design effect is ‘the ratio of the actual variance of a sample to the variance of a simple random sample of the same number of elements’ ([3], p. 258); it also equals 
(m - 1) + 1, where is the intraclass correlation coefficient and m is the average number of consultations per cluster ([3], p. 162). The intraclass correlation coefficient is a measure of the homogeneity of elements within clusters and has a maximum value of +1 when there is complete homogeneity within clusters, and a minimum value of -1/(m - 1) when there is extreme heterogeneity within clusters ([3], p. 171). For survey planners, however, a practical problem with this approach is deciding what number to use for the design effect: firstly, health facility surveys can measure dozens of quality-of-care variables and each variable may have a different design effect; and secondly, to the best of our knowledge, there are no published design effects, or estimates of from which design effects could be calculated, from surveys in developing countries to provide guidance (results have been published from surveys in industrialized countries [4]).
To provide estimates of design effects and from a health facility survey in a developing country, we summarized results from a survey conducted in Ouémé Département, Benin, in 1999. Methodological details and quality-of-care results are published elsewhere [5]. To estimate design effects, we used SUDAAN software [6], with the cluster defined as the health facility; and we estimated with the formula = (design effect - 1)/(m - 1).
Our analysis included 55 dichotomous quality-of-care variables: 19 variables on clinical assessment tasks ([5], Table 1), 11 on illness classification ([5], Table 2), 14 on treatment ([5], Table 3), one on the referral of severely ill children ([5], p. 1628), and 10 on counselling ([5], p. 1629). Design effects and for nine of these variables were not estimated because the variables had a mean value of 0%. A histogram of the design effects for the remaining 46 variables is shown in Figure 1. The median design effect was 1.4 (range 0.8–5.7), and 10 (21.7%) of the 46 design effects were 
2.0. Results were generally similar for variables on assessment tasks (median 1.5, range 0.9–5.7), illness classification (median 1.4, range 0.8–1.8), treatment (median 1.3, range 0.9–3.8), referral (design effect 1.5), and counselling (median 1.7, range 1.0–3.7).
|
A histogram of for the 46 variables is shown in Figure 2. The median value was 0.2 [range (-0.3)-1.0, 25–75% interquartile range 0.1–0.4], and 10 (21.7%) of the 46 values were 0.5. The median m value was 4.2 consultations per cluster (range 1.3–5.4). A table of results for each indicator is available from the authors.
|
Results from one study are inadequate for making broad generalizations about design effects of health facility surveys; however, in Benin, we recommend that future surveys be planned using a design effect of 3.8. Although this design effect is not the highest one we found, it is the highest for a variable of central importance to our project (variable = recommended malaria treatment), which is a malaria control project. For the global public health community, we support the recommendation of Ukoumunne et al., that results from other surveys be published [4]. If survey planners have estimates for m (from records of the health facilities they plan to survey), then values of from other surveys could be used for more accurate design-effect estimates. If estimates for m are unavailable, then design effects from other surveys could be used directly. With such results, survey planners could more accurately design surveys that have the desired level of precision and are no larger (and no more expensive) than necessary.
This survey was funded by the Africa Integrated Malaria Initiative of the United States Agency for International Development (project No. 936-3100).
References
- World Health Organization, Division of Child and Adolescent Health and Development. Final Draft: Health Facility Survey for Integrated Child Health Services. Geneva: WHO, March 2002: http://www.who.int/child-adolescent-health/New_Publications/Overview/HFS/Table_content.htm Accessed May 27, 2002.
- Carolina Population Center. Sampling Manual for Facility Surveys for Population, Maternal Health, Child Health and STD Programs in Developing Countries. MEASURE Evaluation Manual Series, No. 3. University of North Carolina, Chapel Hill: CPC, July 2001: http://www.cpc.unc.edu/measure/publications/manuals/facility.pdf Accessed May 27, 2002.
- Kish L. Survey Sampling. New York: John Wiley & Sons, Inc., 1965.
- Ukoumunne OC, Gulliford MC, Chinn S, Sterne JAC, Burney PGJ. Methods for evaluating area-wide and organisation-based interventions in health and health care: a systematic review. Health Technol Assess 1999; 3: iii–92.[Medline]
- Rowe AK, Onikpo F, Lama M, Cokou F, Deming MS. Management of childhood illness at health facilities in Benin: problems and their causes. Am J Public Health 2001; 91: 1625–1635.
[Abstract/Free Full Text] - Shaw BV, Barnwell BG, Bieler GS. SUDAAN User’s Manual, Release 7.5. Research Triangle Park, NC: Research Triangle Institute, 1997.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||