OUP user menu

Design effects and intraclass correlation coefficients from a health facility cluster survey in Benin

Alexander K. Rowe , Marcel Lama , Faustin Onikpo , Michael S. Deming
DOI: http://dx.doi.org/10.1093/intqhc/14.6.521 521-523 First published online: 1 December 2002

To the Editor: For at least two decades, disease control programs in developing countries have used clinical guidelines for health workers to improve the management of ill children and prevent childhood deaths. To monitor health worker performance and evaluate the implementation of such guidelines, a common method has been to conduct health facility cluster surveys in which health workers are observed during consultations [1,2]; a cluster is usually defined as all, or a sample of, consultations occurring at a given health facility on a given day. Data from these surveys tend to be correlated because of the similarity in case management quality for children seen at the same health facility (often all children are seen by only one or two health workers).

When planning such a survey, the sample size of consultations must account for potential correlation in the data. A common approach is to estimate an ‘effective’ sample size, n, with a formula that does not account for correlation, and then to calculate a final sample size by multiplying n by an estimate of the design effect. The design effect is ‘the ratio of the actual variance of a sample to the variance of a simple random sample of the same number of elements’ ([3], p. 258); it also equals ρ(m − 1) + 1, where ρ is the intraclass correlation coefficient and m is the average number of consultations per cluster ([3], p. 162). The intraclass correlation coefficient is a measure of the homogeneity of elements within clusters and has a maximum value of +1 when there is complete homogeneity within clusters, and a minimum value of −1/(m − 1) when there is extreme heterogeneity within clusters ([3], p. 171). For survey planners, however, a practical problem with this approach is deciding what number to use for the design effect: firstly, health facility surveys can measure dozens of quality-of-care variables and each variable may have a different design effect; and secondly, to the best of our knowledge, there are no published design effects, or estimates of ρ from which design effects could be calculated, from surveys in developing countries to provide guidance (results have been published from surveys in industrialized countries [4]).

To provide estimates of design effects and ρ from a health facility survey in a developing country, we summarized results from a survey conducted in Ouémé Département, Benin, in 1999. Methodological details and quality-of-care results are published elsewhere [5]. To estimate design effects, we used SUDAAN software [6], with the cluster defined as the health facility; and we estimated ρ with the formula ρ = (design effect − 1)/(m − 1).

Our analysis included 55 dichotomous quality-of-care variables: 19 variables on clinical assessment tasks ([5], Table 1), 11 on illness classification ([5], Table 2), 14 on treatment ([5], Table 3), one on the referral of severely ill children ([5], p. 1628), and 10 on counselling ([5], p. 1629). Design effects and ρ for nine of these variables were not estimated because the variables had a mean value of 0%. A histogram of the design effects for the remaining 46 variables is shown in Figure 1. The median design effect was 1.4 (range 0.8–5.7), and 10 (21.7%) of the 46 design effects were ≥2.0. Results were generally similar for variables on assessment tasks (median 1.5, range 0.9–5.7), illness classification (median 1.4, range 0.8–1.8), treatment (median 1.3, range 0.9–3.8), referral (design effect 1.5), and counselling (median 1.7, range 1.0–3.7).

Figure 1

Design effects for 46 quality-of-care variables from a health facility survey, Ouémé Département, Benin. a, variable = health worker determined if the child had an ear problem; b, variable = health worker determined if the child had cough or difficult breathing; c, variables = health worker counselled the caretaker to return immediately if the child becomes more sick, and health worker determined if the child had diarrhoea; d, variable = health worker verified the caretaker’s comprehension of instructions; e, variables = health worker gave advice on using a mosquito bed net, and health worker checked for neck stiffness; f, variable = health worker gave instructions on administering medicines; g, variable = health worker prescribed the recommended treatment for malaria; h, variable = child was weighed.

A histogram of ρ for the 46 variables is shown in Figure 2. The median ρ value was 0.2 [range (−0.3)−1.0, 25–75% interquartile range 0.1–0.4], and 10 (21.7%) of the 46 ρ values were ≥0.5. The median m value was 4.2 consultations per cluster (range 1.3–5.4). A table of results for each indicator is available from the authors.

Figure 2

Intraclass correlation coefficients for 46 quality-of-care variables from a health facility survey, Ouémé Département, Benin. a, variable = health worker prescribed the recommended treatment for non-severe anaemia; b, variables = health worker asked if there was blood in the stool, health worker pinched the skin of the abdomen, and child with a severe illness was referred to a hospital; c, variables = health worker checked for neck stiffness, and health worker gave advice on using a mosquito bed net; d, variable = health worker prescribed the recommended treatment for malaria; e, variables = child was weighed, health worker prescribed the recommended treatment for severe anaemia, and health worker gave instructions on administering medicines.

Results from one study are inadequate for making broad generalizations about design effects of health facility surveys; however, in Benin, we recommend that future surveys be planned using a design effect of 3.8. Although this design effect is not the highest one we found, it is the highest for a variable of central importance to our project (variable = recommended malaria treatment), which is a malaria control project. For the global public health community, we support the recommendation of Ukoumunne et al., that results from other surveys be published [4]. If survey planners have estimates for m (from records of the health facilities they plan to survey), then values of ρ from other surveys could be used for more accurate design-effect estimates. If estimates for m are unavailable, then design effects from other surveys could be used directly. With such results, survey planners could more accurately design surveys that have the desired level of precision and are no larger (and no more expensive) than necessary.

Acknowledgments

This survey was funded by the Africa Integrated Malaria Initiative of the United States Agency for International Development (project No. 936-3100).

References