OUP user menu

Development of the Korean primary care assessment tool—measuring user experience: tests of data quality and measurement performance

Jae Ho Lee, Yong-Jun Choi, Nak Jin Sung, Soo Young Kim, Seol Hee Chung, Jaiyong Kim, Tae-Hee Jeon, Hoon Ki Park
DOI: http://dx.doi.org/10.1093/intqhc/mzp007 103-111 First published online: 13 March 2009

Abstract

Objectives To develop a tool for assessing the performance of primary care services in South Korea from the patient's perspective and to test the validity of the tool under the conceptual framework of the recently developed definition of primary care in Korea.

Design Item development for questionnaire and a cross-sectional survey for tool validation at 16 primary care clinics. All family physicians included in this study were required to have practiced at their current clinic for at least 2 years.

Participants A nine expert panel was assembled for tool development and patients (or guardians) who had visited their primary care clinic on six or more occasions over a period of more than 6 months participated in the survey.

Main outcome measures Scores corresponding to each domain of primary care.

Results A total of 722 effective data sets were used for the analysis. Five items were eliminated from the preliminary 30-item tool after expert discussions at two seminars. Another four items were eliminated by principle component analysis. For each of the four domains (comprehensiveness, coordination function, personalized care, and family/community orientation), tests of scaling assumptions were well satisfied by all Likert-scaled measures. On the other hand, ‘first contact’ turned out to be a composite domain with five independent single-item scales.

Conclusion The Korean primary care assessment tool (version 1) consists of four multi-item scales and one composite scale. Widespread application of this tool will provide an empirical basis for the measurement, monitoring and continuous improvement of primary care in South Korea.

Keywords
  • family medicine
  • health policy
  • Korea
  • primary care/general practice
  • public health
  • quality management
  • quality measurement
  • setting of care
  • surveys

Introduction

The body of evidence supporting the health-promoting and disease-preventing influence of primary care has been growing since researchers began to distinguish primary care from other aspects of the health service delivery system. The evidence also shows that primary care in contrast to specialty care is associated with a more equitable distribution of health throughout the population, and this finding has held true in both national and cross-national studies [1]. The term primary care, as distinguished from primary health care, is generally reserved for clinical activity that is primarily focussed on the individual. It also connotes conventional primary medical care striving to achieve the goals of primary health care [2]. In some health systems, primary care sits at the center of a complex primary health care system, which coordinates a wide range of nursing home and home care services. However, in other systems, primary care has little formal connection with any other primary health care activity. This diversity of opinion regarding the purpose and practice of primary care has made it difficult to adopt a universal definition of primary care [3].

The US Institute of Medicine defined primary care as the provision of integrated, accessible health care services by clinicians who are accountable for addressing a large majority of personal health care needs, developing a sustained partnership with patients and practicing in the context of family and community. It also encouraged the development of appropriate tools for assessing the performance of primary care practices [4]. This publication on the concept of primary care led to the development and implementation of at least three primary care performance measures, including the Component of Primary Care Index [5], Primary Care Assessment Survey (PCAS) [6] and primary care assessment tool (PCAT) [7, 8]. The PCAS influenced the development of the General Practice Assessment Survey (GPAS) and the General Practice Assessment Questionnaire (GPAQ) in Britain [9]. The PCAT has been translated into Castilian (Spanish) and Catalan versions [10] as well as a Brazilian (Portuguese) version [11]. A Puerto Rican (Spanish) version of the PCAT is currently being developed [12].

In South Korea (hereafter referred to as ‘Korea’), where the National Health Insurance system covers only over 60% of health care expenditures and 90% of health care organizations belong to the private sector, the terms ‘primary care’ and ‘family medicine’ have been used interchangeably since the introduction of family practice in 1985. However, many have raised questions about the role and characteristics of family medicine. A 2006 survey conducted by the Korean Medical Association found that only 7.9% of all clinic-based practitioners were family physicians. In addition, all medical doctors can own a private clinic and practice as a first-contact (FC) medical doctor, regardless of their specialty. The conceptual ambiguity regarding primary care remains painfully evident in Korea [13]. Fortunately, the Korean definition of primary care was recently established using the Delphi method and published in a recent report [14]. Conceptually, it contains four core attributes: FC, comprehensiveness, coordination of care and longitudinality, and three ancillary attributes: personalized care, context of family and community, and community-based care. Under the conceptual framework of this definition, the authors tried to develop a tool for assessing the performance of primary care services from the patient's perspective, and we examined the performance of this tool using a questionnaire-type survey.

Methods

This study was approved by the Institutional Review Board of Kangnam St. Mary's Hospital, the Catholic University of Korea.

Item selection

The authors reviewed 190 items that were addressed in previous studies pertaining to primary care assessment [58, 15, 16]. Multiple items were developed for each domain and then evaluated from several aspects. Any item that was exceptionally long, lacked clarity, had questionable relevance or had undesirable similarity to other items was either rewritten or eliminated. In addition, the wording of the items was made as simple as possible to ensure that the vast majority of the patients in the sample could fully comprehend the items. All items were written at or below a middle school reading level.

Response format and scoring method

The items have a 5-point Likert-type response format, ranging from 1 (strongly disagree) to 5 (strongly agree), with neutral as the center option. A 5-point Likert scale response format was selected after considering the patient's ability to discriminate between the response levels and acknowledging that fewer categories reduce the burden of the task for the respondents. An additional ‘don't know’ option was included for respondents who wished to answer in this manner, and it was interpreted and regarded as ‘neutral option = 3’. To make the results easier to understand, all scale scores were reduced by a factor of 1 and multiplied by 25 so that they would fall in the range of 0–100 points, with higher scores indicating more favorable performance.

Content validity and pilot test

Three diverse groups of skilled experts in primary care (three primary care physicians, three health policy researchers and three family medicine professors) were asked to evaluate the content validity of the items comprising the preliminary tool. They evaluated the relevance of the items in each of the seven domains (four core and three ancillary) and sought to measure and assess the clarity and conciseness of each item. Revisions were made based on group discussions and specific comments made before the validation survey. Pilot tests of the preliminary tool were conducted at three different sites. A total of 36 patients visiting three family physicians completed the pilot survey. Several revisions were made to the wording of the items based on comments made by the patients. The primary goal of the pilot testing was to focus on the wording, comprehensibility and content of the items. The preliminary tool used in the survey contained 30 items, but expert opinions from two seminars held after the survey led the authors to consider reducing the number of items from 30 to 25 because five of the items were deemed inappropriate and/or unrealistic for application in the Korean health care system.

Sampling and survey for development and validation of the Korean primary care assessment tool

Because it is very difficult to collect representative national data on primary care in Korea, the authors regarded family physicians as the study population from which to obtain homogenous data and we tried to gather data on primary care provided exclusively by family physicians.

Provider selection

The distribution of family physicians by clinic structure based on survey data published by the Korean Medical Association (2005) was as follows: community private clinic of 83.4%, hospital family medicine clinic of 13.0% and public health center clinic of 3.6%. There were a total of 16 sampling sites. They were selected because they were recommended by the authors as ordinary primary care providers and agreed to participate in this survey. Eight clinics were located in small provincial cities, whereas the other eight clinics were located in the metropolitan city of Seoul. Only providers who had been practicing at the same clinic for at least 2 years were allowed to participate in this study. All of the providers were family physicians (n = 21; M/F = 13/8) who worked in primary care settings and volunteered to participate in this study.

User selection

The patient sample consisted of patients (or guardians) who visited one of the participating clinics and agreed to complete the questionnaire before seeing their physicians. Eligible participants were individuals for whom the clinic served as their usual source of care. The usual source of care was defined as a provider whom the user had visited at least six times over a period of more than 6 months.

Data collection

Data collection began in April 2007 and lasted for 3 months. Four research assistants, who were trained in interactive interviewing techniques by two of the authors, helped the patients (or guardians) complete the questionnaire in the waiting room of each physician's office. Parents or guardians were allowed to complete the questionnaire on behalf of patients who were less than 18 years of age or disabled.

Analyses

Principal component analysis was used to explore the structure of the Korean primary care assessment tool (KPCAT) items and examine its construct validity by determining whether the items fell into the hypothesized scales. Internally consistent factors with eigenvalues >1 after rotation were subjected to interpretation. Four criteria were used to eliminate items and determine the final factors [8, 17, 18]. Factor loadings >0.35 were considered meaningful and used as criteria for item retention. In addition, only factors containing at least three items with loadings >0.35 were retained. All retained items should share the same conceptual meaning or construct. Furthermore, none of the retained items should have secondary loadings >0.35. The Likert scaling assumptions [19] were tested as follows: (i) item-convergent validity, (ii) item-discriminant validity, (iii) equal item-scale correlation and (iv) score reliability tested by Cronbach's coefficient alpha. Descriptive statistics were performed to determine the mean, standard deviation, range, etc. Because respondents who had never seen a specialist did not answer the two coordination questions, the analyses were performed with only one question (Does your doctor recommend health care resources appropriately?).

Results

Among those who had a regular health care provider and were eligible to participate in this survey (n = 871), there were no significant differences between the participants (n = 734, 84.3%) and non-participants (n = 137, 15.7%) in terms of age and sex. The most common reason for refusing to complete the questionnaire was that the patient was too busy. After excluding samples with more than three missing responses (n = 12), a total of 722 effective samples were used for the analysis.

Table 1 shows the socio-demographic features of the survey participants by sex. The average age of the participants was 50.2 years. There were no significant differences in the distributions of age groups and clinic types by sex. However, the male participants reported having a higher income, more years of education, a shorter relationship with their physician and more disease diagnoses than their female counterparts.

View this table:
Table 1

Socio-demographic characteristics of subjects included in the sample

Total 722 (100)Male (%) 279 (38.6)Female (%) 443 (61.4)P
Age (year)50.2 SD 20.3
  <40196 (27.2)74 (26.5)122 (27.5)0.78
 40–64336 (46.5)136 (48.8)200 (45.2)
  ≥65190 (26.3)69 (24.7)121 (27.3)
Clinic types by structure
 Community Private clinic602 (83.4)226 (81.0)376 (84.9)0.31a
 Hospital Family Medicine clinic94 (13.0)40 (14.3)54 (12.2)
 Public Health Center clinic26 (3.6)13 (4.7)13 (2.9)
Income (million Won/month/household)260.6 SD 215.0
  <1.50210 (30.7)78 (28.0)169 (38.2)0.003
 1.50–2.99206 (28.5)85 (30.5)85 (27.3)
  ≥3.00269 (37.3)116 (41.6)116 (34.5)
 Missing37
Education (year)9.8 SD 5.4
  <10310 (44.0)100 (36.6)210 (48.6)0.001
 10–12194 (27.5)78 (28.6)116 (26.9)
  ≥13201 (28.5)95 (34.8)106 (24.5)
 Missing17
Duration since first visit (year)4.1 SD 2.2
  <3227 (31.4)99 (35.5)128 (28.9)0.052
 3–5273 (37.8)103 (36.9)170 (38.4)
  ≥6222 (30.8)77 (27.6)145 (32.7)
Number of diseases1.0 SD 0.8
 0205 (28.4)61 (21.9)144 (32.5)0.043
 1345 (47.9)150 (53.9)195 (44.0)
 2 or more171 (23.7)67 (24.1)104 (23.5)
 Missing1
  • Mantel-Haenszel chi-square test.

  • aChi-square test.

Principal component analysis

Twenty-five items were analyzed in the initial principal component analysis. When the eigenvalue was >1.0, six components that corresponded to the hypothesized attributes of primary care remained. Four criteria were used to reach the final solution based on the results of principal component analysis. Three items were eliminated because their secondary loadings were >0.35. In addition, one item was eliminated because its factor loading was <0.35 (Does your doctor share enough information with you about your health problems and address your concerns?). Only one of the three preliminary items in the ‘longitudinality’ domain remained (Do you get periodic health examinations by your physician?), and it was included in the ‘comprehensiveness’ domain by principal component analysis. Finally, a 21-item questionnaire was constructed and named the Korean primary care assessment tool, version 1 (KPCAT, v. 1).

One composite domain

The ‘FC’ domain could not be assessed using a traditional scale with multiple correlated items. The Pearson correlation coefficients among the items comprising this domain were so low (0.06–0.24) that each item could be regarded as an independent scale. Therefore, the ‘FC’ domain was thought to be a composite domain consisting of the following five independent single-item subscales: FC utilization, facility accessibility, cost appropriateness, demographic accessibility, and basic health care (Table 2).

View this table:
Table 2

Principal component analysisa and item reduction process used to develop the KPCATb

Primary care attributesItems contained in preliminary toolMissing‘Don't know’/Not applicable (%)Factor loading (component)Comments
FCDo you visit this clinic first when a new health problem arises?04 (0.6)0.67 (5)FC utilization
Is it easy for you to access this facility?15 (0.7)0.61 (5)Facility accessibility
Appropriateness of out-of-pocket cost?071 (9.8)0.62 (6)Cost appropriateness
ComprehensivenessYour doctor sees patients regardless of their age and sex?025 (3.5)0.65 (6)Demographic accessibility
Basic health care available?157 (7.9)0.48 (5)Basic health care
Does your doctor treat mental health problems as well as physical health problems?053 (7.3)0.62 (1)Personalized care
Counseling/education for health lifestyle(Deleted)Loading on components 2 and 3
Medical check-up available? (e.g. physical exam, blood sugar, cholesterol, BP controls, etc.)064 (8.9)0.59 (3)Comprehensiveness (four items)
Counsels for cancer prevention and screening?0205 (28.4)0.69 (3)
You (or your family member) get periodic Pap smear tests from your physician?0352 (48.8)0.75 (3)
LongitudinalityPeriodic health examination by your physician?1108 (15.0)0.70 (3)
Is the medical team identical whenever you visit this clinic?(Deleted)Loading on components 1 and 5
CoordinationDoes your doctor recommend health care resources appropriately?0131 (18.1)0.66 (2)Coordination functionb (three items)
Since your doctor started treating you, have you ever visited a specialist?
a. If yes, did your doctor recommend the specialist?33751 (13.3)0.84 (2)
b. If yes, did your doctor review the referral results?35876 (20.9)0.84 (2)
Personalized careDoctor understands patients' words easily?73 (0.4)0.76 (1)Personalized care (five items)
Doctor explains test results in a manner that is easy for patients to understand?05 (0.7)0.79 (1)
Doctor recognizes the importance of the patient's medical history?051 (7.1)0.66 (1)
Trust your doctor's decisions on treatment?09 (1.3)0.72 (1)
Doctor shares enough information with you about your health problems and addresses your concerns?(Deleted)Loading <0.35 on component 2
Context of family and communityDoctor asks about and considers the medical history or health states of your family members?(Deleted)Loading on components 2 and 6
Doctor knows about the health, well-being and environmental problems of your community?0272 (37.7)0.61 (4)Family/community orientation (four items)
Doctor has a concern about the persons living with you?0112 (15.5)0.45 (4)
Community BaseIs the doctor active in promoting the health of your community?3472 (65.7)0.83 (4)
This clinic surveys and reflects people's opinions on health care?1364 (50.5)0.74 (4)
  • aBy orthogonal Varimax rotation (eigenvalue ≥1.0, factor loading >0.35); bKorean Primary Care Assessment Tool.

Testing the Likert scaling assumptions

Table 3 shows a summary of the results of the tests of Likert scaling assumptions using the four multi-item scales. All of the item-scale correlations well exceeded the accepted minimum (0.30), with the majority achieving a correlation of >0.50. Three of the four multi-item scales achieved 100% scaling success, indicating that almost all of the items in these scales showed a substantially greater correlation with items in their hypothesized scale than with items in other scales. As shown in column 1 (range of item-scale correlations), the range is relatively narrow (from 0.13 for ‘family/community orientation’ to 0.21 for ‘comprehensiveness’). Finally, analysis of score reliability showed that, with the exception of the ‘comprehensiveness’ domain (alpha = 0.69), all alpha levels exceeded 0.70 and were sufficiently high.

View this table:
Table 3

Five domains of the KPCAT, v. 1 and Likert scale assumptions of its four ordinary scales

DomainsNo. of itemsItem-scale correlationDiscriminant validityCronbach's alpha
Item, other scale correlationScaling success rate (%)
Personalized care50.49–0.630.05–0.4140/40 (100)0.78
Coordination function30.50–0.69−0.06–0.4024/24 (100)0.77
Comprehensiveness40.38–0.59−0.07–0.4230/32 (93.8)0.69
Family/community orientation40.42–0.550.04–0.3832/32 (100)0.71
One composite domain: FCa
  • Pearson correlation analysis. aOne composite domain, FC, consists of score average of five independent subscales.

Descriptive Feature of the KPCAT

The full range of possible scores was observed for all scales, except for the ‘personalized care’ and ‘FC’ domains. Of the five domains of the KPCAT, v. 1, ‘FC’ and ‘personalized care’ had the highest domain scores (Table 4). The elderly participants (≥65 years) tended to give the ‘coordination function’ domain and the ‘facility accessibility’ and ‘cost appropriateness’ subscales lower ratings than the subjects in other age groups. The female subjects' ratings of the ‘comprehensiveness’ and ‘family/community orientation’ domains were worse than those of their male counterparts. In addition, the subjects with a middle school education or lower, tended to give ‘comprehensiveness’, ‘coordination function’ and ‘total primary care score’ lower ratings than those with more years of education. People of low socioeconomic status whose monthly household income was less than 1.5 million Won (Embedded Image) gave their usual source of care the worst ratings on the ‘coordination function’ domain and on the ‘facility accessibility’ and ‘cost appropriateness’ subscales of the ‘FC’ domain. However, they gave their usual source of care the best ratings on the ‘personalized care’ domain. Patients with a long-term relationship with their usual source of care (≥6 years) gave their primary care physician the best ratings on all domains of primary care, except for the ‘FC’ domain (Table 5).

View this table:
Table 4

Descriptive features of the KPCAT, v. 1

DomainsNo. of itemsNo. of patientsMeanMedianStandard deviationScore range (observed)
Personalized care572290.195.011.640–100
Coordination function372259.966.731.50–100
Comprehensiveness472252.456.324.50–100
Family/community orientation472265.062.517.10–100
FCa572287.090.012.430–100
 FC utilization172288.810020.00–100
 Facility accessibility172190.510019.50–100
 Cost appropriateness172282.010023.30–100
 Demographic accessibility172296.010012.30–100
 Basic health care172177.810030.30–100
Total primary care scoreb72270.970.513.527–100
  • aOne composite domain, FC, consists of score average of five independent subscales.

  • bScore average of five domains.

View this table:
Table 5

Primary care scores on 5 domains of the KPCAT, v. 1 by demographic variables

VariablesDomainsFive independent subscales of FC composite domainTotal primary care score
PCCFCHFCOFCaFCUFACADABHC
Age (years)
 <40 (n = 196)87.7b67.2b49.0c65.288.687.691.8b85.8c96.681.171.5
 40-64 (n = 336)89.659.754.664.786.788.092.081.395.876.471.0
 ≥65 (n = 190)93.452.552.165.385.891.186.279.396.176.269.8
Sex
 Male (n = 279)89.157.152.5c64.8c85.486.287.780.593.774.170.4
 Female (n = 443)88.956.348.262.385.986.988.979.195.574.868.9
Education (years)
 <10 (n = 310)90.951.3b48.4b64.386.889.790.080.296.577.568.3b
 10–12 (n = 194)90.264.056.066.387.087.091.182.796.477.772.7
 ≥13(n = 201)88.668.254.964.687.789.191.684.195.678.172.8
Income (million Won/month/household)
 <1.5 (n = 210)92.8b50.9b51.565.586.390.588.3c77.1b96.179.469.4
 1.5-2.9 (n = 206)88.961.155.764.289.089.489.385.096.075.571.4
 ≥3.0 (n = 268)88.764.650.664.889.987.19383.896.778.771.3
Duration since first-visit (year)
 <3 (n = 227)88.6b53.7b51.2b65.2c87.287.3c92.3c82.894.978.5b69.2b
 3–5 (n = 273)89.758.347.662.986.087.591.481.696.672.968.9
 ≥6 (n = 222)92.168.059.567.388.091.787.481.896.582.675.0
Number of disease
 0 (n = 205)87.6b63.948.0b64.288.387.494.784.196.178.970.4
 1 (n = 344)90.857.0c52.565.786.989.290.382.096.277.170.6
 2 or more (n = 171)91.760.557.564.585.689.685.7b79.595.877.271.9
  • ANOVA, Student t-test and Kruskal–Wallis test.

  • PC, personalized care; CF, coordination function; CH, comprehensiveness; FCO, family/community orientation; FCU, first contact utilization; FA, facility accessibility; CA, cost appropriateness; DA, demographic accessibility; BHC, basic health care.

  • aComposite scale consists of score average of five independent subscales. bP < 0.01; c0.01 ≤ P < 0.05.

Discussion

Previous studies have shown the value of measuring primary care in multi-dimensional terms that parallel its formal definition, and they have highlighted the fallibility of relying on unidimensional proxies for primary care (such as the provider's disciplinary affiliation) [20, 21]. Our research showed that the seven attributes of the Korean definition of primary care did not directly form the seven comparable domains from the user's point of view. However, an assessment based on the user's experience alone may not be sufficient to measure the entire range of primary care performance. In addition, the KPCAT, v. 1 does not assess the technical aspects of quality of care due to the limitations surrounding user-provided information.

‘FC’ turned out to be a composite domain with five independent single-item scales in our study. Considering the average score (70.9 SD 13.5) on all five domains, this domain acquired a relatively high score (87.0 SD 12.4), which may be explained not only by the expansion of health care resources and increase in average household income in Korea, but also by the remarkable improvement in the accessibility of the Korean health care system since the introduction of the National Health Insurance (NHI) system in 1989. The NHI insures 97% of the Korean population. The tax-financed Medical Protection Program covers the health care costs for the remaining 3% of the nation's population, which represents the poorest members of society [22]. In Quebec, Canada, by contrast, FC accessibility of primary care clinics showed the most problematic among three attributes (accessibility, continuity and coordination), in a survey (2002) using the PCAT [23]. ‘Comprehensiveness’ has rarely been evaluated from the patient's perspective [24]. A previous study showed that patients generally have limited knowledge about the range of services provided by family physicians [25]. In our study, ‘comprehensiveness’ was evaluated using only a four-item scale. In this domain, the scaling success rate was not 100% and the Cronbach's alpha value was not over 0.70. More items might have been necessary to sufficiently evaluate this domain.

Longer durations of the doctor–patient relationship correlate with a higher degree of patient satisfaction. For example, patients in 133 Norwegian general practice clinics who had had a relationship with their primary care physician for more than 5 years were over one-third more likely to report being very satisfied with the quality of care than those with patient–doctor relationships lasting 1–5 years [26]. However, it is possible to have a poor long-term relationship and it is possible for a good relationship to develop over a short period of time. Thus, it seems more reasonable and appropriate to assess the strength of the interpersonal linkage between people and their health care providers [27]. In our study, there was a significant difference between patients who had had a relationship with their doctor for more than 5 years and those who had 1–5-year relationships with their physicians in the ‘personalized care’ domain, which is comparable to the longitudinality subdomain of ‘interpersonal relationships’ of the PCAT.

This study has several potential limitations. First, assessment of primary care practices based on user experience is only one aspect of primary care evaluation. If providers were to assess the performance of primary care, the results may have different features. Second, a previous study reported that users with a regular source of care were likely to evaluate their provider's level of performance as being very high or favorable [28], and this tendency may be even higher in Korea, where only 30% of adults have a regular health care provider [29]. Third, the items with relatively high rates of ‘don't know’ responses (e.g. >30%) may have low sensitivity for evaluating the performance that the item represented (e.g. items on Pap smear, community context, etc.). Fourth, this study was confined to a sample of patients who visited family physicians, and the results may not be sufficient to represent primary care as a whole in Korea. Fifth, it may be more appropriate to represent some ‘scales’ as ‘indexes’ (a sum of the scores) rather than averages, particularly the domains related to accessibility. In addition, the scale, ‘FC,’ was established based on an arbitrary decision made by the authors rather than statistical evidence.

In conclusion, the KPCAT, which was designed to comprehensively assess primary care practices from the patient's perspective, appears to measure important features of primary care performance, even though all scales are not sound enough to show good psychometric properties of the KPCAT, v. 1. The results of our study will provide valuable information about primary care practices in Korea. The next phase of the study will be to identify the characteristics of providers and health care organizations that account for the observed differences in primary care performance.

Acknowledgments

We are indebted to Stephen J. Spann, MD, MBA and Robert J. Volk, PhD of Baylor College of Medicine for their contribution to the initial planning of this project and their final reviews of the manuscript. We are grateful and appreciative to Professor Barbara Starfield of Johns Hopkins University for his helpful comments on a later draft of this manuscript. This project was supported in part by the Academic Award from the Korean Academy of Family Medicine (2005).

References

View Abstract