OUP user menu

Is it enough to set national patient safety goals? An empirical evaluation in Taiwan

Cathy Hui-Ying Wung, Tsung-Hsien Yu, Chung-Liang Shih, Chung-Chih Lin, Hsun-Hsiang Liao, Kuo-Piao Chung
DOI: http://dx.doi.org/10.1093/intqhc/mzq093 420-428 First published online: 17 January 2011

Abstract

Objective In 2004, the Taiwan Department of Health set the national patient safety goals. To date, however, there has been no evaluation of these goals. This study aimed to develop a method to evaluate the status of the national patient safety goals in Taiwan.

Design A cross-sectional questionnaire surveys to measure achievements on Taiwan's national patient safety goals. This survey was also followed up with an onsite audit to ensure accuracy.

Setting All hospitals in Taiwan.

Participants A total of 361 hospitals in Taiwan respond to questionnaire survey and 80 randomly selected hospitals for onsite audit.

Intervention(s) None.

Main Outcome Measure(s) Average scores on achievements of the national patient safety goals.

Results Among the 516 hospitals to which the questionnaire was sent, 361 (70%) responded. A total of 80 hospitals were randomly selected according to geographic location and size for onsite audit. The results show that the longer the period of implementation, the higher the average scores on achievements of the goals. After stratified analysis by hospital size, the large hospitals were found to have a higher average score in every goal, especially in the new goals. Furthermore, in terms of the difference between self-report results and the onsite audit, the score in the self-report was higher than the score given by experts upon onsite audit; however, they were similar. Most items were approximately the same in the self-report score and the onsite judgment, and those that differed were merely either one rank higher or lower.

Conclusion The self-report questionnaire combined with an onsite audit appears to be a promising approach for measuring scores on achievements of the national patient safety goals. The Department of Health could conduct this program annually to evaluate the progress and propose coping strategies.

  • national patient safety goals
  • project evaluation
  • questionnaire
  • onsite audit

Introduction

In 1999, the US Institute of Medicine published the landmark report ‘To Err is Human’, which issued a mandate ‘to set the national goals for patient safety, track progress in meeting these goals, and issue an annual report to the President and Congress on patient safety.’ In 2003, the Joint Commission began promoting national patient safety goals. Australia and the UK also took similar actions with the Australian Council for Safety and Quality in Health Care's Priority Programs and the UK's NPSA patient safety initiatives, respectively. The World Health Organization (WHO) also launched the World Alliance for Patient Safety in response to a World Health Assembly Resolution in 2002 that urged the WHO and member states to pay the closest possible attention to the problems of patient safety [1]. Following the trend, the Taiwan Department of Health established the Patient Safety Committee in February of 2003 to promote the patient safety movement and echo public concern. In 2004, the Patient Safety Committee invited experts on patient safety-related fields to set up the annual patient safety goals, strategies, principles and reference procedure. The number of goals went from five in 2004 to eight in 2007; of the latter eight, five consisted of the original goals from 2004 (Table 1). These organizations also drafted the details for implementation, such as the development of requirements and implementation expectations for the national patient safety goals by the Joint Commission. Taiwan also formulated implementation strategies for its national patient safety goals.

View this table:
Table 1

Taiwan patient safety annual goals year of implementation

GoalYear initiated
Improving the safety of using medications2004
Reducing healthcare-associated infections2004
Eliminating wrong-site, wrong-patient, wrong-procedure surgery2004
Improving the accuracy of patient identification2004
Preventing patient falls2004
Encouraging staff to report any incident2005
Improving communication and safety during patient transfer and handoff patients2006
Increasing patient and customer involvement2006

Tracking progress in the achievements of goals is another important issue. Although many organizations have devoted themselves to building a safer health environment, the question of how one evaluates these projects is still unclear. Through a literature review, we found many studies that used personal observation [2, 3], questionnaires [4, 5], score cards [6] and so on to assess the current situation. However, it was hard to find relevant information on the evaluation of patient safety goals. In the USA, the Joint Commission evaluates the national patient safety goals through the accreditation process [7]. Accreditation is costly, however, and the details often are unclear. Therefore, 5 years after national patient safety goals were established in Taiwan, it has become necessary to develop a cost-effective mechanism by which to evaluate progress in the achievements of these goals. This study was designed to develop a national patient safety goals self-evaluation questionnaire and to evaluate the achievements of the goal. In addition, we conducted onsite audits to compare the differences between the two approaches.

Methods

To develop the national patient safety goals self-evaluation questionnaire, we collected 131 candidate items, consisting of 23 items in goal 1, 16 items in goal 2, 17 items in goal 3, 9 items in goal 4, 17 items in goal 5, 16 items in goal 6, 22 items in goal 7 and 11 items in goal 8, from Taiwan's national patient safety goals implementation strategies and referential practices. These implementation strategies and referential practices were made by Taiwan's Patient Safety Committee of the Department of Health. All of these items had scientific literature support, for example, medication safety [810], infection reduction [1012], appropriate surgery [10, 12, 13], patient identification [10, 14], patient fall prevention [10, 12, 15], incident reporting [10, 16], communication/handoffs [10, 17, 18] and patient involvement [10, 19].

We also convened 60 experts (6 anesthesiologists, 17 surgeons, 6 internists, 5 pharmacists, 5 medical technicians, 9 nursing staff members, 7 university professors and 5 hospital managers) to select items. Every goal was reviewed by 15 experts, each of whom was assigned two goals.

In July 2007, mail was sent to these experts asking them to rank each of the candidate items on a 9-point ranking from a scale of 1 (totally disagree) to 9 (totally agree) for importance and feasibility. A total of 48 experts responded, yielding a return rate of 80%. Originally, the plan was to conduct two rounds of the Delphi survey and adopt the Rand/UCLA Appropriateness Method [20, 21] to choose the items, but the result of the first round of the Delphi survey was too consistent to allow differentiation. Therefore, the second mailing was not performed (appendix material is available from the authors on request).

We also modified the appropriateness method criteria to select items because of the first-round result. The score was divided into 3-point regions (A: 7–9, B: 4–6 and C: 1–3); we excluded items that more than one expert rated 4–6 in the importance and feasibility fields. A total of 33 items were kept at this stage, after expert panel discussion. Three items (operation site marking, anesthesiologist personally communicate and explain to the patient, and other measures to identify the patient) were re-included for comprehensiveness. We also wanted to know the overall achievement except for these selected items in every goal; thus, we added overall implementation achievements as the last item of every goal.

The level of implementation was divided into five ranks: full implementation (rank 4), implementation of most (rank 3), partial implementation (rank 2), implementation of a few (rank 1) and not available or not applicable (rank 0). For ease of calculation, rank 4 was coded 100 points, rank 3 was coded 75 points, rank 2 was coded 50 points, rank 1 was coded 25 points and not available or not applicable was coded 0. Finally, some questions were added concerning the hospital's characteristics, manpower and patient safety activities. Before the survey began, 19 hospitals of different levels and districts were invited to do a pre-test in June 2007, and the feedback from these hospitals was taken to modify the wording.

Survey administration and onsite audit

The national patient safety goals self-evaluation questionnaire was mailed to 516 hospitals across the nation between August and September 2007 with an attached letter to each superintendent in the hope of maximizing the response rate. At the same time, through this questionnaire, we also investigated the hospital's willingness to participate in an onsite audit. The onsite audit was conducted to ensure the accuracy of the self-report. According to hospital accreditation requirements, these hospitals were divided into three groups according to bed quantity: <100 beds, 100–249 beds and >250 beds. Among the 516 hospitals, 361 (70%) responded and 152 agreed to the onsite audit. The distribution of the responding hospitals was similar to the distribution of all hospitals, and large hospitals (250 beds and above) had a relatively higher response rate (83%) (Table 2)

View this table:
Table 2

Questionnaire overall response rate

Hospital size<100 beds100–249 beds>250 bedsP-value
No. of responding hospitals196581070.12
Total number across nation30285129
Response rate (%)656883

Subsequently, 80 hospitals were randomly selected according to geographic location and size. The Taiwan Joint Commission on Hospital Accreditation patient safety task force, which included 12 physicians, 2 pharmacists, 1 medical technician, 4 nursing staff members and 4 hospital managers, was invited to the onsite audit.

For consistency of the onsite audit, a consensus meeting was held in September 2007. All surveyors were required to attend the meeting, and the onsite audit procedure was adapted according to the Joint Commission's tracer methodology. We developed three tracers (emergency care, outpatient care and inpatient care); the surveyor used these tracers as a road map to move through an organization; they also used the same questionnaire to assess and evaluate the hospital self-evaluation. In addition, a standard booklet was issued to ensure the consistency of the audit procedure.

Statistical analysis

The scores on achievements of every goal were calculated by an indicator average approach, by which an equal weight was assigned to each individual item in each goal. Descriptive statistics were used to examine the hospital's characteristics, status of hospital patient safety implementation and patient safety goal achievements. We also conducted the chi-square test to examine the difference between agreeing and not agreeing to attend the onsite audit. We carried out an ANOVA test to examine the difference in overall achievements among hospitals of different size, and Scheffé’s test for post hoc comparison. Finally, to compare the differences between the hospital self-report and the onsite audit, we conducted a paired t-test. All analyses of this survey were stratified by hospital size; in addition, we calculated the distribution of rank discrepancies. Statistical testing was two-sided and P < 0.05 was considered statistically significant. For all statistical analyses, SPSS software, version 16, was utilized.

Results

Among the hospitals that responded to the survey, 98% had patient safety committees. Most of these hospitals (68%) had at least one designated staff member for patient safety affairs, 21% of the hospitals did not have such designated staff and 1% did not answer this item. The chairperson of the committee was usually the superintendent or deputy superintendent of the hospital (90%). Of the hospitals with patient safety committees, 77% have set an annual patient safety goals working schedule, 17% did not have a safety plan and 5% have adopted a medical system's safety plan (e.g. strategic alliances).

Table 3 shows the results of the national patient safety goals self-report. All goals received similar average scores (86–97 points), except for goal 3 (‘Eliminating wrong-site, wrong-patient, wrong-procedure surgery’). Goals 1, 2 and 4, however, had smaller variations than did goals 6, 7 and 8. On average, the larger hospitals demonstrated higher average scores on the goals (91.4–98.1 points) and demonstrated smaller variation (4.6–13.7 points) in scores; the variation of new goals was also larger than that of old ones, especially in small size hospitals. In addition, hospitals located in metropolitan areas score higher on the goals than those in non-metropolitan areas.

View this table:
Table 3

Average implementation score of self-report questionnaire stratified by hospital size

Annual GoalA11G1 (size <100 beds)G2 (size =100–249 beds)G3 (size >250 beds)F-value
nMean (SD)nMean (SD)nMean (SD)nMean (SD)
Improving the safety of using medications34994.2 (8.8)19092.1 (10.0)5894.2 (8.3)10198.1 (4.6)16.69***
Reducing healthcare-associated infections35194.8 (8.1)19393.1 (9.1)5895.9 (5.8)10097.4 (5.9)10.80***
Eliminating wrong-site, wrong-patient, wrong-procedure surgerya28196.6 (7.5)14595.5 (8.3)4897.3 (6.8)8898.1 (6.2)3.47*
Improving the accuracy of patient identification34897.1 (7.6)19196.3 (8.4)5897.0 (8.4)9998.6 (5.5)2.89
Preventing patient falls35194.8 (11.1)19293.1 (12.0)5895.3 (7.6)10197.4 (10.8)5.01**
Encouraging staff to report any incident34686.2 (13.67)18882.2 (14.8)5887.9 (12.2)10092.6 (9.2)21.82***
Improving communication and safety during patient transfer and patient handoffs35285.7 (17.2)19582.6 (18.5)5686.0 (16.0)10191.4 (13.7)9.09***
Increasing patients’ and customers’ involvement35385.6 (17.8)19580.5 (20.4)5789.0 (13.6)10193.4 (10.0)20.67***
  • *P < 0.05; **P < 0.01; ***P < 0.001.

  • aHospitals that answered N/A in all items in goal 3 were omitted.

In this survey, we found that the hospital's willingness to attend an onsite audit might be influenced by hospital size: large hospitals tended to attend the onsite audit, whereas only one-fourth of small hospitals agreed to attend the onsite audit. Hospitals that agreed to attend the onsite audit scored higher on the goals and showed smaller variations in scores than hospitals that did not attend the onsite audit (Table 5). The geographic factor did not have influence on the hospital's willingness to attend an onsite audit.

View this table:
Table 4

Comparison of attend and not attend onsite audit among patient safety goals

Not attend the onsite audit mean(SD)Attend the onsite audit mean(SD)P-value
Goal 192.90 (9.58)95.89 (7.38)0.001
Goal 293.76 (8.61)96.10 (7.27)0.006
Goal 371.50 (42.17)84.89 (33.71)0.001
Goal 496.14 (8.51)98.34 (6.09)0.005
Goal 593.01 (13.42)97.21 (5.90)<0.001
Goal 683.01 (14.93)90.42 (10.41)<0.001
Goal 782.68 (17.72)89.99 (15.54)<0.001
Goal 882.90 (18.96)89.36 (15.48)<0.001
  • Note: Large size hospitals had higher attendance rate (69.3%) than middle (50%) and small (26.5%) size hospitals.

View this table:
Table 5

The comparison of average implementation scores between self-report and onsite audit stratified by hospital size

Annual goalG1 <100 beds (n = 33)G2 100–249 beds (n= 14)G3 >250 beds (n= 33)
Self-reportExperts’ opinionP-valueSelf-reportExperts’ opinionP-valueSelf-reportExperts’ opinionP-value
Improving the safety of using medications92.7 (8.5)84.6 (11.8)<0.000192.6 (9.1)89.0 (10.4)0.11197.6 (6.0)97.2 (4.7)0.447
Reducing healthcare-associated infections93.9 (7.4)90.9 (8.6)0.09796.2 (6.0)92.1 (8.1)0.10096.7 (7.1)94.8 (7.6)0.158
Eliminating wrong-site, wrong-patient, wrong-procedure surgery76.4 (41.3)70.1 (41.9)0.07491.5 (26.6)88.8 (26.1)0.11195.1 (18.2)94.5 (18.0)0.447
Improving the accuracy of patient identification97.7 (7.3)94.4 (10.1)0.11398.8 (3.0)95.2 (7.8)0.13998.2 (5.4)97.5 (6.7)0.620
Preventing patient falls94.6 (8.6)89.3 (11.4)0.00395.7 (5.8)92.6 (6.5)0.09798.5 (4.7)96.3 (6.8)0.05
Encouraging staff to report any incident85.2 (13.3)78.7 (14.8)0.00790.8 (9.1)89.0 (10.2)0.029*93.8 (8.6)90.3 (8.5)0.008
Improving communication and safety during patient transfer and patient handoffs87.0 (20.0)83.1 (20.5)0.06584.9 (14.1)82.1 (13.2)0.09594.4 (10.2)92.3 (10.6)0.313
Increasing patients’ and customers’ involvement83.4 (19.4)76.4 (20.5)0.00788.4 (13.2)88.4 (10.1)1.00093.94 (11.95)93.94 (12.45)1.000
  • Note: Mean (SD).

The onsite audits were conducted from October through November 2007. Table 5 presents the average score discrepancies between the self-report and the onsite audit. The results show that the hospital self-evaluation was slight higher than the onsite audit, but the variations were minimal in the hospital self-evaluation. There were statistically significant differences in scores on goals 1, 5, 6 and 8 among small hospitals. And goal 6 showed a significant difference in medium and large hospitals. However, as we went further to check the rank discrepancies, we found that large-scale discrepancies were not common; the most seen modification was a one-rank adjustment.

Discussion

The major findings were that larger hospitals scored higher on the goals than smaller hospitals and that the earlier implementation goals (goals 1–5, see Table 1) were more fully achieved. The second finding was that experts demonstrated a high degree of consensus about the national safety goals. The last one was that the self-report questionnaire demonstrated a high degree of consistency with onsite audit. Several issues remain to be discussed. The first pertains to item selection. As mentioned above, we intended to adopt the two-round Delphi method and appropriateness method to select items [21, 22] because these approaches are the most popular in indicator development. Our approaches were developed with evidence and expert consensus [23], but these items were mostly part of current work. Therefore, it was difficult to discriminate which items were important and feasible. Although we modified the appropriateness method criteria, our criteria were stricter. According to the appropriateness method's definition, if there are less than three or four members ranking outside the region containing the median, the panel has agreement with this item. Hence, the most important and feasible items were selected by our panel, making this approach as scientifically sound as the appropriateness method.

The second additional point of discussion is the questionnaire format design. Taiwan's national patient safety goals focus on acute care, and most of them are suitable for psychiatric and chronic care, except for goal 3. In addition, small hospitals in Taiwan typically do not perform surgery at the present; goal 3, therefore, results in a lower average score and larger variations because the rate of these hospitals is zero. After the hospitals that do not implement surgical programs were removed from consideration, the overall achievement score for goal 3 became 96.6 with a standard deviation of 7.5. In the preliminary design of the survey, the ‘not available’ and ‘not applicable’ terms were combined into one category to avoid confusion, but this consideration becomes a limitation in discriminating which response actually applies. Future research should consider this.

The third issue is the feasibility of the national patient safety goals self-evaluation questionnaire. There are several reporting systems (Taiwan Patient Safety Reporting System and National Reporting System of Adverse Drug Reactions) and quality indicator projects in Taiwan, but the data from these cannot be released for confidentiality reasons. Thus, we adopted two subjective methods to evaluate national patient safety goals without other sources. Although the differences were inevitable, the purpose of this survey was not to compare hospitals’ scores on achievements of the goals, but to combine the self-report questionnaire and the onsite audit approaches to evaluate the current state. In this survey, we found that the discrepancies between self-reports and onsite audits were not great. Most of them only ranked one value above or below, and the results still can reflect the current state. The survey showed that the results of the national patient safety goals self-evaluation questionnaire can be used as references through the small-scale onsite audits adjustment. We suggest that this survey be conducted annually and that it be combined with small-scale onsite audits (e.g. annual inspection by local authorities or accreditation programs). The more implementation items or goals that can be retired or rotated, the better those items or goals can be put into annual inspection or accreditation [24] to allow hospitals to focus their attention on the important ones.

The fourth topic for discussion is the ability to measure national patient safety goals. Most quality improvement projects have indicators that policy makers or project managers can use to track progress [25], but Taiwan's national patient safety goals lack such markers. Hence, we developed the national patient safety goals self-evaluation questionnaire through a literature review, rather than from the national patient safety goals themselves. Recently, patient safety indicators (PSIs) have become popular in the mainstream. AHRQ, OECD [26] and other researchers [27] have developed PSIs; there also is a study [28] in Taiwan to develop PSIs. However, there are still some challenges with PSIs [29, 30]. Such challenges always focus on specific practices, but the national patient safety goals are a broader concept. It is not easy to link these two domains. Even so, we still can set a clear target with each goal or develop a checklist [31] for improving safety, and it also is the way of further works.

Last but not least, it is important to enhance regional cooperation. The Department of Health of Taiwan has been implementing the medical network integration project since July 1985. One of the main goals is to construct a system of regional healthcare and balance the resource differences of rural and urban areas. After the year 2000, policy directions shifted to patient safety promotion. The medical network integration project separates Taiwan into six regions; we found in our visits that small hospitals indicate not having clear ideas to implement national patient safety goals because they are too small to have the economic scales to develop various materials for national patient safety goals. If large hospitals can share their experiences through regional cooperation, it would not only conform to the cost benefit but would also improve the difficult situations of small hospitals.

Limitation

There were several limitations to this study. First, the questionnaire format was not completely clear. In an attempt to minimize confusion, the ‘not available’ and ‘not applicable’ terms were combined into one category. However, some small hospitals and psychiatric hospitals did not have surgical operations and thus they responded with N/A answers in goal 3. Hence, we could not evaluate the achievements of the national patient safety goals exactly. The second limitation was selection bias, a shortcoming of the questionnaire survey. The actual situation of a non-respondent and one not willing or able to join the onsite audit cannot be inferred from our result. A final limitation was the absence of a gold standard with which to compare.

Conclusion

In summary, we investigated the implementation status of patient safety goals in Taiwan. We found that those goals having a longer implementation period had higher achievements; larger hospitals also had higher achievements. The results of the self-evaluation questionnaire and the onsite audit were similar. This indicates that the self-evaluation questionnaire can measure the national patient safety goals achievements effectively, although it might need a validation through an onsite audit where possible. The results of this survey can be a reference for the setting of new goals in the future. We suggest that this process be included in setting explicit goals for continued monitoring of patient safety.

Acknowledgements

This study was supported by a grant from the Department of Health (DOH-96-TD-M-113-025). We thank the 74 experts and 80 hospitals for their cooperation in joining the onsite audit voluntarily. We also appreciate the Ministry of Education, Aiming for Top University (MoE ATU) Plan for its support and cooperation.

Appendix

View this table:
Appendix 1

National patient safety goals self-evaluation questionnaire

Goal 1 Improving the safety of using medications
 A pharmacist dispenses SOP with a review mechanism
 The medicine pouch has all 13 labeled items
 The patient's drug allergy information is displayed in the index of the medical chart or presented on the computerized physician order entry (CPOE) system
 Medication or other solution separated from the original packaging is clearly relabeled on the container (such as injections, medicine cups or basins)
 High-alert medications are labeled, and a complete administrative mechanism and clear labeling rules are established
 Overall implementation of medicine safety improvement at your facility
Goal 2 Reducing healthcare-associated infections
 Adequate sinks are set up or alcohol hand antiseptic is provided to improve the convenience for hand washing
 Routine inspection is conducted and various types of hand washing equipment are maintained at each department
 The infection control unit is in operational status
 A healthcare-associated infections control manual is being implemented
 A healthcare-associated infections tracking system is being implemented
 Routine inspection is carried out on cleaning, disinfection and sterilization measures of various equipment
 Overall implementation of health care-associated infections at your hospital
Goal 3 Eliminating wrong-site, wrong-patient, wrong-procedure surgery
 Preoperative verification is conducted and the patient's identity is confirmed
 Preoperative operation is conducted and the surgical site and markings are confirmed
 The surgeon and the anesthesiologist personally communicate and explain to the patient the information regarding to the surgery and the anesthesia
 Overall implementation of improving surgical site accuracy at your facility
Goal 4 Improving the accuracy of patient identification
 The identity of the patients is confirmed before admission, and some sort of measurement is used to attach the information (such as bracelets) with the patient for verification by the medical staff
 Medical staff will use other measures to identify the patient if no bracelet is found
 Overall implementation of improving the accuracy of patient identification at your facility
Goal 5 Preventing patient falls
 The hospital ensures that the environment is not messy and the floors are not wet
 Relevant measures are established to prevent falls and to train the medical staff
 Environment orientation is conducted and instruction is given on how to use the call button, and on slips and falls prevention
 Signs are used to identify high-risk patients (such as stickers, hang tags, arm bands or bracelets) to remind the staff and the caregivers to pay attention
 Fall prevention education or relevant measures are provided to the patient and the family members; posters, flyers or multimedia on fall prevention are available with a list of precautions
 A reporting procedure is established; the records identify who, when, where and the contributing factors; follow-up monitoring and evaluations are carried out
 Overall implementation of fall prevention at your facility
Goal 6 Encouraging staff to report any incident
 A clearly defined scope is established for incident reports
 The patient safety reporting process is established
 A convenient and independent reporting channel is available
 A management mechanism and a procedure after the reporting are available
 Relevant units and reporting staff receive feedback on the result of the analysis and the effectiveness of process improvements
 There is involvement of the national patient safety reporting system
 Overall implementation of encouraging incident reports at your facility
Goal 7 Improving communication and safety during patient transfer and handoff patients
 The SOP of new staff training covers the ‘Improving communication and safety during patient transfer and handoff patients’
 A safe environment for patients who require transfer (such as the location for ventilator, IV pump and EKG monitor) is provided
 Confirmation that the necessary materials are already prepared (such as medical chart, X-ray slides, Buscopan and other medications) is made before transferring the patient
 Should the patient's condition worsen during a transfer, emergency response measures are available (such as calling the emergency hotline, emergency transportation to the nearest nursing station for emergency treatment)
 Procedures and reports are available for the communications between the departments and cases of abnormal transfer to facilitate solid improvements
 Overall implementation of improving the communication and the safety of transferring patients at your facility
Goal 8 Increasing patients and customers involvement
 A delegated department is responsible for tracking, managing and handling complaints
 Relevant health education is sponsored to enhance patient safety and impart the knowledge to the patient
 Posters are placed in the ward, waiting area, resting area and so on (including actively inquire about treatment plan, pharmacy consulting, participation of patient identification etc.) to encourage the public to participate in activities to enhance patient safety
 Overall implementation of increasing patients and customers involvement

References

View Abstract