International Journal for Quality in Health Care Advance Access originally published online on March 12, 2008
International Journal for Quality in Health Care 2008 20(3):184-191; doi:10.1093/intqhc/mzn004
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Attitudes toward the large-scale implementation of an incident reporting system
1 Centre for Clinical Governance Research, University of New South Wales, Australia
Address reprint requests to: Joanne Travaglia, Research Fellow, Centre for Clinical Governance Research, Faculty of Medicine, University of New South Wales, Sydney, NSW 2052, Australia. Tel: ++61 2 9385 2594; Fax: ++61 2 9663 4926; E-mail: j.travaglia{at}unsw.edu.au
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Objective. An electronic Incident Information Management System implemented system-wide by the Department of Health, New South Wales, Australia was evaluated. We hypothesized that health professionals (i) would support the system via utilization and favourable attitudes and (ii) that their usage and attitudes would vary according to profession with nurses being most, and doctors least, favourably disposed.
Design, setting and participants. An online, anonymous questionnaire survey of 2185 health practitioners.
Main outcome measures. Undertaking system training, satisfaction with training, reporting incidents, incident reporting rates since system introduction and attitude questions focusing on use, security and evaluation of the system and workplace safety cultures.
Results. The first hypothesis received partial support. The majority of respondents had undertaken training and rated it highly. Most had reported incidents and maintained their previous reporting levels. Most attitudes regarding using the system and its security were favourable. Mixed attitudes were held about workplace safety cultures and the value of the system. Deficiencies in quality of reporting, feedback on incident reports and resources to analyse incident data were problems identified. The second hypothesis was confirmed. Nurses were most, and doctors least, likely to undertake training, report incidents and express favourable attitudes. Allied health responses were intermediate to those of the other professions.
Conclusions. The system implementation was relatively successful, but more so with some professions. Problems identified indicated that expectations as to the goals achievable in the short term were optimistic, but these are amenable to planned interventions.
Keywords: electronic incident reporting, health professionals, patient safety, quality improvement, safety attitudes, systems change
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Incident reporting
Incident reporting is generally held to be a core initiative in addressing patient safety [1]. Considering that studies in several countries have shown that 10% of patients admitted to acute settings are harmed [2, 3] it is not a trivial strategy in international context to try to improve incident reporting and the subsequent management of incidents detected [4]. According to proponents, the analysis of incidents (defined as adverse events and near misses) can provide information on which to base policy and practice decisions likely to reduce future occurrences [5]. Aggregated together, data on multiple incidents have the potential to help identify patterns, trends and categories of incidents for follow-up, creating opportunities for systems improvements [6]. Specific or synthesized information can be shared with other providers thereby supporting the diffusion of innovations and improved policies or practices. Shared data can also identify unsafe activities [7]. Deconstructing incidents and aggregating the information can help garner insights which in turn assist in the formulation of strategies such as redesigning equipment or devising forcing functions.
Structured incident management systems, usually software-based, have emerged by which to capture instances of harm or near misses [8]. Definitions and taxonomies [9, 10] of what constitutes a reportable incident have been developed, as have systems to categorize reportable incidents according to severity, impact or potential impact and the likelihood of incident types recurring [11]. The severity assessment coding category of the Veterans Administration in the USA of America is widely used [12].
Problems and barriers
There are various barriers to reporting. First, individual hurdles include concerns about personally admitting a mistake, reluctance to appear foolish or incompetent, discomfort at reporting confidential information, uncertainty as to how to report, fear of litigation and worry about reprisals [13]. Expectations of how others view professionals, reputation management and feelings of personal or job insecurity can each play a part in poor reporting levels [14]. Second, collective constraints include inhibitive reporting cultures, lack of adequate systems [15] and patterns of sociologically implicit or explicit discouragement or blaming mores [14]. Significant differences have been found in the attitudes toward incident reporting of various health professions with nurses and allied health staff being more favourably disposed than doctors [16, 17]. A third set of stumbling blocks beyond the individual and collective is technical [18]. User dissatisfaction, poor software performance, lack of structured reporting back and running costs [19, 20] are frequently cited reasons for information technology failure.
Clearly, it will be hard to change the reporting culture across entire health systems. Despite good intentions and high aspirations for incident reporting expressed in the literature there has been limited empirical demonstration of system-wide success. There are suggestions to the effect that as electronic information reporting systems are embraced, particularly as a component of a relatively blame-free environment, increases in the numbers of detected adverse events, rather than decreases in the numbers of actual adverse events due to improvements, are likely to prevail [21]. This set of circumstances may well persist for some time [6].
Context of the present research
In 2002, the Department of Health and the Clinical Excellence Commission in New South Wales implemented a Safety Improvement Programme. In the following 3 years, training courses were held for 3500 senior staff. Most courses comprised 2-day structured educational sessions focusing on safety improvement, classifying incidents using a severity assessment code and conducting root cause analyses to determine causes of incidents. A multi-methods, triangulated evaluation of this programme indicated that participants rated it highly and had applied the safety practices learnt [22]. In 2005, over half of this group had served on root cause analysis teams and 88% believed the process had improved patient safety [23].
In 2004–05, a core component of the safety improvement programme, an electronic Incident Information Management System (which is known and referred to as IIMS), was installed across the state in publicly funded facilities superseding a multiplicity of electronic and paper-based schemes. The Incident Information Management System is a localized version of the Advanced Incident Management System [9] of the Australian Patient Safety Foundation [24]. Its objectives are to record all healthcare incidents, assist managers through a workflow module to manage incidents in their area and to produce reports and analyses of all incidents in the system. The introduction of the system was supported by tailored training programmes to ensure staff had practical skills to utilize the system.
Aims and hypotheses
Our research evaluated health practitioners' utilization of, and attitudes toward, the Incident Information Management System a year after its introduction. The hypotheses were derived from findings of previous research into incident reporting [13, 15, 17] and earlier stages of the New South Wales patient safety initiatives [16, 22, 25]. We predicted (H1) that health practitioners would support the system as indicated by undertaking training in use of the system, rating their training highly, utilizing the system by reporting incidents, maintaining or exceeding their previous incident reporting levels, and expressing favourable attitudes regarding the use, security and value of the system and the safety culture in their workplaces. We also predicted (H2) that health professional groups (doctors, nurses and allied health professionals) would differ in their responses to the experiential and attitudinal variables investigated by the first hypothesis with nurses being the most, and doctors the least, favourably disposed toward the system.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Questionnaire
A questionnaire was developed, drawing on work in related research [19, 26–28], and advice of those involved in the system's implementation, to investigate health professionals' demographic characteristics (professional background, type of facility where most work conducted, years of work in healthcare post-graduation, gender, percentage of work spent on managerial duties), training in and use of the system (see Table 2 for items) and attitudes towards the system (see Table 1). The attitude items consisted of rating scales ranging from disagree strongly (5) through neutral (3) to agree strongly (1). The final four items were only answered by respondents with some managerial responsibilities. Other items covering workplace details, technical knowledge of the system and text comments on improvements needed are not included in this report. The questionnaire's reliability as measured by Cronbach's alpha was 0.83 [29].
|
|
Setting and sample
A third of the Australian population of 21 million reside in the state of New South Wales. The state's health system is a mixed public–private system, with about two-thirds of health services provided publicly. The publicly funded health system employs some 100 000 people and is divided administratively into 11 Area Health Services, broadly the equivalent to English National Health Service trusts. The Australian health workforce, which New South Wales mirrors, is comprised 13.3% doctors, 54.2% nurses, 10.5% allied health staff and 22.0% others [30]. The research's target population was staff in all public health facilities in the state. As Table 1 shows doctors were under-represented and allied health staff over-represented in the sample of health practitioners who responded to the survey.
Procedure
The research team was independent of the implementation process of the system. The anonymous questionnaire was placed on a secure, dedicated website for a month from April 2006. Access to staff was negotiated with senior administrators in all Area Health Services. They invited their staff to participate in the survey via internet mail, weekly staff bulletins or intranet websites. The invitation gave directions to follow the link to the website and complete the questionnaire.
Responses of the professional groups were compared using Chi-square analyses, ANOVAs and Duncan Range tests. In this report, to facilitate ease of comprehension of the large number of results, some of the response categories were combined. Significance level was set at 0.05.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Characteristics of respondents
The demographic characteristics of respondents are shown in Table 1. Comparison of the demographic characteristics of four professional groups revealed several significant differences. Doctors (46.2%) and ‘other’ staff (58.8%) were less likely to be women than were allied health (76.3%) and nurses (85.7%) (P < 0.001). Doctors (19.8 years) and nurses (20.6 years) had worked longer in healthcare post-graduation than allied health professionals (16.8 years) who had greater experience than the ‘other’ group (14.4 years) (P < 0.001). Among those with managerial responsibilities nurses (74.7%) and ‘other’ staff (73.7%) estimated that they devoted more of their work time to these activities than did allied health staff (64.7%) or doctors (43.4%) (P < 0.001). Although the sample represents 2.2% of the total health workforce, we do not know what proportion of the workforce received information about the survey. Evidence emerged of variations in Area Health Services' distribution of invitations to staff to participate.
System utilization and attitudes of total health professional sample
Table 2 gives details of respondents' training in use of the system, their evaluation of training received and their incident reporting since the introduction of the system. Respondents' type of training was significantly associated with their reported skills acquisition (P < 0.001). Participants who had attended courses were more satisfied (78.5%) than were those who received an explanation from a colleague (68.8%) or learnt via a CD, DVD or online (65.4%). The majority had reported incidents on the system but most had not changed their rate of reporting. Those who had changed their reporting patterns were slightly more likely to have decreased than increased their reporting.
Table 3 shows the responses of the total sample to the attitude items. The most favourable attitudes were held regarding the ‘Security of the System’ with most agreeing that it was secure for staff and patients. In the domain, ‘Use of the System’, more favourable than negative attitudes were given to 75% of items. ‘I have a sufficient understanding of what defines a reportable incident’ attracted the strongest endorsement (79.2%) of all attitude items. The items attracting most disagreement asserted that reporting incidents on computer were easier than using paper and that the follow-up status of incident reports can be determined. Although more respondents agreed than disagreed that their workplace had a non-punitive reporting culture few thought there was a philosophy of ‘the higher the number of incident reports the better’. In the domain ‘Evaluation of the System’ only half the items attracted greater agreement than disagreement. These asserted that the system had improved patient safety and that reporting was a good use of staff time and resources. However, few respondents considered that the system encouraged open disclosure to patients and only a quarter agreed that computerized incident reports were more accurate than paper reports. This evaluation domain evoked the most ‘neutral’ responses.
|
Staff with managerial responsibilities have a broader view of the system as many have the responsibility of receiving and dealing with incident reports using the workflow module. They are those likely to request reports of system data. In managers' evaluation as canvassed in the final four items, the majority agreed that the system provides incident data to departments in a timely fashion and that it has increased knowledge of quality and risk measures. However, more managers disagreed than agreed that reports from staff contained all relevant details of the incidents and that analysis of incident data by the system is prompt.
Professional differences in utilization and attitudes
Table 4 compares the training and utilization of the incident reporting system by the three professional groups. Doctors were significantly less likely to have undertaken system training. Among professionals who had received training doctors were less likely than nurses and allied health staff to have attended a course and more likely to have learnt from a colleague or online. However, the professional groups' evaluations of the skills provided by their training were similar and mainly favourable. Doctors were least, and nurses most, likely to have reported an incident using the system. The most frequent response regarding reporting rates was that these had not changed with the introduction of the system. Thus, as doctors were least likely to have reported an incident on the system, we assume that prior to its implementation doctors reported fewest incidents and nurses the most.
|
Table 5 shows the results of the Chi-square analyses comparing the attitudes of the professional groups. The ‘Use of the System’ domain revealed significant professional differences in responses to seven of the eight items. Doctors were most likely to agree that they knew the severity ratings to assign and that reporting on computer was easier than on paper. Doctors were least likely to agree that there were enough PCs available, and that the system is easy to use, has a logical classification of incidents and permits follow-up of reports. Nurses were most likely to say that they understood what constitutes a reportable incident that the system's classification is logical and that the status of a report can be ascertained. They were least likely to claim that reporting on computers is easier than using paper. Allied health professionals were most likely to say their access to PCs was adequate. Although the professional groups held similar views on whether the system was operating when needed and provided security for staff, nurses were more convinced of the security provided for patients. Doctors were most, and allied health staff least, likely to say that they had a non-punitive work culture.
|
There were significant professional differences in responses to six of the eight evaluative items. Doctors were less likely to agree that computerized incident reports were more accurate than paper that the system provided timely incident data to departments, that resources are available to analyse incidents or that the system had improved knowledge of quality and risk measures. Nurses were those most likely to agree that the system encouraged open disclosure to patients. Nurse managers held the most favourable views on the ‘Further Evaluation’ items. Allied health participants were most likely to consider reporting is a good use of staff resources.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
There was support for the first hypothesis. Participants had embraced training and there was agreement that their training had imparted the skills to report incidents on the system particularly among the large proportion who had undertaken a course. Overall the incident reporting levels of the sample had been maintained following the implementation of the system. Although respondents did not perceive an overall increase in reporting rates, there is recent evidence that these actually did increase significantly [25]. Further increases in reporting should occur if barriers to reporting identified by the survey continue to be addressed. Participants held more favourable than negative attitudes regarding the system's use and value. Of the propositions listed in Table 3, 35% elicited agreement by over half of respondents. No items elicited disagreement by more than 50% of respondents. Another way of considering the data is that 13 of the 20 attitude items elicited more favourable than negative responses while the reverse occurred for seven items. The items showing the greatest discrepancy of negative over favourable responses were ‘The culture in this facility is "the higher the number of incident reports the better"‘, ‘Resources are available to analyse real-time incident data promptly’, ‘I can determine the follow-up status on any incident’, ‘Computerized reports are more accurate than paper incident reports’ and ‘The incident reports I review contain all relevant details’. Many of these may reflect typical teething problems associated with relatively new systems, or medium term issues. Although 42% of respondents agreed that they had a non-punitive culture of reporting in their workplace, 35% disagreed. This reflects differences in reporting cultures found between various facilities and Area Health Services which are being further investigated.
The second hypothesis was strongly supported. Significant professional differences were found in responses to 75% of the attitude items. This points to the professional divides which exist in healthcare, and the need to promote multidisciplinary teams [16]. Nurses gave the highest agreement ratings to 10, doctors to three and allied health staff to two items. Doctors displayed least involvement with the system, being less likely than other professionals to have undertaken training or reported an incident. Nevertheless they were the group most likely to complain of the lack of PCs. Their under-representation among survey participants also suggests less involvement while the over-representation of allied health respondents implies the contrary.
Our findings point to three aspects of incident reporting which need to be factored in to future research activities designed to study progress. First, it is important to measure attitudes toward incident reporting. Although these may or may not be specific to an electronic system, they can highlight areas which need to be addressed, e.g. further education, culture change and incentives. Second, research into existing electronic systems can provide vital information on aspects of software which can be improved. Third, we need more data on how software is deployed in health settings. There are very few arm's length health sector studies of software use in situ.
A limitation was that the study did not involve a random sample of health practitioners but instead was largely self-selected. Some Area Heath Services publicized and encouraged participation to a greater degree and provided a longer time period to answer the questionnaire. Some appear to have informed mainly senior staff about the survey. Allied health and staff with managerial roles were over-represented in the sample and doctors were under-represented when compared with workforce statistics. These factors limit the generalizability of the findings.
This is the first large-scale evaluation of an incident system used by over 100 000 people. Issues were identified in the culture, logistics and in the software. The survey has proven useful in allowing problems to be identified, and its recommendations in areas of the design of future versions, and the deployment of call centres as an alternative to direct electronic recording, have been, respectively, taken into account by the software designers and the New South Wales Health Department. Over time, these kinds of efforts may converge to realize the integrated framework for the management of safety, quality and risk that Runciman and his group have recently advocated [8].
|
Funding |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Funding for this project was provided by Clinical Excellence Commission, NSW, Australia; NSW Health, Sydney, Australia; and University of New South Wales, Sydney, Australia. The studies of which this paper reports part were conducted under the auspices of the quality improvement mechanisms of NSW Health; no additional ethics committee approval was warranted.
|
Acknowledgements |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
The support of key staff in NSW Health (Michelle Wensley, Kathleen Ryan, Jo Montgomery, Annika Sander, Debbie Edwards and Michael Smith), faculty of the NSW Safety Improvement Program (Paul Douglas, Tom Hugh, Sarah Michael and John Overton) and the staff of the NSW Clinical Excellence Commission (Cliff Hughes and staff) is appreciated. Maureen Robinson and Ian O Rourke stimulated the acquisition of IIMS in NSW, sponsored it, and were unwavering in their commitment to its evaluation. Sarah Michael managed IIMS for NSW Health and, subsequently, CEC. Sue Evans of the University of Adelaide, South Australia provided invaluable advice on a draft of the questionnaire and Angus Corbett, Bill Runciman and Peter Hibbert made insightful comments on an earlier draft. To those who completed questionnaires we express our gratitude.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
- Pittet D, Donaldson L. Challenging the world: patient safety and health care-associated infection. Int J Qual Health Care (2006) 18:4–8.
[Abstract/Free Full Text] - Thomas EJ, Studdert DM, Runciman WB, et al. A comparison of iatrogenic injury studies in Australia and the USA. I: context, methods, casemix, population, patient and hospital characteristics. Int J Qual Health Care (2000) 12:371–8.
[Abstract/Free Full Text] - Vincent C, Neale G, Woloshynowych M. Adverse events in British hospitals: preliminary retrospective record review. BMJ (2001) 322:517–9.
[Abstract/Free Full Text] - Wachter RM. The end of the beginning: patient safety five years after ‘To Err Is Human. Health Aff (2004) 23:W4-534–45.
- Vincent CA. Analysis of clinical incidents: a window on the system not a search for root causes. Qual Saf Health Care (2004) 13:242–3.
[Free Full Text] - Wood KE, Nash DB. Mandatory state-based error-reporting systems: current and future prospects. Am J Med Qual (2005) 20:297–303.
[Abstract/Free Full Text] - Williams SK, Osborn SS. The development of the National Reporting and Learning System in England and Wales, 2001–2005. Med J Aust (2006) 184:S65–68.[Web of Science][Medline]
- Runciman WB, Williamson JAH, Deakin A, et al. An integrated framework for safety, quality and risk management: an information and incident management system based on a universal patient safety classification. Qual Saf Health Care (2006) 15:82–90.[CrossRef]
- Runciman WB, Moller J. Lessons from the Australian Patient Safety Foundation: setting up a national patient safety surveillance system—is this the right model? Qual Saf Health Care (2002) 11:246–51.
[Abstract/Free Full Text] - Chang A, Schyve PM, Croteau RJ, et al. The JCAHO patient safety event taxonomy: a standardized terminology and classification schema for near misses and adverse events. Int J Qual Health Care (2005) 17:95–105.
[Abstract/Free Full Text] - Aspden P, Corrigan JM, Wolcott J. Patient Safety: Achieving A New Standard for Care (2004) Washington: National Academies Press.
- Bagian JP, Gosbee J, Lee CZ, et al. The Veterans Affairs root cause analysis system in action. Jt Comm J Qual Improv (2002) 28:531–45.[Medline]
- Lawton R, Parker D. Barriers to incident reporting in a healthcare system. Qual Saf Health Care (2002) 11:15–8.
[Abstract/Free Full Text] - Coles J, Pryce D, Shaw C. The Reporting of Adverse Clinical Incidents—Achieving High Quality Reporting: The Results of a Short Research Study (2001) London: Clinical Accountability, Service Planning and Evaluation (CASPE) Research.
- Barach P, Small SD. Reporting and preventing medical mishaps: lessons from non-medical near miss reporting systems. BMJ (2000) 320:759–63.
[Free Full Text] - Westbrook MT, Braithwaite J, Travaglia JF, et al. Promoting safety: varied reactions of doctors, nurses and allied health professionals to a safety improvement program. Int J Health Care Qual Assur (2007) 20:555–71.[CrossRef][Medline]
- Kingston MJ, Evans SM, Smith BJ, et al. Attitudes of doctors and nurses towards incident reporting: a qualitative analysis. Med J Aust (2004) 181:36–9.[Web of Science][Medline]
- Kaplan B, Shaw N. Future directions in evaluation research: people, organizational and social issues. Methods Inf Med (2004) 43:215–31.[Web of Science][Medline]
- Evans SM, Berry JG, Smith BJ, et al. Attitudes and barriers to incident reporting: a collaborative hospital study. Qual Saf Health Care (2006) 15:39–43.
[Abstract/Free Full Text] - Leape L. Reporting of adverse events. N Engl J Med (2002) 347:1633–8.
[Free Full Text] - Force MV, Deering L, Hubbe J, et al. Effective strategies to increase reporting of medication errors in hospitals. J Nurs Adm (2006) 36:34–41.[CrossRef][Web of Science][Medline]
- Braithwaite J, Westbrook MT, Travaglia JF, et al. Are health systems changing in support of patient safety? A multi-methods evaluation of education, attitudes and practice. Int J Health Care Qual Assur (2007) 20:585–601.[CrossRef][Medline]
- Braithwaite J, Westbrook MT, Mallock NA, et al. Experiences of health professionals who conducted root cause analyses after undergoing a safety improvement programme. Qual Saf Health Care (2006) 15:393–9.
[Abstract/Free Full Text] - Runciman WB. The Australian patient safety foundation. Anaesth Intensive Care (1988) 16:114–6.[Web of Science][Medline]
- Michael S, Robinson M, Douglas P, et al. Changing an incident reporting and management paradigm: the NSW Safety Improvement Program. Asia Pac J Health Manage (2006) 1:29–38.
- Spigelman AD, Swan J. Review of the Australian Incident Monitoring System. Aust N Z J Surg (2005) 75:657–61.[CrossRef]
- Evans SM, Berry JG, Smith BJ, et al. Anonymity or transparency in reporting of medical error: a community-base survey in South Australia. Med J Aust (2004) 180:577–80.[Web of Science][Medline]
- Berkowitz EG, Ferrant ME, Goben LB, et al. Evaluation of Online Incident Reporting Systems (2005) Durham, North Carolina: Duke University School of Nursing.
- Westbrook MT, Travaglia J, Braithwaite J. Evaluation of the Incident Information Management System in New South Wales: Study No. 5—Assessment of the Satisfaction of IIMS Users with the System (2006) Kensington: Centre for Clinical Governance Research, University of NSW.
- Australian Institute of Health and Welfare. Australia's Health 2004 (2004) Canberra: Australian Institute of Health and Welfare (AIHW).
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||