OUP user menu

Hospital Survey on Patient Safety Culture in Slovenia: a psychometric evaluation

Andrej Robida
DOI: http://dx.doi.org/10.1093/intqhc/mzt040 469-475 First published online: 4 June 2013

Abstract

Objective To study the psychometric properties of a translated version of the Agency for Healthcare Research and Quality Hospital Survey on Patient Safety Culture (HSOPSC) in the Slovenian setting.

Design A cross-sectional psychometric study including principal component and confirmatory factor analysis. The percentage of positive responses for the 12 dimensions (42 items) of patient safety culture and differences at unit and hospital-level were calculated.

Setting Three acute general hospitals.

Participants Census of clinical and non-clinical staff (n = 976).

Main Outcome Measures Model fit, internal consistency and scale score correlations.

Results Principal component analysis showed a 9-factor model with 39 items would be appropriate for a Slovene sample, but a Satorra-Bentler scaled χ2 difference test demonstrated that the 12-factor model fitted Slovene data significantly better. Internal consistency was found to be at an acceptable level. Most of the relationships between patient safety culture dimensions were strong to moderate. The relationship between all 12 dimensions and the patient safety grade was negative. The unit-level dimensions of patient safety were perceived better than the dimensions at the hospital-level.

Conclusion The original 12-factor model for the HSOPSC was a good fit for a translated version of the instrument for use in the Slovene setting.

  • patient safety culture
  • survey
  • psychometric analysis

Introduction

A safety culture is a basic necessary prerequisite for the improvement of patient safety. To assess the patient safety culture self-reported surveys on patient safety climate are often conducted [1]. They have been used to engage healthcare professionals and leadership in the improvement of patient safety and to establish strategies and programmes of patient safety in their organizations [2]. Patient safety climate is a shared perception among healthcare staff about important characteristics of patient safety. It reflects patient safety culture understood as fundamental values, behaviours and beliefs in a healthcare organization's approach to patient safety [3]. Many countries have begun to measure the perception of patient safety culture of healthcare professionals. In the countries of the European Union, there is also increasing external pressure on healthcare organizations through the Directive on Patients' Rights in Cross-Border Healthcare where quality and safety of healthcare services is a priority [4]. Although many instruments for assessing patient safety culture exist [5], a questionnaire developed by Sorra and Nieva [6] has met more psychometric criteria compared with other instruments [7] and has been used in most studies for hospital surveys in the USA, Europe and elsewhere [814]. The use of the same instrument would allow for future European comparisons of patient safety culture for patients seeking care in other European countries.

This study evaluated the psychometric properties of a Slovenian translation of the Agency for Healthcare Research and Quality supported Hospital Survey on Patient Safety Culture (HSOPSC) questionnaire [7] to assess its appropriateness for Slovenian settings.

Methods

The instrument

The HSOPSC questionnaire consists of 42 items addressing 7 unit-level, 3 hospital-level aspects of safety culture and 4 outcome variables of which 2 (overall patient safety grade and number of events reported in the last 12 months) were single-item measures. The questionnaire was translated into the Slovene language by a bilingual healthcare professional and by an expert bilingual translator. The draft translation was then pre-tested by several hospital physicians and nurses as well as by non-clinical staff for comprehension. The pre-test findings were appropriately incorporated into the final version of the questionnaire. Adaptations were made only in demographic items concerning departmental structure of the participating hospitals and difference in professional groups.

Sample

The paper-based questionnaire was distributed to all clinical and non-clinical staff of three acute general hospitals in 2010 (n = 1745) and 1048 questionnaires were returned (response rate of 60%). Of these 72 respondents who answered less than one entire section of the questionnaire, fewer than half of the items throughout the entire survey or where scoring of the items was the same were excluded, leaving 976 questionnaires available for analysis.

Statistical analysis

The average missing values were 4% (range 0.9% A3 to 8.8% RF5). Pairwise deletion was used to calculate relevant statistics. A Likert scale from 1 (strongly disagree and never) to 5 (strongly agree and always) was used to score the answers. Descriptive statistics were calculated with mean, standard deviation (SD), skewness and kurtosis.

The sample was split at random into two independent groups because principal component analysis (PCA) and confirmatory factor analysis (CFA) cannot be performed on the same data set [15]. A PCA was performed on the first half of the sample (n = 501) to study the component structure of the new translation of the original instrument into another language and different cultural setting. A cut-off value for factor loading of 0.4 was chosen to minimize item cross-loadings [16]. CFA was applied next to assess fit of the model to the data in the Slovenian sample [17]. Two separate CFAs were done on the other half of the data set (n = 475) in order to compare the model fit of the original 12-factor versus the alternative model. Two CFA fit indices were employed: the comparative fit index (CFI), with values >0.90 indicating an acceptable fit and >0.95 indicating a good fit and the root mean square error of approximation (RMSEA), with values <0.05/<0.08 considered as an indication of a good/acceptable model fit to the data [18]. To evaluate the difference in fit between the original 12-factor, 42-item model and nested 9-factor, 42-item model the Satorra-Bentler scaled χ2 difference test was calculated [19].

To assess internal consistency for the 12-factor, 42-item model, the Cronbach α score was calculated using the whole sample, construct validity was assessed by computing scale scores for each factor by calculating the mean score of the items for each particular factor for each respondent. Inter correlations between the scale scores for the 12 factors and the ‘overall patient safety grade’ were computed to determine the discriminant validity. For the ‘number of events reported’, no correlation was calculated due to the skewed nature of this item (43% of the respondents did not report any events and 33% had reported only 1–2 events; n = 976).

Strengths were defined artificially as those positively worded items which ≥75% of respondents endorse by answering ‘agree/strongly agree’, or ‘most of the time /always’ (or when ≥75% of respondents disagreed with negatively worded items). Areas with the potential for improvement were identified as items which ≤50% of respondents answered positively. The average positive percentage of each dimension and item with 95% confidence interval was calculated [20]. The z-test was used to calculate the difference between the percentage of positive responses on dimensions of patient safety culture at unit and hospital-level. For statistical analysis SPSS 17, AMOS 18 and EQS 6.1 for Windows statistical packages were used.

Results

The Kaiser-Meyer-Olkin coefficient was >0.9, indicating common variance among the items and the Bartlett test of sphericity (χ2 = 7086.2; df = 861; P < 0.001) demonstrating inter-item correlations sufficient for conducting PCA. After exclusion of three items with loadings <0.4 or cross-loading: after we make changes to improve patient safety, we evaluate their effectiveness (A13), staff are afraid to ask questions when something does not seem right (RC6), shift changes are problematic for patients in this hospital (RF11), the resulting PCA model explained 57.5% of the total variance. This model differed from the 12-factor 42-item model in that three factors from the original model (communication openness and feedback and communication about error, staffing and overall perceptions of safety and hospital handoffs and transitions and teamwork across hospital units) collapsed into a single factor.

The CFA for the original 12-factor model with 42 items (χ2 = 3892.3; df 820; P < 0.0005, n = 976) showed CFI was 0.9 and RMSEA was 0.048. The standardized factor loadings were generally large (>0.60) and ranged from 0.21 (organizational learning-continuous improvement) to 0.94 (frequency of event reporting). The alternative 9-factor model with 39 items (χ2 = 3515.0; df 703; P < 0.0005, n = 976) also fitted adequately with CFI 0.9 and RMSEA 0.049. The standardized factor loadings were also generally large (>0.60) and ranged from 0.2 (organizational learning-continuous improvement) to 0.93 (frequency of event reporting).

Next, CFA was computed for the nested 9-factor model with adding of the three items (A13, RC6 and RF11) that were dropped from the 9-factor, 39-item model. This nested model (χ2 = 3119.2; df 595; P < 0.0005, n = 976) likewise had an acceptable fit (CFI was 0.9 and RMSE was 0.050). The standard factor loading ranged from 0.21 (organizational learning–continuous improvement) to 0.92 (frequency of event reporting) with generally high loadings (>0.60). To evaluate the difference in fit between the original 12-factor, 42-item model and nested 9-factor, 42-item model the Satorra-Bentler scaled χ2 difference test was calculated [19]. The Satorra-Bentler scaled χ2 difference test showed that the 12-factor model with 42 items was a significantly better fit than the more constrained 9-factor nested model with 42 items (Satorra-Bentler scaled χ2 difference = 117.428; df 30; P < 0.001).

The Cronbach α score of the 12 scales (Table 1) indicated an acceptable level of internal consistency (>0.70) except for hospital handoffs and transitions, staffing, non-punitive response to error (0.61–0.66). Cronbach's alpha score was low for organizational learning and continuous improvement (0.36). George et al. [21] provide the following rules of thumb regarding levels of internal consistency: >0.9, excellent; >0.8, good; >0.7, acceptable; >0.6, questionable; >0.5, poor and <0.5, unacceptable. A comparison of internal consistency with some other countries is also given in Table 1.

View this table:
Table 1

Internal consistency

Scales (number of items)Cronbach α
SISCTUSNLNOUKBECH
Unit-level safety culture dimensions
 Supervisor/manager expectations (4)0.740.790.750.700.770.680.770.79
 Organizational learning and continuous improvement (3)0.360.640.760.570.510.660.590.68
 Teamwork within units (4)0.740.800.830.660.770.730.660.75
 Communication openness (3)0.740.730.720.720.680.670.650.64
 Feedback and communication about error (3)0.720.780.780.750.700.800.780.78
 Non-punitive response to error (3)0.610.770.790.690.640.650.680.70
 Staffing (4)0.650.600.630.490.650.580.570.65
Hospital-level safety culture dimensions
 Hospital management support for patient safety(3)0.820.790.830.680.790.690.720.83
 Teamwork across hospital units (4)0.740.700.800.590.650.700.660.77
 Hospital handoffs and transitions (4)0.660.740.800.690.650.770.710.72
Outcome measures
 Overall perceptions of safety (4)0.650.840.840.790.820.830.850.76
 Frequency of event reporting (3)0.880.710.740.620.760.670.580.70
  • SI, Slovenia; SCT, Scotland [14]; US, United States of America [8]; NL, Netherlands [11]; NO, Norway [10]; UK, United Kingdom [13]; BE, Belgium [9]; CH, Switzerland [12].

Intercorrelation coefficients between HSOPSC scales were significant (P < 0.01). The highest correlation was between feedback and communication about error and communication openness (0.63). Overall patient safety grade weakly to strongly and negatively correlated with all 12 factors (Table 2). The highest positive percentage score was on frequency of events reported and the lowest on staffing (Table 3). The unit-level dimensions (54% of positive responses) of patient safety are perceived better than the dimensions at hospital-level (44% of positive responses), z-score = 4.995 (P < 0.01). More than 80% of respondents graded patient safety as good to acceptable. Forty-three percent did not report any event in the 12-month period (Table 4).

View this table:
Table 2

Mean, SD and intercorrelation coefficients for 12-factor patient safety culture and patient safety grade

FactorComposite of patient safety cultureMeanSDFA1FA2FA3FA4FA5FA6FA7FA8FA9FA10FA11FA12
FA1Overall perceptions of safety3.450.73
FA2Frequency of events reported3.940.980.22
FA3Supervisor/manager expectations and actions promoting patient safety3.640.690.510.18
FA4Organizational learning–continuous improvement3.590.550.400.200.38
FA5Teamwork within units3.500.720.370.160.430.42
FA6Communication openness3.720.820.410.270.540.380.44
FA7Feedback and communication about error3.590.790.430.320.490.470.450.63
FA8Non-punitive response to error3.130.760.380.120.380.250.270.380.29
FA9Staffing2.800.780.500.080.360.220.230.230.220.39
FA10Hospital management support for patient safety3.110.870.430.190.370.370.400.320.390.340.34
FA11Teamwork across hospital units3.110.870.430.190.360.320.400.360.400.210.260.53
FA12Hospital handoffs and transitions3.400.660.380.180.260.180.240.290.250.220.290.320.55
Safety grade−0.52−0.16−0.30−0.31−0.26−0.29−0.30−0.20−0.34−0.37−0.36−0.30
  • P < 0.01; FA, factors.

View this table:
Table 3

Scores for 12 dimensions patient safety culture for a Slovene sample (n = 976)

Dimensions and items of patient safety cultureScore95% confidence interval
Overall perceptions of safety5653–59
 Patient safety is never sacrificed to get more work done (A15)6561–66
 Our procedures and systems are good at preventing errors from happening (A18)5046–53
 It is just by chance that more serious mistakes do not happen around here (RA10)5753–60
 We have patient safety problems in this unit (RA17)5551–58
Frequency of events reported6966–72
 When a mistake is made, but is caught and corrected before affecting the patient, how often is this reported? (D1)7269–75
 When a mistake is made, but has no potential to harm the patient, how often is this reported? (D2)6158–64
 When a mistake is made that could harm the patient, but does not, how often is this reported? (D3)7471–77
Supervisor/manager expectations and actions promoting patient safety6562–8
 My supervisor/manager says a good word when he/she sees a job done according to established patient safety procedures (B1)5147–54
 My supervisor/manager seriously considers staff suggestions for improving patient safety (B2)6865–71
 Whenever pressure builds up, my supervisor/manager wants us to work faster, even if it means taking shortcuts (RB3)6360–66
 My supervisor/manager overlooks patient safety problems that happen over and over (RB4)8077–82
Organizational learning—continuous improvement6461–67
 We are actively doing things to improve patient safety (A6)8078–83
 Mistakes have led to positive changes here (A9)5855–61
 After we make changes to improve patient safety, we evaluate their effectiveness (A13)5350–56
Teamwork within units6158–64
 People support one another in this unit (A1)6461–67
 When a lot of work needs to be done quickly, we work together as a team to get the work done (A3)8280–84
 In this unit, people treat each other with respect (A4)5451–57
 When one area in this unit gets really busy, others help out (A11)4441–47
Communication openness6157–64
 Staff will freely speak up if they see something that may negatively affect patient care (C2)6865–79
 Staff feel free to question the decisions or actions of those with more authority (C4)6057–63
 Staff are afraid to ask questions when something does not seem right (RC6)5350–57
Feedback and communication about error5653–57
 We are given feedback about changes put into place based on event reports (C1)3027–33
 We are informed about errors that happen in this unit (C3)6461–67
 In this unit, we discuss ways to prevent errors from happening again (C5)7471–76
Non-punitive response to error3936–42
 Staff feel like their mistakes are held against them (RA8)4138–44
 When an event is reported, it feels like the person is being written up, not the problem (RA12)4239–45
 Staff worry that mistakes they make are kept in their personnel file (RA16)3532–38
 Staffing3129–34
 We have enough staff to handle the workload (A2)2321–26
 Staff in this unit work longer hours than is best for patient care (RA5)2724–30
 We use more agency/temporary staff than is best for patient care (RA7)4240–46
 We work in ‘crisis mode’ trying to do too much, too quickly (RA14)3229–35
 Hospital management support for patient safety3936–42
 Hospital management provides a work climate that promotes patient safety (F1)4138–44
 The actions of hospital management show that patient safety is a top priority (F8)3936–42
 Hospital management seems interested in patient safety only after an adverse event happens (RF9)3734–40
 Teamwork across hospital units4140–46
 There is good cooperation among hospital units that need to work together (F4)4845–51
 Hospital units work well together to provide the best care for patients (F10)4441–47
 Hospital units do not coordinate well with each other (RF2)2018–23
 It is often unpleasant to work with staff from other hospital units (RF6)5349–56
 Hospital handoffs and transitions5350–57
 Things ‘fall between the cracks’ when transferring patients from one unit to another (RF3)5855–61
 Important patient care information is often lost during shift changes (RF5)7168–74
 Problems often occur in the exchange of information across hospital units (RF7)3835–41
 Shift changes are problematic for patients in this hospital (RF11)4643–50
  • R, reversed items.

View this table:
Table 4

Overall patient safety grade and number of events reported and submitted in the last 12 months

% of respondents (n = 976)
Overall patient safety gradea
 Excellent9
 Good47
 Acceptable38
 Poor5
 Failure1
Number of events reported and submitted in the last 12 monthsb
 None43
 1–233
 3–514
 6–107
 11–201
 ≥211
  • a9% and b13% of respondents did not answer.

Discussion

Before the adoption of HSOPSC questionnaires from different cultural settings and languages a psychometric analysis is necessary to see if the questionnaire needs modifications. Psychometric properties of the original AHRQ questionnaire have been validated in US hospital settings and exploratory factor analysis resulted in 12 dimensions [6]. In different European countries exploratory factor analysis and CFA showed that some modifications were necessary. The Dutch translated version of HSOPSC showed an 11-factor model [11], the German version disclosed an 8-factor model [12], the UK sample showed a 9-factor model [13] and the Scottish sample revealed a 10-factor model [14].

In the present study three models were explored: the original AHRQ 12-factor, 42-item model; the 9-factor, 39-item model and nested 9-factor, 42-item model to see how they fit the Slovenian data. PCA findings indicated an alternative 9-factor model which only slightly differ from the original 12-factor model. Furthermore, the Satorra-Bentler scaled χ2 difference test demonstrated that a 12-factor model significantly better fit our data. This was similar to Sarac et al. [14] where the difference between their 10-factor model and the original 12-factor model also showed the latter better fit their data. In other studies, Satorra-Bentler scaled χ2 difference test was not computed [1113].

In view of possible cross-border comparison of patient safety culture no modification to the original 12-factor model is necessary and thus it would better serve to judge patient safety in Slovene hospitals when mobility of patients in the European Union becomes a reality in 2014.

The Cronbach α score of the 12 scales indicated an acceptable level of internal consistency for most of the dimensions. Findings were comparable with other data [914], except for organizational learning and continuous improvement where α score was low. Internal consistency: scores of ≥0.6 are considered acceptable, and <0.5–unacceptable [21]. CFA revealed a low standardized regression weight for the item: mistakes have led to positive changes here. This was similar to the findings of Olsen [10]. Removing of this item may be appropriate but could diminish the opportunity for benchmarking.

The relationship among patient safety culture dimensions demonstrated strong correlation between supervisor/manager expectations/actions promoting patient safety and overall perceptions of safety, between supervisor/manager expectations/actions promoting patient safety and communication openness, between hospital management support for patient safety and teamwork across hospital units and between hospital handoffs/transitions and teamwork across hospital units. Other relationships were weak to moderate [22]. The relationship between all 12 dimensions and the patient safety grade was negative indicating that this outcome variable is in contradiction with staff perception on the 12 dimensions of patient safety culture and might have reflected the staff perception of patient safety grades more positively than the rest of patient safety culture dimensions.

None of the patient safety culture dimensions reached the artificially set value of 75% of positive answers. There were also some contradictory results, for example, between frequency of events reported (69% of positive answers) and non-punitive response to error (39% of positive answers). This difference may be explained by the understanding of the importance by the hospital staff to report errors on the one hand, and fear of the prevailing blame and shame culture in Slovenia [23] on the other hand, as unintentionally made errors may be and indeed are prosecuted in criminal courts and licenses of healthcare professionals might be temporarily withdrawn.

Several areas of HSOPSC suggested opportunities for improvement in safety culture: non-punitive response to error, staffing, hospital management support for patient safety and teamwork across hospital units. These dimensions received ≤50% of positive responses. The score on staffing was particularly low. This reflects the perception of respondents that problems for patient safety are mainly due to ‘inadequate staffing’. Although staffing is important [24], it is not the sole solution to problems of patient safety.

The study showed that survey's items and dimensions are psychometrically sound at the individual level. Sorra and Dyer [8] found that the same is true at the unit and hospital levels.

The perception of patient safety culture dimensions at unit-level was significantly better than at hospital-level. This could reflect either better cooperation and teamwork at unit-level or socially desirable answers where respondents tend to give positive self-descriptions of their unit [25, 26]. Further studies are necessary to investigate this in other cultural settings.

The study was limited to self-reported outcome variables and was not validated against objectively measured frequency of adverse events and near misses.

The present study is a foundation for raising awareness of problems of patient safety among staff. Although the study was limited to self-reported outcome variables and was not validated against objectively measured frequency of adverse events and near misses, the results can help hospitals introduce a system approach to patient safety, plan patient safety solutions, follow changes in patient safety culture over time and eliminate dangerous practices of name, blame and shame culture. To improve patient safety a reporting and learning atmosphere should be created. Support from the government must be obtained to achieve non-punitive approach to patient safety with protection of those who report errors [27].

Funding

This study was supported by the Ministry of Health of the Republic of Slovenia [N2711-10-0699].

References

View Abstract