OUP user menu

Causes of inappropriate hospital days: development and validation of a French assessment tool for rehabilitation centres

Cécile Paillé-Ricolleau, Christophe Leux, Romain Guilé, Hélène Abbey, Pierre Lombrail, Leïla Moret
DOI: http://dx.doi.org/10.1093/intqhc/mzr089 121-128 First published online: 24 January 2012


Objective To develop and validate a list of criteria to assess the causes of inappropriate hospital days for patients admitted to rehabilitation centres and sub-acute care units.

Design The tool was developed by a multidisciplinary panel of 33 experts, using a formalized consensus method. It collects both the needs of patients (distinguishing healthcare needs (9 criteria) and accommodation needs (9 criteria)) and the reasons for inappropriateness (19 criteria). Inappropriate days were identified using the tool to assess appropriateness of hospital days validated previously. Reliability was studied by measuring agreement between two independent simultaneous ratings.

Setting The validation study was performed on a randomized sample of 576 hospital days from 22 wards in France.

Main Outcome Measures Inter-rater reliability was evaluated using the κ-statistic and prevalence-adjusted and bias-adjusted kappa (PABAK).

Results For patient accommodation needs, the inter-rater reliability was estimated by a κ-value of 0.80 (95% confidence interval (95% CI) 0.66–0.92) and a PABAK of 0.80 (95% CI 0.63–0.91). There was good agreement on the reasons for inappropriateness, with κ-values from 0.30 to 0.60 and PABAK from 0.46 to 0.69. The κ-coefficient varied from 0.33 to 0.49 for the assessment of patient healthcare needs, with PABAK ranging from 0.49 to 0.72.

Conclusions The instrument is suitable and valid to assess the causes of inappropriate hospital days in rehabilitation centres and sub-acute care units. The study showed that the tool can be easily used by healthcare workers, which makes it useful for quality improvement.

  • appropriateness of hospital use
  • causes of inappropriateness
  • rehabilitation centres
  • reliability
  • quality of health care
  • utilization review


Many health systems are confronted with the growing challenge of the efficient use of available resources, in particular in hospitals. Providing evidence-based care not only maximizes the use of scarce resources but also provides optimal health outcomes. Limiting the exposure of patients to iatrogenic risks linked to inappropriate hospital stays contributes to improving the quality of care [12] and determines access to care by patients who genuinely need it [3].

The appropriateness of hospital admissions and days has been widely explored in the USA over the last 30 years, and more recently in Europe [4]. Most of the studies published were performed using the Appropriateness Evaluation Protocol (AEP) developed by Gertman and Restuccia [5].

The AEP and its adaptations into other languages [68] enable detection of inappropriate admissions and hospital days, specifically for acute care facilities. The non-availability of places in downstream care facilities is one of the main causes of inappropriateness found by several European studies [911]. In France, the recent instatement of case-mix funding in acute care facilities encourages reduction in the duration of hospital stays and has put even more pressure on follow-on and rehabilitation care (Soins de Suite et de Réadaptation—SSR). However, to our knowledge, no tool has been published to date enabling both the assessment of the appropriateness of hospital days in these facilities and a detailed analysis of causes of inappropriateness, so as to make it useful for internal quality procedures.

In this context, a working group in the QualiSanté healthcare network in western France set out to review the appropriateness of hospital days in these facilities (hereafter referred to by their French acronym, SSR). This multidisciplinary group of 33 experts developed a grid comprising 16 appropriateness criteria specific to the SSR sector. The presence of any one of these 16 criteria, objective and independent from diagnosis, is considered sufficient to conclude that a hospital day is appropriate. Using the method followed by Robain et al. [6] in the validation of the French version of the AEP (AEPf), the validation study was conducted in spring 2007 on a random sample of 406 hospital days in 17 SSR facilities. It showed that the grid possessed acceptable metric properties (reproducibility (kappa coefficient (κ): 0.71), external validity (κ: 0.42)) and was easy to use for all types of health professionals [12]. This evaluation concluded that nearly 73% of the hospital days were appropriate.

The aim of the project was thus to develop an analysis grid for the causes of inappropriateness, and then to validate the complete instrument, in terms of reproducibility. The aim was also to evaluate the appropriateness of hospital days studied and to analyse the causes of inappropriateness.


Development of the grid

The experts followed the same methodology for the development of this second grid (formalized expert consensus derived from the Nominal Group technique [1314] and the RAND Appropriateness Method (RAM) developed by RAND/UCLA [15]) (Figure 1).

Figure 1

Flowsheet for the development of the grid analysing causes of inappropriateness

First, a brainstorming session was held to list the reasons that might explain inappropriate hospital days in the SSR unit. An analysis of the literature was also performed.

Following this meeting, a first anonymous individual rating was performed independently by the experts on the 57 criteria selected, using a numerical scale ranging from 1 to 9 (1: item not relevant; 9: item highly relevant). The results were analysed using the method validated by RAND/UCLA [15] (median of responses ≥7 and absence of disagreement among experts: criterion accepted; median ≤3 and absence of disagreement among experts: criterion rejected; median between 3 and 7, or existence of disagreement: criterion uncertain, requiring further discussion).

A second rating procedure was performed, but this time the experts were aware of the results of the group overall (medians) and their own previous ratings.

The 39 accepted criteria were discussed again during the second meeting. Certain items were grouped (in case of redundancy) or re-worded, so as to make the grid as explicit as possible. Finally, 19 causes of inappropriateness of hospital days obtained a consensus among the experts and were grouped into categories (internal organizational factors (9 items), external organizational factors (7 items) and reasons relating to the patient or his/her family (3 items)).

The third meeting was devoted to the identification of care procedures and services received by the patient in an inappropriate hospital day, and the determination of the best suited place of stay. This work was conducted on the basis of the corresponding items in the AEPf grid, given the clinical state of the patient, in an ideal situation (existence and availability of facilities) and also with regard to the state of the patient (family and socio-economic environment) [16].

The protocol of the validation study and the data collection guide were finalized during the fourth and final meeting.

Finally, the analysis grid for inappropriateness of hospital days in SSR units comprised three parts: care procedure or service accounting for the presence of the patient (9 items); best suited place of stay (9 items); and the main reason explaining the presence of the patient in the facility (19 criteria). The grid and the user guide were then pre-tested.

Validation study

Study sample

All the healthcare facilities with follow-on or rehabilitation activities and covered by the QualiSanté network were invited to participate in the study (n = 22). It was conducted via a cross-sectional procedure on a given day for each participating unit, between May and August 2008. A sample of approximately 30 patients hospitalized the day before the study was included in each unit, which corresponds to the mean number of patients hospitalized in the target units. Patients entering or leaving hospital on the study day were excluded, as were those hospitalized in day-hospital departments. In all, 22 wards in 18 healthcare facilities volunteered to take part in the study.

Data collection grids

Each selected hospital day was assessed using the previously validated appropriateness grid [12]. The day was rated appropriate if at least one of the appropriateness criteria was marked as being met. If the day was rated inappropriate according to the grid, the assessors could still override this rating and rate the day as appropriate by resorting to ‘expert’ opinion [5]. The analysis grid for causes was completed solely for hospital days rated as inappropriate (i.e. no appropriateness criteria marked and no ‘override’). The socio-demographic and medico-social characteristics of patients were collected.


Two assessors from each ward involved were used for data collection (a physician (rater A) and a paramedical health professional (rater B)) and were supplied with the user guide.

They were asked to complete the appropriateness grid concurrently and independently. In case of inappropriateness of the day, the analysis grid for the causes of inappropriateness was also completed using the same methodology, in the presence of a trained outside investigator. If several causes explaining the presence of the patient were interwoven, they were instructed to retain only the primary cause.

Information could be obtained from the patient's medical file (paper or computerized) or from the nursing records.

Statistical analysis

The data analysis was performed using SPSS© 16.0 and R© 2.8.1. The results are expressed in the form of mean (±SD), median or as percentages. The statistical unit was the hospital day.

The reliability of the two grids was assessed by studying agreement between raters, quantified by Cohen's κ-coefficient [17]. The κ-coefficient was adjusted by calculating the PABAK (prevalence-adjusted and bias-adjusted kappa) [18]. The 95% confidence intervals (95% CI) were determined using a bootstrap procedure.

With regard to the analysis grid for inappropriateness, inter-rater agreement was studied by grouping the items according to categories. Four categories of items were defined for the main care procedure or service required by the patient: medical care (diagnostic and therapeutic opinion and procedures, therapeutic surveillance), paramedical care (health education, functional rehabilitation, psychological support), services of a social nature (social assistance, assistance with everyday living) and absence of care or services. The best suited place of stay was split into two categories: home versus follow-on accommodation facility.

The sensitivity and specificity of the instrument were assessed using the evaluation performed by the doctor (rater A) as a reference.


Patient socio-demographic characteristics

Overall, 576 days of hospitalization were assessed. The mean age of patients was 73.5 years (±15.4), and 62.7% were women. The median duration of stay prior to the assessment was 24 days. More than 92% of the patients were living in their own homes before hospitalization (n = 514). Among these, more than half had no professional assistance at home (n = 298) and over half were living alone (n = 259). More than three quarters had people available in their close circle (75.7%, n = 380). More than two-thirds (67.1%, n = 349) had no behavioural disorder on the day of the study, and 20.1% were in a situation of dependency.

Inappropriate hospital days

Among the days studied, 11.3% (n = 65) were assessed by only one professional, so that agreement could not be assessed.

Since traceability in the patient's files was considered inadequate in 19 instances by rater A and 15 times by rater B, these days were removed from subsequent analyses. Among the 480 hospital days for which complete and traceable assessments were available, 116 days were rated as inappropriate by rater A (24.2%) and 118 by rater B (24.6%) (Tables 1 and 2).

View this table:
Table 1

Appropriateness assessment performed by each rater

Rater A (physician)Rater B 
Inappropriate daysAppropriate days
Inappropriate days8927116
Appropriate days29335364
  • Number of observations: 480.

View this table:
Table 2

Inter-rater agreement on the grid for appropriateness of hospital days in SSR

 Proportion of appropriate hospital daysAgreementobserved(%)κ (95% CI)PABAK(95% CI)
Rater A(physician) %Rater B(paramedical) %
Appropriateness grid alone63.863.385.80.69 (0.63–0.76)0.74 (0.66–0.82)
Appropriatenessgrid + expert opinion (override)75.875.488.30.68 (0.61–0.76)0.84 (0.77–0.89)
  • Number of observations: 480.

Causes of inappropriateness

A care procedure or a service of a social nature most often accounted for the presence of the patient in the facility on the day assessed (40.8% of patients for rater A, 57.5% for rater B). Most often, the need was for help in everyday living (respectively, 30.8 and 49.6% of cases).

Absence of care delivered to the patient was noted in >20% of cases by both raters (20.0/20.4%).

A paramedical care procedure, and in particular functional rehabilitation, accounted for the presence of 23.3% of the patients for rater A, and 10.6% of the patients for rater B. Medical care procedures were noted for 15.8% of patients by rater A, and 11.5% for rater B.

The patient's own home, with or without professional assistance, was assessed as the best suited place of stay for more than half of the patients (respectively, 51.6 and 53.4%). A follow-on accommodation facility, whether medical or other, was considered the most suitable place of stay for 46.0% of patients by rater A and for 44.8% by rater B.

Causes outside the ward (in particular problems relating to follow-on facilities) explained more than half of the inappropriate days (respectively, 52.7 and 52.3%), these proportions being well above ratings for causes relating to internal organizational factors (27.7/29.4%), and also above ratings for causes linked to the patient or his/her family (19.6/18.3%) (Table 3).

View this table:
Table 3

Main reason explaining inappropriate hospital days according to raters A and B

Rater A (N = 112),n (%)Rater B (N = 109),n (%)
Internal organizational factors31 (27.7)32 (29.4)
 Admission to SSR inappropriate12 (10.7)9 (8.3)
 Late request for complementary examination or specialist opinion1 (0.9)1 (0.9)
 Lack of coordination among different internal and external protagonists in catering for the patient01 (0.9)
 Absence of the evaluation of appropriateness in the course of hospitalization4 (3.6)2 (1.8)
 Lack of information/communication with patient and his/her family01 (0.9)
 Delay in the intervention of social services in the course of hospitalization in the facility2 (1.8)2 (1.8)
 Conservative habits in the ward or facility00
 Waiting for the opinion of a physician in the facility (including analysis of results of complementary examination)5 (4.5)4 (3.7)
 Waiting for the finalization of the discharge plan for the patient7 (6.2)12 (11.0)
External organizational factors59 (52.7)57 (52.3)
 Delay in obtaining complementary examinations, specialist opinion or surgical intervention (or delay in obtaining results)4 (3.6)2 (1.8)
 Delay in or absence of the intervention of social services prior to hospitalization in SSR5 (4.5)8 (7.3)
 Returning home
  Waiting for the instatement of assistance (material, human), either the intervention of home assistance services or fitments/adjustments in the home (date fixed)10 (8.9)8 (7.3)
  Non-availability of assistance1 (0.9)3 (2.8)
 Downstream facility
  Waiting for transfer (date fixed)01 (0.9)
  Unavailability37 (33.0)33 (30.3)
  Absence of suitable downstream facility2 (1.8)2 (1.8)
Reasons linked to the patient or his/her family22 (19.6)20 (18.3)
 Non-availability of the main caregiver (including hospitalization)2 (1.8)5 (4.6)
 Social and/or family environment (isolation, conflict, financial issues, no home and so on)5 (4.5)7 (6.4)
 Discharge refused by the patient or the family whether to home or to follow-on facility15 (13.4)8 (7.3)
  • Number of observations: 112 (rater A), 109 (rater B).

Among these external causes, the non-availability of a downstream facility able to admit the patient was pinpointed in 33.0% of cases by rater A and in 30.3% by rater B.

When required accommodation and reason identified for inappropriate days were crossed, external organizational factors were highlighted for most patients rated as requiring follow-on accommodation (81.5/79.6%). Conversely, for patients considered suited to being in their own homes, the most frequent causes related to the internal organization of the facility (respectively, 43.9 and 43.1%), or to the patient or his/her family (29.8/25.9%).


With regard to the appropriateness of hospital days, the agreement observed was 85.8% with a κ-coefficient of 0.69 and PABAK at 0.74. Agreement was even better if the ‘override’ was taken into account (observed agreement 88.3%, κ 0.68 and PABAK 0.84) (Table 2).

The sensitivity of the tool was 81.0% and specificity 88.6%. When the override was taken into account, sensitivity was 76.7% and specificity 92.0%.

The reproducibility of the second part of the instrument was assessed on the 80 days rated as inappropriate by both raters and that had no missing data. For the care procedure or service accounting for the presence of the patient in the facility, agreement observed ranged from 74.4 to 85.9% according to the type of care procedure, with κ-coefficients ranging from 0.33 to 0.49, and PABAK from 0.49 to 0.72 (Table 4). For the most suitable place of stay, agreement was 90%, with κ-coefficient and PABAK at 0.80.

View this table:
Table 4

Inter-rater agreement on the grid for analysing causes of inappropriateness

Agreementobserved (%)κ (95% CI)PABAK (95% CI)
Main care procedure or service accounting for the presence of the patient in the facility
 Medical care procedure85.90.48 (0.21–0.72)0.72 (0.54–0.85)
 Paramedical care procedure84.60.42 (0.14–0.67)0.69 (0.49–0.82)
 Social care procedure74.40.49 (0.29–0.67)0.49 (0.29–0.67)
 Absence of care procedure or service78.20.33 (0.07–0.58)0.56 (0.36–0.73)
Best suited place of stay for patient
 Home/follow-on facility90.00.80 (0.66–0.92)0.80 (0.63–0.91)
Main reason accounting for inappropriate hospital day
 Internal organizational factors72.90.30 (0.04–0.55)0.46 (0.23–0.65)
 External organizational factors80.00.60 (0.40–0.78)0.60 (0.36–0.76)
 Reasons linked to patient or his/her family84.30.50 (0.20–0.74)0.69 (0.48–0.82)
  • Number of observations: 80.

The agreement observed for the identification of the main reason ranged from 72.9 to 84.3% according to the category, with κ-coefficients between 0.30 and 0.60, and PABAK between 0.46 and 0.69.


This grid was developed by a panel of experts using strict methodological procedures, which certainly contributed to its satisfactory metric properties. The study also confirms the excellent acceptability of our instrument among professionals and the feasibility of the data collection procedure.

Inappropriate hospital days

Nearly 25% of the days were rated as inappropriate. This estimation is comparable with that reported in studies using a similar methodology performed in acute care facilities in the last two decades in the USA and Europe [67, 9, 19, 20]. A recent study conducted in two Australian rehabilitation facilities showed that more than half the patient days were inappropriate, according to the InterQual utilization review criteria [21].

Moreover, the percentages of inappropriate hospitalizations, before and after expert opinion, are slightly lower than those obtained in 2007. Fifteen of the 22 participating units had taken part in this first evaluation and some of them had developed improvement plans, which could have decreased their inappropriateness rate.

The ‘override’ function was used in 12% of cases for both experts involved. Although this is below the figure noted in the first study, it is nevertheless higher than the threshold generally applied of 5% of the days studied [9]. The reasons given by the experts were similar, and mainly concerned rehabilitation care provision, implemented in the hospital as a result of associated problems not taken into account by the appropriateness criteria (extreme old age and so on). Higher proportions are also observed in several international studies [2224].

Causes of inappropriateness

Problems relating to external organizational factors were identified for more than half the inappropriate days pinpointed. These causes were mainly the non-availability of a downstream place of stay for a third of the patients, and the wait for the instatement of assistance at home for >8% of patients. Several studies have underlined the preponderant share of problems of availability of follow-on facilities, in particular subsequent to hospitalization in acute care facilities. Thus, Panis et al. [7] showed that it accounted for nearly 40% of inappropriate days. Two French studies [9, 11] noted that these difficulties explained 68 and 78%, respectively, of inappropriate days. More recently, a Danish study [10] pointed to problems linked to alternative facilities that related to both rehabilitation and provision of services at home.

Although several reasons can be interwoven for a patient, our instrument only enables to retain the primary cause. This choice had been made initially by the experts so as to restrict the data load and facilitate its use by health professionals. Indeed, the pedagogical aspects of such a study seem to us to be important, in the perspective of ongoing improvement.

It is possible that certain latent organizational causes may have been masked by patent structural causes, thus leading to the over-estimation of external causes in explaining inappropriate days. Thus, the earliness of discharge preparation during hospitalization is an issue frequently pointed to [7, 19, 25].

In addition, depending on the characteristics of each facility, certain causes connected to external organizational factors can still be accessible to the internal quality procedures (e.g. obtaining an imaging procedure).

The weight of causes related to the patient and the family should be underlined. Some of these causes are directly linked to the professional practices of communication and information of the patient and the family. Healthcare teams need to implicate patients in decisions throughout the hospital stay. The need to take the family's viewpoint into account at an early stage in relation to the patient's return home has been highlighted by Zureik et al. [26]. The literature generally notes a much smaller influence of these causes on the number of inappropriate days (from 5 to 10%) than that we found. It is probable that the implication of the family is more pronounced among patients presenting a loss of autonomy or suffering from cognitive disturbances, more numerous in sub-acute facilities. A similar observation was noted in a recent study conducted in two acute geriatric wards in France [27], and also in a Swiss study concerning elderly subjects [28].


The results of this second study confirm the good reproducibility, according to the Landis and Koch classification [29], of the first part of our instrument developed specifically for the SSR sector. The κ-value is close to that found in the first validation study (0.71) [12]. After adjustment, the results are even better, with a PABAK value of 0.84. This authorizes the conclusion that the appropriateness grid shows excellent inter-rater agreement.

Our results are also comparable with those obtained in numerous international studies for the validation of measures mentioned earlier [57, 19, 23, 30].

The intrinsic properties of the appropriateness grid are improved (sensitivity close to 80%, specificity >90%), justifying its use in screening for inappropriate hospital days by health professionals.

Moreover, inter-rater agreement for the analysis of causes of inappropriateness appears satisfactory overall in this first evaluation. This grid was developed by the same panel of experts as the one that developed the appropriateness grid. This ensured that they had a good apprehension of the concept as initially defined by Gertman and Restuccia [5].

The agreement on the best suited place of stay for the patient appears very satisfactory between doctors and paramedical staff (κ and PABAK 0.80), but more moderate for the identification of the main care procedure or service accounting for the patient's day. It also varies according to the category of main cause of inappropriateness.

After the validation of the second part of the AEPf, Lombard et al. [16] underlined the inadequate reproducibility generally observed in a detailed questionnaire. As suggested, we assessed inter-rater agreement on the causes of inappropriateness after grouping the items into large categories.

In the present study, rater B was most often from the facility's management staff (94.1%). In contrast, the majority of similar validation studies found in the literature used nurses to assess inter-rater agreement [5, 16, 24]. This could in part explain the differences in assessment between the two raters. Otherwise, certain categories (in particular the social care procedure and the internal organizational factors) need to be redefined. Nevertheless, its reproducibility is better than that registered by the Delay tool developed by Selker et al. [31] at the end of the 1980s, which comprises 9 main categories and 166 sub-categories [9]. The present grid seems to be a better compromise, as is the second part of the AEPf [16]. Indeed, the small number of items facilitates the memorization by users and improves the reliability of data collection, as underlined by Gertman [5], while retaining a sufficient degree of precision for an internal quality procedure.

Limitations of the study

A few limitations need to be underlined. The study sample was formed on a volunteer basis, which may have altered its representativeness with respect to the hospitalized population in SSR facilities. The lack of case-mix description may be a limit for benchmarking between facilities.

Regarding the grid for causes of inappropriateness, the lack of traceability noted in certain patient files and missing data reduced the size of the sample, restricting the validity of the results. Moreover, while face and content validity can be assumed to be ensured by the multi-professional panel of experts and the strict methodology, the construct validity of the grid was not assessed.


This study has enabled to develop an analysis grid for the causes of inappropriateness, complementary to the previously validated appropriateness grid. These two grids form an assessment tool for the appropriateness of hospital days in SSR, easy to use for internal quality audits. The implementation of this project within the Qualisanté network enabled facilities to discuss their results and share their improvement plans, so as to reduce their inappropriate rates [3234].

A model for comparing hospital performances, derived from the WHO-Europe PATH [35] and specific to SSR facilities, has been developing in the network. Within this project, an indicator of the appropriateness of hospital days in SSR units will enable the evaluation of the impact of improvement action undertaken.

Complementary work could also be conducted on the appropriateness of admissions to SSR facilities, where a link is established with the appropriateness of the hospital stay [19].

However, on account of the weight of the external problems that were systematically highlighted, reflection on the appropriateness of resorting to hospital facilities should be widened to the complete medical and social sphere, and involve all the agents concerned, patients and families included.


The authors would like to thank the professionals in the Qualisanté network for their logistic support (I. Mahé-Galisson, P. Rolland, X. Fouquet), and participating healthcare facilities for their implication in the project (CH de Maubreuil, HL de Clisson, HL de la Presqu'île, HL Sèvre et Loire, CHU de Nantes, Hôpital Bellier, Hôpital du Pays de Retz, Etablissement Roz Arvor, Etablissement Le Bodio, Etablissement Le Confluent, HL Loire et Sillon, Etablissement du Bois Rignoux, CH de St Nazaire, Etablissement La Tourmaline, Etablissement de Pen Bron). They would also like to thank Angela Swaine Verdier for help with the drafting of the paper.


View Abstract