International Journal for Quality in Health Care 14:403-410 (2002)
© 2002 International Society for Quality in Health Care
Paper |
A prospective study of prolonged stay in the intensive care unit: predictors and impact on resource utilization
1Department of Intensive Care
2Department of Quality Management, King Fahad National Guard Hospital, Riyadh, Kingdom of Saudi Arabia
Objective. To evaluate the predictors of prolonged Intensive Care Unit (ICU) stay and the impact on resource utilization.
Design. Prospective study.
Setting. Adult medical/surgical ICU in a tertiary-care teaching hospital.
Study participants. All admissions to the ICU (numbering 947) over a 20-month period were enrolled. Data on demographic and clinical profile, length of stay, and outcome were collected prospectively. The ICU length of stay and mechanical ventilation days were used as surrogate parameters for resource utilization. Potential predictors were analyzed for possible association with prolonged ICU stay (length of stay >14 days).
Results. Patients with prolonged ICU stay formed only 11% of patients, but utilized 45.1% of ICU days and 55.5% of mechanical ventilation days. Non-elective admissions, readmissions, respiratory or trauma-related reasons for admission, and first 24-hour evidence of infection, oliguria, coagulopathy, and the need for mechanical ventilation or vasopressor therapy had significant association with prolonged ICU stay. Mean APACHE II and SAPS II were slightly higher in patients with prolonged stay. ICU outcome was comparable to patients with 
14 days ICU stay.
Conclusions. Patients with prolonged ICU stay form a small proportion of ICU patients, yet they consume a significant share of the ICU resources. The outcome of this group of patients is comparable to that of shorter stay patients. The predictors identified in the study can be used in targeting this group to improve resource utilization and efficiency of ICU care.
Keywords: intensive care, length of stay, predictors, prospective study, resource utilization, severity of illness
Research on Intensive Care Unit (ICU) outcomes provides valuable inputs in developing more improved models for patient-centered outcomes, more robust predictions of resource use, better individual outcome prediction, and alternative outcome predictions under different treatment paradigms [1]. Among the studies examining strategies to improve quality and reduce costs by changing the way care is provided to critically ill patients, attention has recently focused on assessing patients with a prolonged length of stay (LOS) in the ICU [2–4].
Prolonged ICU stay can adversely affect the health status by increasing the risk of infection, complications, and, possibly, mortality [5]. Operationally, it impacts upon ICU bed availability and results in cancellation of elective surgeries, leading to long waiting times. The lead-time, defined as the time spent on the ward before ICU admission, is also prolonged, a factor known to affect patient outcome. These issues become all the more relevant in the context of the ongoing global shortages of critical care nursing staff [6].
There is a need for optimizing an efficient distribution and use of ICU beds [7]. Collection, analysis, and interpretation of relevant objective data on the utilization of ICU beds will help plan for reducing the length of ICU stay and facilitate covering more patients who require this care. Identification of predictors of prolonged stay may help in determining the proper timing of ICU discharge. It may also suggest the need for intermediate care units (IMCU) for ICU patients who are stabilized and no longer need critical care, but are not stable enough to be discharged to the floor.
There are many measures to assess ICU resource utilization. A simple and readily available measure is ICU LOS. Cost analysis studies have found that the ICU cost per day per patient is remarkably consistent across most diagnoses [8]. Therefore, ICU LOS has been used as a surrogate measure of resource utilization in the ICU [3, 9–11]. Another measure is the duration of mechanical ventilation, as this is one of the most common procedures in the ICU.
The study was planned as a prospective study to determine the influence of certain factors as possible predictors of prolonged stay in the ICU and their impact on resource utilization. It was expected that this would lead to some practical suggestions for better utilization of the resources.
Materials and methods
Study population and setting
The King Fahad National Guard Hospital is a 550-bed tertiary-care teaching hospital in Riyadh, Saudi Arabia. Its 12-bed adult medical/surgical ICU is staffed with full-time Board-certified intensivists, and has about 600 admissions annually. A comprehensive database was developed to record prospectively data related to admission, ICU course, and outcome. The hospital does not have an IMCU. It has a separate coronary care unit and a cardiac surgical ICU; the patients admitted to these units were not included in the study.
This study included all consecutive admissions in this ICU over a 20-month period from March 1999 to October 2000. Data analyzed included the demographics and the clinical profile of each new admission. These included: age; gender; main reasons for ICU admission (see Appendix); the nature of admission—whether elective or emergency; whether it was a first admission or a readmission; presence/absence of coma, oliguria, coagulopathy, or infection, and the need for mechanical ventilation or vasopressors in the first 24-hours; whether tracheostomy was done; and ICU outcome. ICU stay was calculated as the number of calendar days from ICU admission to discharge, and was considered prolonged if it exceeded 14 days. This cut-off point is consistent with other studies [3, 12].
Two means were used to assess severity of illness: the Acute Physiology and Chronic Health Evaluation (APACHE) II score [13] and the Simplified Acute Physiology Score (SAPS) II [14]. Both are derived from objective and readily obtained clinical data reflecting physiological abnormalities in patients’ vital signs, derangements in laboratory values, patient age, presence of comorbid conditions, and neurological impairment. Data for generating the scores were collected prospectively using the original published protocols on all patients staying for 
24 hours in the ICU [13, 14].
The ICU LOS and mechanical ventilation days were used as surrogate parameters for resource utilization as described by other investigators.
Statistical analysis
The MinitabTM statistical software release-13 (Minitab Inc.) was used for analyzing the data collected. Continuous variables were expressed as mean ± standard deviation (SD). Continuous parametric variables were analyzed by analysis of variance (ANOVA). Continuous non-parametric variables were analyzed using the Kruskal–Wallis test, and median values were reported. Categorical variables were expressed in absolute and relative frequencies, and were analyzed using the 
2 test. Univariate analysis was used to identify the significant predictors of prolonged ICU stay, and results were expressed as odds-ratio (OR) and 95% confidence interval (CI). The P value was considered significant if it was 0.05.
Results
Profile of the study group
Tables 1 and 2 detail the characteristics and clinical profile of the study group.
|
|
Age and gender. Over the 20-month period, 947 patients were admitted to the ICU. They ranged in age from 12 to 100 years, with a mean age of 49.1 ± 20.3 years. Male patients formed the majority (62.4%) of the study group.
Nature of admission. The majority of patients were admitted for non-elective (emergency medical/surgical) reasons. The main reasons for admission are listed in Table 2 (see Appendix for definitions).
Severity of illness and first 24-hour admission data. The study group had a mean APACHE II score of 19 ± 9, and a mean SAPS II score of 38 ± 20. The first 24-hour admission characteristics are shown in Table 2. During the first 24 hours, 478 (50.5%) patients required mechanical ventilation, and 211 (22.3%) patients needed vasopressor therapy.
Outcome. The ICU mortality rate was 20.4%; the majority (89.6%) of deaths occurred in the first 14 days. Mortality rate among the patients with LOS >14 days did not differ significantly from that among patients with LOS 14 days. Non-elective cases had a significantly higher mortality rate (23.14%) than elective cases (7.69%) (P < 0.001).
Utilization of resources
Figure 1 shows the distribution of patients by their ICU LOS and number of ventilation days.
|
ICU LOS. The study group utilized a total of 6392 ICU patient-days. The LOS in the ICU ranged from 1 to 132 days, with a mean of 6.7 ± 10.4 days and median of 3 days. There were 104 patients with a prolonged ICU stay of >14 days, forming 11% of the study group. Their mean ICU LOS was 27.7 ± 2 days (median 21 days) as compared with a mean LOS of 4.2 ± 3.2 days (median 3 days) in the case of those patients who had a LOS of 14 days (P < 0.001). The group with prolonged stay consumed 2880 ICU days, which forms 45.1% of the total ICU days.
Ventilation days. The study group used a total of 4604 ventilation days, which formed 72% of the ICU patient-days. The ICU LOS had a high correlation with the duration of mechanical ventilation (r2 = 0.89, P < 0.001) (Figure 2). Patients with a LOS >14 days utilized 2556 ventilation days, and accounted for the majority (55.5%) of the total ventilation days. All patients with an ICU LOS of >14 days required ventilation, the mean ventilation duration being 23.8 ± 39.1 days (median 17 days). In the case of patients with ICU LOS 14 days, only 521 (61.8%) needed ventilation; the mean number of ventilation days in their case was 9.1 ± 4.1 (median 3 days). The differences between the two groups were statistically significant (P < 0.001).
|
Tracheostomy was done in 113 (11.9%) patients. This procedure was performed in a significantly higher proportion (58.7%) of patients with prolonged stay than those with stay 14 days (6.2%) (P < 0.001).
Predictors of prolonged stay
Age. There was no significant difference in the age of the patients when grouped according to the LOS.
Gender. The gender distribution in the two groups showed no significant difference.
Type of admission. The ICU LOS of elective surgical patients and non-elective (emergency: medical and surgical) patients were 4.8 ± 9.7 and 7.1 ± 10.5 days, respectively. This difference was highly statistically significant (P < 0.001). The majority (97%) of elective surgical cases had a LOS of 14 days; only five patients (3%) stayed for >14 days compared with 99 (13%) of the non-elective cases. This difference was also statistically significant (P < 0.001). Non-elective admissions were significantly associated with prolonged stay (OR 4.7, 95% CI 1.9–11.7).
Readmissions. Patients getting readmitted to the ICU were more likely to have a prolonged LOS than patients admitted for the first time (OR 2.1, 95% CI 1.1–3.8).
Main cause of admission. Patients admitted to the ICU for respiratory and trauma-related causes were more likely to have a prolonged ICU stay (OR 2.2, 95% CI 1.4–3.6, and OR 2.1, 95% CI 1.4–3.4, respectively; P < 0.001 in both cases). With respect to non-trauma surgical cases, the reverse was the case with an OR of 0.3 (95% CI 0.1–0.5). There was a significantly lesser proportion of non-trauma surgical cases among patients with a LOS of >14 days (P < 0.001). For cardiovascular and neurological conditions, there was no significant difference between the groups.
First 24-hour data. Patients whose first 24-hour data revealed infection, oliguria, coagulopathy, or the need for mechanical ventilation or vasopressor therapy had a higher likelihood of prolonged ICU stay (Table 2).
Severity of illness. The group with a LOS of >14 days had a slightly but significantly higher mean APACHE II score (21 ± 8) than the other group (19 ± 9) (P = 0.016). The former group also had higher mean SAPS II score (43 ± 16) than the other group (37 ± 20) (P = 0.003). The relationship between these two scores of severity of illness and ICU LOS is illustrated in Figure 3. The longest ICU LOS is seen for patients with ‘intermediate’ severity of illness, whereas patients with low and high APACHE II and SAPS II scores tend to stay for shorter periods, giving the plot a bell-shaped appearance. To clarify the relationship between severity of illness and ICU LOS further, we identified the quartiles of APACHE II and SAPS II scores. We considered patients with scores below the first quartile to represent low severity of illness; those with scores higher than the third quartile represented high severity of illness, and those between the first and third quartiles represented intermediate severity of illness. Figure 4 shows the proportions of patients with low, intermediate, and high severity of illness according to ICU LOS. The prolonged ICU LOS group had fewer patients in the low severity of illness subgroups and more patients in the intermediate subgroups (P = 0.008 for APACHE II, and P < 0.001 for SAPS II).
|
|
Discussion
The main findings of our study can be summarized as follows. (1) Patients with prolonged ICU stay form a small proportion of ICU patients, yet they consume a significant proportion of ICU resources. (2) The outcome of patients with prolonged ICU stay is comparable to that of patients with shorter stays. (3) The characteristics of patients with prolonged ICU stay are quite different from those with shorter ICU stay in terms of the reasons for admission and physiological abnormalities on ICU admission. This enabled us to identify predictors of prolonged ICU stay. (4) Patients with prolonged ICU stay had slightly higher APACHE II and SAPS II scores. More importantly, the distribution of patients’ severity of illness scores is different in this group, having fewer patients with low severity of illness and more patients with moderate severity of illness. While other investigators have reported findings similar to some of ours [3], our study provides detailed assessment of prolonged stay. Furthermore, it represents the first study on prolonged stay from a non-western ICU.
ICUs consume a large proportion of hospital resources. The Medical Care Provider Analysis and Review (MEDPAR) database of the the United States Health Care Financing Administration (HCFA), now known as the Centers for Medicare and Medicaid Services (CMS), shows that ICU days formed 11.84% of the total hospital days in 1997, and that ICU charges comprised 6.15% of all hospital costs [15]. In UK, the median cost per patient day in an ICU is US$1356 (range $1242–1745) [16]. A medical/surgical ICU at a tertiary hospital in Canada reported an average ICU LOS of 4.74 days, with 7.3% of admissions staying for >14 days [3]. In our study, we had a small proportion (11%) of patients with a prolonged stay. Yet, with their 45% share of the ICU patient days and 56% of the ventilation days, they consumed a significant proportion of ICU resources. We found a good correlation between the duration of mechanical ventilation and ICU LOS. This also explains why patients who stay in an ICU for >14 days are more likely to undergo tracheostomy. Thus, this group of patients with prolonged stay should be targeted for promotion of more optimal bed utilization by decreasing their ICU LOS. A 50% reduction in ICU days among this small group with prolonged stay would result in a 25% reduction in the number of total ICU days. Such a reduction will have a major impact in the following ways.
- Logistic/operational. More patients can be admitted and cared for in the ICU with available resources; lead-time for admission in the ICU will be reduced; more elective surgeries (requiring ICU bed availability for follow-up care) can be performed.
- Qualitative. This will ensure a more optimal utilization of scarce resources for providing quality care to the ICU patients really in need of it.
- Financial. Decrease in ICU LOS will reduce the cost per patient in the ICU.
Our study identified certain predictors of prolonged ICU stay. Understanding some of these predictors can help plan strategies to improve resource utilization. Non-elective admissions, readmissions, respiratory or trauma-related causes of admission, first 24-hour evidence of infection, oliguria, coagulopathy, and the need for mechanical ventilation or vasopressor therapy were found to be significantly associated with prolonged ICU stay. In the present study, respiratory and trauma cases were more than twice as likely to have a prolonged stay. It is well established that for trauma patients, although they stay longer and utilize more resources, the outcome is encouraging with good survival and functional outcome [4, 17–18]. Thus, they constitute groups of patients worth investing in. However, beyond a certain point, they do not require ICU care and can be cared for in a less demanding area, such as the IMCU. Based on our findings, planning for an IMCU must take into consideration the above-mentioned factors. For example, in order for the IMCU to make an impact on ICU resource utilization in a hospital like ours, it must have the capacity of caring for stable mechanically ventilated patients. In addition, nurses recruited for such an area should have the proper background to care for trauma and ventilated patients.
Similarly, strategies to streamline the process of care should target those patients who are more likely to have prolonged ICU stay. As stated before, we found patients admitted with respiratory conditions to have an increased risk of prolonged ICU stay. We also found that the duration of mechanical ventilation strongly correlated with the ICU LOS. Therefore, protocols that accelerate weaning are likely to improve resource utilization. Variability is frequently seen in the practice of weaning patients from mechanical ventilation, even within the same ICU. Studies [19–21] have demonstrated significantly improved patient outcomes (e.g. shorter durations of mechanical ventilation, lower incidence of ventilator-associated pneumonia, fewer patient complications) with protocol-guided weaning of mechanical ventilation in the ICU setting. Continuous infusion of sedative drugs prolongs the duration of mechanical ventilation and LOS in the ICU. Protocols based on daily interruption of sedative-drug infusions would also decrease these [22].
The relation between severity of illness and ICU LOS is interesting (Figures 3 and 4). Patients with low severity of illness have short ICU stays, probably because they require intensive care for a short period only, then they get discharged from the ICU. In contrast, patients with very high severity of illness have a shorter ICU stay because they die early in the ICU course.
Based on our findings, our hospital adopted some of the above suggestions. An IMCU is being operationalized, and protocols for extubation and sedation are being implemented.
Strengths and limitation
Ours was a prospective study: planning in advance ensured collection of all required data; all patients admitted during the study period were included and followed up for outcome of their ICU stay. However, one possible limitation would be that the study was conducted in a single center. As a tertiary center, our ICU receives referrals of complicated medical and surgical cases with high levels of severity of illness. This might suggest the possibility of selection bias due to case-mix accounting for some of our findings. It is expected that a larger, multi-center study involving a number of ICUs with varying case-mixes or patient subgroups may bear out the findings. Yet, we believe that many of our salient observations are applicable to most ICUs.
Conclusions
ICU patients are a heterogeneous group with severe illnesses, multiple system dysfunctions, and multiple coexisting medical problems. Agreement on a minimum care data set of clinical and physiological information can improve ICU audit and help identify effective care policies. A systematic evaluation of LOS information, while being a challenging task, provides information of practical and operational significance that is essential for such strategic planning. In the present study, prospective collection and analysis of demographic and clinical data on ICU patients helped identify predictors of prolonged ICU stay, and evolve pragmatic strategies for decreasing the ICU LOS. This has profound implications for the quality of health care in the ICU, as well as its efficiency. Such systematic and well planned studies can provide valuable inputs for providing quality care for more patients through better targeted and more effective services.
Appendix:
The material in this Appendix is adapted with permission from Knaus WA, Wanger DP, Zimmerman JE et al. APACHE II: A severity of disease classification system. Crit Care Med 1985; 13: 818–829.
Reasons for ICU admission
Respiratory. Asthma/allergy, COPD exacerbation, non-cardiogenic pulmonary edema, respiratory arrest, aspiration/poisoning/toxic, respiratory infection, pulmonary embolus, pulmonary neoplasm.
Cardiovascular. Hypertension, rhythm disturbance, congestive heart failure, hemorrhage/hypovolemic shock, coronary artery disease, sepsis, post cardiac arrest, dissecting thoracic/abdominal aneurysm, cardiogenic shock.
Neurologic. Seizure disorders, intracranial/subdural/subarachanoid hemorrhage.
Other non-operative. Drug overdose, diabetic ketoacidosis, gastrointestinal (GI) bleed, other metabolic/renal, other respiratory, other neurological, other cardiovascular, other gastrointestinal
Non-operative trauma. Multiple trauma, head trauma.
Operative trauma. Post-operative multiple trauma, post-operative head trauma.
Post-operative. Post-operative chronic cardiovascular disease, post-operative peripheral vascular surgery, post-operative heart valve surgery, post-operative craniotomy for neoplasm, post-operative renal surgery for neoplasm, post-operative renal transplant, post-operative thoracotomy for neoplasm, post-operative craniotomy for intracranial/subdural/subarachnoid hemorrhage, post-operative laminectomy and spinal cord surgery, post-operative with hemorrhagic shock, post-operative GI bleed, post-operative GI neoplasm, post-operative respiratory insufficiency, post-operative GI obstruction/perforation, post-operative other neurological, post-operative other cardiovascular, post-operative other respiratory, post-operative other GI, post-operative other metabolic/renal.
Address reprint requests to Y. Arabi, Deputy Chairman, Intensive Care Department, King Fahad National Guard Hospital, MC 1425, PO Box 22490, Riyadh 11426, Saudi Arabia. E-mail: yaseenarabi{at}yahoo.com 
Accepted for publication June 12, 2002.
REFERENCES
- Rosenberg AL, Hofer TP, Hayward RA et al. Who bounces back? Physiologic and other predictors of intensive care unit readmission. Crit Care Med 2001; 29: 511–518.[Web of Science][Medline]
- Weissman C. Analyzing intensive care unit length of stay data: problems and possible solutions. Crit Care Med 1997; 25: 1594–1600.[Web of Science][Medline]
- Wong DT, Gomez M, McGuire GP, Kavanagh B. Utilization of intensive care unit days in a Canadian medical-surgical intensive care unit. Crit Care Med 1999; 27: 1319–1324.[Web of Science][Medline]
- Miller RS, Patton M, Graham RM, Hollins D. Outcomes of trauma patients who survive prolonged length of stay in the intensive care unit. J Trauma 2000; 48: 229–234.[Web of Science][Medline]
- Gilio AE, Stape A, Pereira CR, Cardoso MF, Silva CV, Troster EJ. Risk factors for nosocomial infections in a critically ill pediatric population: a 25-month prospective cohort study. Infect Control Hosp Epidemiol 2000; 21: 340–342.[Web of Science][Medline]
- Clarke T, Mackinnon E, England K, Burr G, Fowler S, Fairservice L. A review of intensive care nurse staffing practices overseas: what lessons for Australia? Aust Crit Care 1999; 12: 109–118.[Medline]
- American Thoracic Society. Fair allocation of intensive care unit resources (ATS Board of Directors Position Statement). Am J Respir Crit Care Med 1997; 156: 1282–1301.
[Free Full Text] - Noseworthy TW, Konopad E, Shustack A, Johnston R, Grace M. Cost accounting of adult intensive care: methods and human and capital inputs. Crit Care Med 1996; 24: 1168–1172.[Web of Science][Medline]
- Knaus WA, Wanger DP, Zimmerman JE et al. Variations in mortality and length of stay in intensive care units. Ann Intern Med 1993; 118: 753–761.
[Abstract/Free Full Text] - Rapoport J, Teres D, Lemeshow S et al. A method for assessing the clinical performance and cost-effectiveness of intensive care units: a multicentered inception cohort study. Crit Care Med 1994; 22: 1385–1391.[Web of Science][Medline]
- Eagle KA, Mulley AG, Skates SJ et al. Length of stay in the intensive care unit: effects of practice guidelines and feedback. J Am Med Assoc 1990; 264: 992–997.
[Abstract/Free Full Text] - Heyland DK, Konopad E, Noseworthy TW, Johnston R, Gafni A. Is it ‘worthwhile’ to continue treating patients with a prolonged stay (>14 days) in the ICU? An economic evaluation. Chest 1998; 114: 192–198.
[Abstract/Free Full Text] - Knaus WA, Draper EA, Wagner DP et al. APACHE II: A severity of disease classification system. Crit Care Med 1985; 13: 818–829.[Web of Science][Medline]
- Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. J Am Med Assoc 1993; 270: 2957–2962.
[Abstract/Free Full Text] - Health Care Financing Administration, Office of Information Services. MEDPAR database. Available at http://www.hcfa.gov/stats/medpar/medpar.htm.
- Edbrooke D, Ridley SA, Hibbert CL, Corcoran M. Variations in expenditure between adult general Intensive Care Units in the UK. Anaesthesia 2000; 56: 208–216.
- Fakhry SM, Kercher KW, Rutledge R. Survival, quality of life, and changes in critically ill surgical patients requiring prolonged ICU stays. J Trauma 1996; 41: 999–1007.[Web of Science][Medline]
- Goins WA, Reynolds HN, Nyanjom D, Dunham CM. Outcome following prolonged intensive care unit stay in multiple trauma patients. Crit Care Med 1991; 19: 339–345.[Web of Science][Medline]
- Kollef MH, Shapiro SD, Silver P et al. A randomized, controlled trial of potocol-directed versus physician-directed weaning from mechanical ventilation. Crit Care Med 1997; 25: 567–574.[Web of Science][Medline]
- Marelich GP, Murin S, Battistella F, Inciardi J, Vierra T, Roby M. Protocol weaning of mechanical ventilation in medical and surgical patients by respiratory care practitioners and nurses: effect on weaning time and incidence of ventilator-associated pneumonia. Chest 2000; 118: 459–467.
[Abstract/Free Full Text] - Wood G, MacLeod B, Moffatt S. Weaning from mechanical ventilation: physician-directed vs a respiratory-therapist-directed protocol. Respir Care 1995; 40: 219–224.[Medline]
- Kress JP, Pohlman AS, O’Connor MF, Hall JB. Daily interruption of sedative infusions in critically ill patients undergoing mechanical ventilation. N Engl J Med 2000; 342: 1471–1477.
[Abstract/Free Full Text]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  K. B. Laupland, A. W. Kirkpatrick, J. B. Kortbeek, and D. J. Zuege Long-term Mortality Outcome Associated With Prolonged Admission to the ICU. Chest, April 1, 2006; 129(4): 954 - 959. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. M. Levy, E. Abraham, M. Zilberberg, and N. R. MacIntyre A Descriptive Evaluation of Transfusion Practices in Patients Receiving Mechanical Ventilation Chest, March 1, 2005; 127(3): 928 - 935. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 T. J. Iwashyna Critical Care Use during the Course of Serious Illness Am. J. Respir. Crit. Care Med., November 1, 2004; 170(9): 981 - 986. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||