OUP user menu

An evidence-based, multidisciplinary process for implementation of potentially better practices using a computerized medical record

Alfonso F. Pantoja, John R. Britton
DOI: http://dx.doi.org/10.1093/intqhc/mzr012 309-316 First published online: 19 April 2011


Although the Institute of Medicine of the USA has recommended elements for healthcare reform, the optimal means for incorporation of these elements into a healthcare setting remain undefined. A process for the implementation of potentially better practices is described that incorporates a computerized medical record into an evidence-based, multidisciplinary continuous quality improvement effort. Steps in the process include the following: fostering a culture change that incorporates key habits for improvement; identification of a potentially better practice; review of existing evidence and analysis of local experience; delineation of proposed outcomes and potential confounders; guideline formulation and implementation; monitoring of change effectiveness; ongoing multivariate data analyses; and policy formulation. Trainee education and family participation characterize all steps in the process. Consequently, the process incorporates all of the elements recommended by the Institute of Medicine of the USA for healthcare reform and may be adapted to any healthcare setting.

  • quality improvement
  • evidence-based medicine
  • evidence-based practice
  • computerized medical records
  • multidisciplinary team
  • potentially better practices
  • patient-centered care
  • neonatal intensive care


In 2001, the Institute of Medicine of the USA recommended five key elements for healthcare reform, stating that ‘all health professionals should be educated to deliver patient-centered care as members of an interdisciplinary team, emphasizing evidence-based practice, quality improvement approaches and informatics’ [1]. They defined quality healthcare as that which is safe, effective, efficient, equitable and patient-centered. Since the recommendations were issued, many attempts have been made to incorporate these elements individually into healthcare systems [2, 3].

The quest for improved quality of care underlies all of these recommendations, and it may be argued that efforts to implement any of the elements individually are in effect efforts to improve quality. Although evidence-based practice, informatics, and the multidisciplinary team approach may be incorporated into a healthcare system, doing so may not necessarily lead to improved quality. For example, use of multidisciplinary teams has become common in many healthcare specialties, but evidence that they improve care is limited [3, 4]. Moreover, despite good evidence for specific interventions, considerable variation in health practice continues to occur [5, 6]. Although the computerized medical record has led to improved accuracy in charting and better exchange of patient information among healthcare professionals, its implementation alone does not guarantee improvement in care [7] and methods for integrating the computerized medical record into improvement efforts have not been well delineated.

The goal of quality improvement initiatives is the determination of evidence-based best practices or potentially better practices, incorporation of these practices into the clinical decision process and continuous assessment of outcomes to facilitate further improvement. In the present paper, we describe a novel process to achieve this goal based upon healthcare cultural change that incorporates a computerized medical record into a multidisciplinary, evidence-based program of continuous quality improvement. In addition, the process fosters patient-centered care [8] and provides opportunities for professional education in quality improvement [9]. Although instituted in a newborn intensive care unit, its elements are generic and may be extended to any healthcare setting.


The process described evolved from participation in the Neonatal Intensive Care Quality Improvement Collaborative Year 2000 project of the Vermont Oxford Network [10, 11]. The Vermont Oxford Network is a voluntary collaborative of newborn intensive care units whose goal is the integration of clinical research into practice. In addition to continued participation in this collaborative, we have developed an expanded process to facilitate internal quality improvement projects in our unit. In a previous report [12], we described the results of a project using these methods. Here, we outline in detail steps in the process and emphasize aspects of its general applicability.

The newborn intensive care unit: a multidisciplinary setting

The newborn intensive care unit at Exempla St Joseph Hospital in Denver is a 33 bed level IV-B unit staffed by 7 neonatologists. Non-physician staff includes 9 neonatal nurse practitioners, 79 nurses, 13 respiratory therapists, 6 pharmacists, 3 occupational/physical therapists, 2 social workers, 5 unit secretaries, 1 equipment technician, 2 environmental services personnel and a dietitian. Members from all groups participate in the quality improvement process. In addition, family practice residents and students from a variety of disciplines participate in improvement efforts during rotations through the unit. Family-centered care is emphasized, and parents are included on all quality improvement teams.

Cultural change: fostering habits for quality improvement

Central to the implementation of our initiatives was a conscious effort to change the culture of our unit. This goal was pursued by fostering four habits recommended by Plsek [13] as key to success in quality improvement (quoted below). The first was the ‘habit of viewing clinical practice as a process’ that depends on the complex coordination of many factors and efforts of many people. Multidisciplinary teamwork has been proved to accelerate the implementation of quality improvement [14]. Consequently, our initiatives were recognized as innovations that would impact the entire system, leading to alterations in behaviors of all care providers and resulting in changing patterns of care. Recognition of this process underscored the need to develop a ‘habit for collaborative learning’, emphasizing an open and curious approach, rather than defensiveness. Since practice changes were likely to be new to all, we were learning together and could benefit mutually from each other's observations and experiences in a non-threatening environment. Frequent meetings, conference calls and internet communications during all initiatives helped to implement this habit. A corollary to collaborative learning was acknowledgement of the need to adopt a ‘habit for evidence-based practice’ among the entire group, recognizing that the evidence in favor of practice changes was supportive, but that future reports in the literature might modify our use of such practices. This also fostered development of the ‘habit for change’, a dramatic departure from an earlier attitude of ‘the way we have always done it’.

These habits were formally introduced at a series of organizationally supported 1-day retreats to which all healthcare providers in the unit, together with representatives of the hospital administration, were invited. Retreats were supervised by a trained facilitator and emphasized teamwork in healthcare and quality improvement. One of the authors (A.F.P.) conducted formal presentations on the evidence-based approach and use of the computerized literature search. Sessions also addressed plans for implementing the improvement process.

The quality improvement committee

This committee, which meets on a monthly basis, serves as the central forum for review of practices in potential need of improvement and initiation of practice change. It includes directors of physician, nurse practitioner, nursing, pharmacy, respiratory therapy, occupational/physical therapy and social service groups in the unit. A chairperson is elected by the membership. Representatives of other disciplines are included on an ad hoc basis if needed, and two parents of former newborn intensive care unit patients serve for 3-year terms. Shared decision-making is central to the function of this multidisciplinary group.

Identifying the practice in need of improvement

The next step is identification of a practice in potential need of improvement. The practice may involve any aspect of evaluation, prevention or treatment of a health problem perceived to be sub-optimal. Any individual or group in the unit may submit practice concerns to the quality improvement committee, either verbally or in writing. The quality improvement committee reviews each concern and decides whether or not it warrants further consideration.

If judged to be a potential candidate for quality improvement, the issue is summarized in a written statement of the concern, its associated health problem and the current practice related to the problem. If possible, a potentially better practice is formulated. The phrase ‘potentially better practice’ emphasizes the uncertainty of whether a practice will result in improvement at a center until it is tested locally [13]. As emphasized by Donabedian [15], the practice usually takes the form of an intervention that involves either a structural element of healthcare or a process.

Review of existing evidence

The next step is a review of existing evidence relevant to the issue. Working from the written statement from the quality improvement committee, a librarian performs a literature search for potentially relevant articles, which are screened by a physician, who assigns selected publications to various individuals for review. At a group ‘journal club’ session open to all staff, each article is critically appraised for relevance to the issue. As an accredited activity, continuing education credits are provided.

The review attempts to define the nature and incidence of the problem associated with the practice, factors that may contribute to the problem and related practices that may be effective, including the current practice and any potentially better practices. Reported complications of the practices are also elucidated. If a potentially better practice is suggested, a hypothesis for practice change is formulated. Every attempt is made to choose practices that are highly evidence-based using a standard grading system [16].

Often, no potentially better practices emerge from this review and the group concludes that current approaches to the problem are adequate. For many practices, variation may be great and the literature may provide support favoring no particular variant. This may be the conclusion of the literature review, and if so the group agrees to accept this variability.

If a potentially better practice is identified, the means of assessing its impact is determined. Primary outcomes of the practice, together with secondary outcomes that may include potential complications, are enumerated. Other factors impacting the outcomes (confounders) are also determined. Identification of confounders is important because the effects of the intervention may be obscured by the impact of such confounders if they are not identified and controlled for in subsequent analyses.

Several sessions of the journal club may be required to make these determinations. At their conclusion, a consensus statement is formulated that summarizes the state of knowledge on the topic as it relates to the potential practice change. The information is presented on a critically appraised topic (CAT) worksheet (see Appendix), which focuses thinking on both critical appraisal of the evidence and its practical significance [17]. The proposed practice change, potential outcomes and confounders are summarized. The results of this analysis are forwarded back to the quality improvement committee and to a data analyst.

The computerized medical record

Central to the work of the data analyst is use of a computerized medical record that fulfills the criteria suggested by the Institute of Medicine of the USA (Neodata, Isoprime, Lisle, IL [18]). This program is problem-oriented with associated medications and procedures and permits patient charting through structured data entry into pre-defined fields. Uniform clinical information for all patients is entered into the database.

In this system, data in the form of numeric variables (e.g. body weight, vital signs) or string identifiers (e.g. name, hospital number) is entered directly into blank fields. For information that may be summarized by categorical variables (e.g. procedures, medications), choices are selected from customizable lists in drop-down menus. Sometimes addition of a new field may be required to tabulate data previously not so recorded. Demographic information, pregnancy, labor and delivery data from the maternal record, aspects of the physical exam, problem-oriented diagnoses, laboratory values, procedures, medications and other forms of treatment such as fluid therapy and respiratory therapy are entered into their respective fields. Guidelines for data entry are made available to all providers, both at the time of their initial training and as protocols contained within the computerized medical record. An ongoing attempt is made to maintain the quality of the data entered by periodic audit of cases, determination of variance with the charting guidelines and educational interventions such as presentations and messaging to foster compliance.

The computerized medical record contains a protocol section separate from the charting function that serves as a repository for free-text clinical practice aids such as guidelines, hospital policies and supportive literature. It also serves as a site for posting of improvement guidelines and their evidence base.

From a quality improvement perspective, the ability to access data via query constitutes one of the most important aspects of the computerized medical record. Although one of the authors (J.R.B.) serves as data analyst for the projects and performs most queries, other improvement team members may do so as needed. Queries are simple to perform and their results are usually available within minutes to hours without need for assistance of a programmer.

Our patient records have been entered into this database since 1 January 2001. The vast majority of structured data fields have remained unaltered during this time period, allowing for query of particular patient attributes from the entire population. Results of the queries are exported into a spreadsheet program file that may be opened by a statistical software package. All further variable modification and statistical analyses are performed with this software.

Local need and feasibility assessment: potential outcomes and confounders

The data analyst reviews the report from the journal club sessions and queries the database to determine the local incidence of the problem, frequency of the current practices and associated outcomes and confounders. These are compared with similar data gathered from the journal club and literature review. This analysis yields an accurate determination of the magnitude of the issue and its importance in the unit.

Then, the feasibility of measuring the proposed practice change is addressed in the context of a historical control study. Patients are compared with respect to outcomes and confounders before and after the institution of the new practice [19]. For some initiatives, a controlled before and after approach, using comparison data from an affiliated hospital with an identical computerized medical record but not instituting the initiative, is used to control for secular trends [20]. In determining feasibility, the analyst addresses several questions.

The first question is: Does the computerized medical record contain data (variables) in defined fields that may facilitate accurate measurement of the proposed outcomes and confounders? An advantage of our computerized medical record is that many variables relevant to our healthcare specialty are already contained in pre-defined fields, and additions may be made to existing lists within these fields to accommodate new variations. If the computerized medical record cannot accommodate the desired data in existing fields, then new fields may be added. This facilitates future documentation of desired information but requires a change in charting practices by providers. Rarely, retrospective re-organization of information previously contained in free text is required, a laborious process.

The second question has three parts: (i) How do incidences of the problem, practices, outcomes and confounders compare with those reported in the literature? If possible, local performance is compared with that of others, the process of benchmarking [21]. (ii) Are the confounders reported in the literature associated with the proposed outcomes in our population? (iii) Are there other confounders associated with the outcomes unique to our population?

The third question is: How many patients must be studied prospectively, and how many historical controls will be required, to provide a sample size to adequately power the study? Queries of the computerized medical record, combined with a priori sample size calculations, facilitate this determination. Based upon the number of patients required and the frequency with which such patients are encountered, the duration of the project is estimated.

The analyst summarizes these issues in a report to the quality improvement committee that indicates: (i) the local incidence of the problem, together with the frequency of current practices, outcomes and confounders when compared with the literature; (ii) specific outcomes and confounders contained within, and amenable to query from the computerized medical record, in addition to any modifications of the computerized medical record required to conduct the project and (iii) expected time for project completion.

Table 1 shows examples of recent and ongoing improvement initiatives in our unit, with associated concerns, practices, outcomes and confounders.

View this table:
Table 1

Quality improvement initiatives

ConcernProblemCurrent practiceInitial data analysisPotentially better practiceOutcomesConfounders
Transfusions appear to be increasing, with increased risk of associated morbidityAnemiaTransfuse for hematocrit < 30%Transfusion frequency similar to published; increasing in recent years, especially preterm infantsTransfusion protocol with tolerance of lower hematocritPercent of infants transfused; transfusions per transfused infantDischarge hematocrit; oxygen requirement at 36 weeks; length of stay; weight gain to discharge; retinopathy of prematurityGestational age; birth weight; initial hematocrit; delayed cord clamping
Early fluid administration to preterm infants may be excessive, predisposing to PDA and chronic lung diseaseDehydration and undernutritionParenteral fluid of 100–150 ml/kg daily during first 3 daysAverage 120 ml/kg daily; frequent hypernatremia; average oxygen concentrations exceed 45% on day 3Environmental humidity coupled with fluid limitation and high calorie parenteral nutritionFluid received Days 1–3; patent ductus arteriosus; oxygen requirement on Day 3Oxygen requirement at 36 weeks; peak serum sodium and urea Days 1–3; caloric intake Day 3; urine output Days 1–3Gestational age; birth weight; surfactant administration
Delayed attainment of nipple feedings may prolong length of stayGestational immaturity of suck, swallow, breathe coordinationOffer increasing nipple feedings at progressive gestational agesOver 90% of infants remain hospitalized only because of poor nipplingFeeding protocol based upon infant cuesAge full nippling achieved; length of stayNecrotizing enterocolitis; oxygen at 36 weeks and at discharge; weight gain to dischargeGestational age, duration of mechanical ventilation; type of feeding; receipt of probiotics; antibiotic usage
Prolonged ventilation increases risk of chronic lung disease and long-term neurodevelopmental sequelaeRespiratory insufficiencyMechanical ventilationAverage duration of ventilation 5 daysBubble continuous positive airway pressure (see reference 12)Duration of mechanical ventilation; oxygen concentration at 36 weeksPatent ductus arteriosus; length of stay; intracranial hemorrhage; retinopathy of prematurityGestational age; birth weight; intrauterine growth restriction; receipt of vitamin A; gender

The potentially better practice working group: guideline formulation and implementation

Using this information, the quality improvement committee makes a final decision regarding pursuit of the practice change. Although feasibility and applicability are weighed, only potentially better practices with high levels of supporting evidence are usually considered for implementation. The potentially better practice may be revised as needed and forwarded to a working group. Practice changes judged unworthy of further pursuit are listed in a file that is reviewed semi-annually by the quality improvement committee for potential merit. A letter describing reasons for non-selection is sent to the individual or group initially submitting the concern.

Working groups, like the quality improvement committee, are multidisciplinary in composition and may be either pre-existing or ad hoc. Pre-existing groups are organized to address present and future improvement initiatives in a topical area, such as nosocomial infections, pulmonary care or nutrition. These groups may address several initiatives simultaneously. Ad hoc groups are formed only to address single specific issues, such as fluid administration or transfusion practices, and they may be dissolved when the practice change is accomplished. For both types of group, a physician appointed as chair is responsible for recruitment of other members through individual contact, e-mail requests and signs posted in the newborn intensive care unit. The exact composition of the group depends upon the nature of the practice. For most groups, parents of current or previous patients in the unit are included, and sometimes input from other parents is solicited on an ad hoc basis. Students and residents from all healthcare disciplines participate in activities of these groups during their rotations. An invitation to participate is also extended to students in graduate training programs of affiliated educational institutions, providing a potential opportunity for thesis work. Recruitment continues until ∼5–10 members are enlisted representing all potential disciplines involved with the practice.

Initial work of the group focuses upon review. The literature review, CAT and hypothesis underlying the practice change are again reviewed and discussed. Additional information may be obtained by further literature review, correspondence with other centers utilizing the new practice or site visits to witness practice implementation. Expert opinion, sometimes from outside the institution, is occasionally sought.

Details of the practice change are then formulated into procedural guidelines. These are as detailed as possible, describing in a step-by-step fashion the mechanism for achieving the practice change and including supportive references.

The next step is an educational phase. During this phase, the procedural guideline and CAT are presented to staff via posters, video presentations and in-services offered at frequent intervals. Questionnaires are administered before and after each educational effort to assess understanding of the practice change and the measures required for its implementation. Questionnaires also explore perceived problems, occasionally prompting modifications to the guidelines as deemed appropriate by the working group. Educational interventions continue until all staff have been exposed and surveyed, a process that usually requires several weeks for completion. During this time, the guideline, CAT and supporting references are incorporated into the protocol section of the computerized medical record, rendering them available for review by all providers.

Next, implementation of the practice change is initiated on a pre-determined date and in accordance with the guideline. As providers begin to utilize the new practice, records of all patients manifesting the problem related to the practice are prospectively audited by members of the potentially better practice working group to determine the extent of practice implementation and compliance with the guidelines [2]. Audits are performed in collaboration with the data analyst, who queries the database repeatedly for the problem, practice, outcomes and confounders. Feedback is provided to all staff during this phase, together with continued education, support and encouragement. Lack of compliance with the practice and/or guideline is addressed in this manner, either via group educational measures or by direct communication with individual providers deemed to be non-compliant.

During the implementation phase, use is made of rapid small-scale cycle improvement (PDSA cycles: plan-do-study-act) to achieve consensus regarding the practical aspects of the practice and to further foster the habit for change [13, 22]. In our unit, these cycles have been used to adapt the practice change to local context and refine specific aspects of implementation. They have been especially helpful in the adoption of new technologies, such as methods of mechanical ventilation. Audits, feedback, education and support continue to be provided by the working group for the remainder of the project. Ongoing projects are summarized on a large ‘Quality Bulletin Board’ in the unit.

Data analysis

Data analysis continues throughout the project. As part of this process, the selected practice, outcomes and confounders are tabulated in the computerized medical record for each case. Run charts and control charts are used to document changes in these variables over time [23]. The project continues until the target number of subjects exposed to the practice (cases) has been reached, determined by the a priori sample size analysis. An equal number of historical controls is chosen from the period prior to implementation of the potentially better practice.

Cases are compared with controls with respect to primary and secondary outcomes and confounders using appropriate bivariate analyses. Then, the independent impact of the potentially better practice upon selected outcomes is assessed by multivariate techniques, such as MANOVA or regression, which permit separation of the contributions of the various confounders [24]. In such analyses, the practice change and control populations may be operationalized as dichotomous predictor variables. At the termination of the project the results are summarized and forwarded to the quality improvement committee.

Policy formulation

Based upon the literature and the project results, a hospital policy with appended guidelines is formulated by members of the working group. This is reviewed, revised and approved by the quality improvement committee. All such policies are reviewed and revised as needed on a periodic basis, sometimes prompting further literature reviews, journal club sessions and practice change proposals.


Reported barriers to quality improvement include lack of knowledge, low self-efficacy, negative outcome expectations, inertia of previous practice and lack of time, motivation and staff support by healthcare providers [25, 26]. The process we describe provides a mechanism to overcome many of these barriers. In our unit, the single most important prerequisite for successful pursuit of quality initiatives was a culture change that incorporated principles of evidence-based practice, collaborative learning and multidisciplinary teamwork [13]. Establishment of this culture change was a direct consequence of participation in quality improvement initiatives of the Vermont Oxford Network, which were based upon these principles [10]. Enhanced knowledge, self-efficacy, optimism and a sense of teamwork and mutual support were fostered, producing an environment conducive to constructive practice change.

Central to our improvement efforts is a computerized medical record that not only facilitates documentation and communication but also provides for easy data retrieval and analysis. Characteristics of this record include the following: (i) structure: use of pre-defined fields that may accommodate structured numerical and categorical data; (ii) customizability: customizable lists for categorical fields that permit addition of new options and the ability to add new structured fields for information entry as needed; (iii) evidence-linked: ability to accommodate, within the record, evidence to support clinical practice and improvement initiatives, such as guidelines, hospital policies, supportive literature and references; (iv) query: amenability to quick and efficient query by an analyst with little or no prior computer training, preferably a healthcare provider; (v) exportability: the ability to export query results into a spreadsheet format that may be read and manipulated by a statistical program. We believe that these characteristics of a computerized medical record, which have not been previously emphasized in the literature, are crucial for the efficient facilitation of continuous quality improvement in any healthcare setting.

Coupled with our use of the computerized medical record is our method of data analysis. Although some have recommended the use of randomized controlled trials in quality improvement initiatives [27], others, including the Institute of Medicine of the USA, have advocated other designs, recognizing that such initiatives often cannot be subjected to the same rigor as clinical trials [28]. Consistent with these recommendations, our projects have taken the form of historical control studies [3, 29] to which a number of methodological refinements have been added. These include the use of a priori sample size determinations, controlled before and after designs, quantification of multiple pre-intervention and post-intervention measurements including potential confounders, comparisons of such measurements with tests of significance and multivariate analyses. We have also attempted to pursue only those potentially better practices that have the highest level of supportive evidence in our initiatives. Berwick [30] has argued that improvement endeavors are social changes, multi-component interventions that often involve interpersonal, nonlinear changes in complex social systems. Research in the social and psychological sciences, disciplines in which randomized trials are infrequently used, relies heavily upon techniques such as repeated measures design, multivariate analysis and structural equation modeling [3133], and we suggest that quality improvement projects could well benefit from similar approaches.

In the newborn intensive care unit, patient-centered care is synonymous with family-centered care [8], and inclusion of parents on multidisciplinary quality improvement teams in our unit has enhanced our focus on the patient. Similarly, participation of students and residents in our initiatives has provided for an introduction to the science of improvement, an aspect of professional education that has recently been emphasized but infrequently implemented [9, 34]. In some cases, improvement projects have been adapted for thesis work by students in advanced degree programs; one such project by a master's degree student was the focus of a recently published initiative [12].

Implementation of the process we describe required substantial investment of institutional and personal resources, and the magnitude of such investment must be weighed by any institution embarking upon a similar program. For some aspects, such as the retreat and institution of the computerized medical record, initial financial investment was substantial. Time spent for committee meetings, computerized record subscription and data analysis incurred additional ongoing financial expense. Yet with time, many activities became tightly integrated with care delivery and could be accomplished simultaneously, such that additional expense was not incurred. Non-monetary compensation, such as continuing education credit, was also provided for the journal club and other educational activities. Finally, personal gratification from individual sense of contribution proved adequate compensation for many, who frequently donated their time and energy to the collective effort.

Our process for quality improvement, initially rooted in multidisciplinary, evidence-based principles of the Vermont Oxford Network emphasizing healthcare culture change, has expanded into a formal mechanism for local implementation of practice change unique in its use of continuous data analysis from a computerized medical record. By including family members and provider trainees in the process it incorporates all of the elements for healthcare improvement recommended by the Institute of Medicine of the USA. In addition, it contains all aspects of the vision for transformation of healthcare culture of the Leape Institute [35]: transparency, integrated multidisciplinary approach, patient partnership, joy and meaning in healthcare work and medical education reform. The components of this process are generic and may easily be adapted to any healthcare setting in which individual and institutional commitment to improved quality of care exists.


Funding for all quality improvement initiatives was provided by Exempla St Joseph Hospital.

Appendix: CAT worksheet

Potentially better practice:

Applicable patient population:

Overall conclusion:

Study reference1234
Type of study
Level of evidence
Randomization (Y/N)?
Blinded (Y/N)?
Group sample size
Similar initial group characteristics (Y/N)?
Similar treatments other than practice change/intervention (Y/N)?
Equal follow-up (Y/N)?
Main outcome
 Practice change effective (Y/N)?
  Control event rate (CER)
  Experimental event rate (EER)
  Absolute risk reduction (ARR)
  Relative risk reduction (RRR)
  Number needed to treat (NNT)
Other outcomes (list)
 Favorable or neutral
Potential confounders (list)
  • aCalculations: ARR = CER − EER. RRR = (CER − EER)/CER. NNT = 1/ARR.

  • References:

  • Supportive literature:


    View Abstract