International Journal for Quality in Health Care Advance Access originally published online on December 12, 2007
International Journal for Quality in Health Care 2008 20(2):136-143; doi:10.1093/intqhc/mzm064
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
A comparison of electronic records to paper records in mental health centers
Department of Psychology, Indiana University-Purdue University Indianapolis (IUPUI)
Address reprint requests to: Jack Tsai, Department of Psychology, Indiana University-Purdue University Indianapolis (IUPUI), Tel: +317 274-6760; Fax: +317 988-2719; E-mail: jatsai{at}iupui.edu
 |
Abstract |
|---|
 Top Abstract MethodsResultsDiscussionFundingAppendix AReferences |
|---|
Objective. Medication documentation is a critical aspect of quality patient care. The current study examined whether electronic medical records provide medication documentation that is more complete and faster to retrieve than traditional paper records.
Method. This study involves a comparison of archived paper medical records to recent electronic medical records through chart review. A convenient sample of three large community mental health centers in Indiana was used. Medical charts for 180 patients with schizophrenia were rated on a checklist composed of 16 items that was adapted from a national project. Documentation that existed before implementation of the electronic medical record system was compared with that after implementation at each of the three centers. The main outcome measures were completeness and retrieval time of medication documentation.
Results. Electronic medical records provided medication documentation that was more complete and faster to retrieve than paper records across all centers and within each center. On average, electronic medical records were 40% more complete and 20% faster to retrieve.
Conclusion. Electronic records have potential to improve medication management for patients in mental health centers over traditional records. However, medication documentation for patients diagnosed with schizophrenia was found to be deficient in many areas, regardless of documentation format.
Keywords: community mental health centers, documentation, electronic medical records, medication management, schizophrenia
Medical records contain treatment history and relevant experiences pertaining to the care of the individual. As medical records are continually updated, they provide written proof of the medical life of a patient over time which can aid future courses of treatments and provide decision support. Traditionally, clinical documentation has been handwritten on forms and filed into paper medical records. However, the shortcomings of paper records are well known [1]. Handwritten medical records can be illegible, incomplete and poorly organized, making it difficult to ensure quality of care [2].
The advent of computer technology has introduced enormous possibilities for electronic documentation and usage of electronic medical records. Electronic medical records are defined as medical records located on a shared computer network that are both read and written electronically on a relational database through a graphic user interface. Dudman [3] describes six levels of sophistication in electronic medical record systems, which were used to characterize electronic medical records in this study. (i) Level 1 is the most basic level supporting administrative functions of an organization through patient administration with independent departmental systems. (ii) Level 2 is Level 1 plus integration via master patient index. (iii) At Level 3, true clinical support is available with many practical uses, such as electronic clinical orders, results reporting, prescribing and multi-professional integrated care pathways. (iv) Level 4 has Level 3 plus electronic access to knowledge bases, embedded guidelines, electronic alerts and expert system support. (v) Level 5 has Level 4 plus specific clinical models and document imaging. (vi) The most advanced level is Level 6. It has telemedicine and other multi-media applications such as picture archiving and communication systems.
The potential benefits of electronic records in healthcare, such as increased communication between users, reduced paperwork, fewer medical errors and cost savings have been widely discussed [4–9]. Electronic records allow for ‘just in time’ access and have led to faster data searches and increased physician efficiency [10]. Surprisingly, the direct evidence of the advantages of electronic medical records over paper records is meager. Although there is an extensive literature on data accuracy in paper records [11], this does not appear to be the case for electronic records [12]. Despite their potential advantages and strong federal recommendations [13], the mental health field has lagged behind other healthcare specialties in utilizing electronic medical records [14].
Unfortunately, there is reason to be concerned with the quality of documentation in psychiatric records. Surveys in community settings have found (i) management of antipsychotic medications is often at variance with evidence-based recommendations, (ii) documentation of target symptoms and side effects is frequently inadequate and (iii) documentation of treatments and their outcomes is often missing from medical records [15, 16]. Thus, there is a strong need to develop methods to assess and improve medication management and documentation. One large project that has made such efforts is Medication Management Approaches in Psychiatry (MedMAP) that will now be described.
As part of the National Evidence-Based Practices Project, MedMAP was identified as an evidence-based practice for severe mental illness [17] and a ‘toolkit’ was developed to facilitate its implementation [18]. The toolkit was limited to medications for schizophrenia, with the intent ultimately to expand to other disorders. The content of this toolkit was guided by a national panel of experts [19], findings from the Texas Medication Algorithm Project [20] and the Schizophrenia Patient Outcomes Research Team recommendations [16, 21]. The MedMAP toolkit provides a set of criteria for medication documentation considered necessary for adequate patient care and its contents were used in this study.
The present study aimed to determine whether electronic medical records provide higher quality documentation than paper records, thereby improving the medication management of individuals with schizophrenia. No previous study found has examined this and it was hypothesized that electronic records would provide documentation that was more ‘complete’ and ‘faster to retrieve’ than paper records.
|
Methods |
|---|
|
TopAbstractMethodsResultsDiscussionFundingAppendix AReferences |
|---|
This study was a retrospective chart review of medication information compiled before and after the adoption of electronic medical records, comparing archived paper records to recent electronic records for the completeness and retrieval time of documentation at three mental health centers. All procedures were approved by the university's institutional review board.
Sampling
A convenience sample of three community mental health centers in Indiana was used. Each center served over 4000 patients annually, including over 200 patients diagnosed with a schizophrenia-spectrum disorder who received medication services. Two sites were located in a large city, whereas the third was located in a rural area. All three centers had been using electronic records for over 2 years.
The chart inclusion criteria were patient diagnosed with a schizophrenia-spectrum disorder, prescribed an antipsychotic and received medication services from the community mental health center for at least a year. At each center, 30 paper and 30 electronic records were randomly selected by each site's staff using a random numbers table. Patients selected for the paper record sample were excluded from the electronic record sample. A total sample of 90 electronic and 90 paper records were sampled across three mental health centers.
Assessors
Two clinical psychology graduate students served as assessors. Before data collection, they received a brief orientation at each site on its medical record system to identify the common data locations for the medication-related items. Assessors also received brief orientations to both the electronic medical and paper medical record systems at each site.
Measures
The Medication Management Approaches in Psychiatry (MedMAP) Checklist (see Appendix A) was a measure that was developed specifically for this study and was adapted from the prescriber level fidelity scale from the MedMAP toolkit [19]. The checklist contains 16 items with dichotomous ratings of ‘present or absent’ on items such as ‘year of last hospitalization’, ‘level of current medication adherence’ and ‘past psychotropic medications’. Assessors rated each medical chart for documentation of these items within 1 year of the most recent note. Medical documentation beyond the 1 year period was considered outdated. A protocol with descriptions of item rating decisions, such as when to count an item as present or absent, was used to ensure standardization.
Completeness
Completeness was defined as the total number of items that were found to be present on the MedMAP Checklist. For each chart, items were summed for a total score, which ranged from 0 (all items absent) to 16 (all items present).
Retrieval time
Retrieval time was measured by the time needed for each assessor to find and rate all items on the MedMAP Checklist. For each chart, assessors self-timed themselves using a stopwatch that was started with the first item on the MedMAP Checklist and stopped after the last item was completed. The average times for the two assessors for each chart were calculated and used as the measure of retrieval time in data analyses. All the primary analyses in the study were repeated using individual assessor times, yielding similar results, suggesting that the mean of the two assessors was a satisfactory measure.
Data collection
A brief structured interview was conducted with program directors to obtain information about their medical record systems and documentation-related changes over the years. Then, the random sample of paper records was independently rated and timed by each assessor using the MedMAP Checklist. Assessors rated charts in different order, but the same group of charts on the same day. At the end of the day, assessors discussed their individual ratings and reached consensus. After paper records were rated, the same procedure was repeated with the random sample of 30 electronic records. Paper and electronic records were assessed on separate days.
Data analysis
The data were checked for outliers; assumptions of normality and homogeneity of variance were tested. The inter-rater reliability between assessors was calculated with intra-class correlation coefficients based on a two-way mixed model using the average measure reliability [22]. To gauge whether assessors became faster with more experience in rating charts at a site, ‘learning curves’ were visually inspected and correlations were conducted between retrieval time and the order charts were assessed. Descriptive statistics for patient demographics were calculated and differences between documentation format samples were tested. The main outcomes, completeness and retrieval time were summarized as the mean total number of items per chart and mean retrieval time per chart, respectively. To test each of the main outcomes, two factor analyses of variance were used with the family wise error rate set at a two-sided alpha level of 0.05. Correlations were conducted between the two main outcomes for each documentation format separately and together to observe any relationships. At the item level, frequency analyses of completeness were conducted and independent t-tests were used to test for differences.
|
Results |
|---|
|
TopAbstractMethodsResultsDiscussionFundingAppendix AReferences |
|---|
Preliminary analysis
The mental health centers in this study all used different electronic medical record systems. Using Dudman's [3] differentiation of electronic medical records, the first author rated all three electronic medical record systems at a Level 2, meaning they had an integrated patient administration system that was indexed and had independent departments. One chart had a particularly long retrieval time and was an outlier so it was excluded from all related analyses. Distributions of completeness and retrieval time values were roughly normal for both documentation formats with skewness and kurtosis all within the range of –1 to 1. A log transformation was performed on retrieval times to meet the assumption of homogeneity of variance. There was adequate inter-rater reliability on completeness for paper records (intra-class correlation coefficient = 0.65, P < 0.001) and electronic records (ICC = 0.71, P < 0.001) when aggregated across the three study sites. On retrieval time, there was also adequate inter-rater reliability for paper records (ICC = 0.78, P < 0.001) and electronic records (ICC = 0.66, P < 0.001), when aggregated across sites. At the item level, aggregated across sites, there was high agreement between assessors on ratings (Cohen's Kappa = 0.61, P < 0.001).
Learning curve effects were examined to observe whether there were differential learning curves in retrieval time between documentation format and between sites that may have influenced or confounded the results. There appeared to be learning curves for both documentation formats with significant correlations found between retrieval time and the order that charts were assessed (P < 0.05). But no consistent pattern emerged across sites. Various supplementary analyses were conducted on the learning curves and they were found to have no material influence on the main results.
There were no significant patient demographic differences between the PMR sample and the electronic medical record sample (see Table 1). As expected, the duration of treatment documentation was significantly longer for paper records than electronic records.
|
Differences on completeness and retrieval time
Two-way analysis of variance analyses found highly significant main effects of documentation on completeness and retrieval time, showing that documentation in electronic records were significantly more complete and faster to retrieve than paper records across the three sites, as shown in Table 2. On average, electronic records had about two more items on the MedMAP Checklist documented than paper records or were 40% more complete; each electronic medical record also took 89 s less to rate or was 20% faster to retrieve than paper records. The correlation between completeness and retrieval time for paper records was r = –0.15 (n = 90) and not significant; but for electronic records, it was r = –0.31 (n = 89) and was significant (P < 0.01).
|
Table 3 shows item-level analyses of the MedMAP Checklist and which items tended to be less complete than others. To assess whether paper records and electronic records showed a similar pattern of completeness, a correlation was calculated using the 16 pairs of item percentages as data points. This yielded a correlation of 0.86, which suggests similar kinds of information were being omitted in both documentation formats. All 16 items showed improvement in completeness moving from paper records to electronic records, except for Item 15—‘Documentation of Weight'.
|
|
Discussion |
|---|
|
TopAbstractMethodsResultsDiscussionFundingAppendix AReferences |
|---|
Electronic medical records have begun to be implemented in the mental health field. This study looked at three mental health centers that have recently replaced their paper medical records with electronic records. Documentation in electronic records was found to be significantly more complete and faster to retrieve than paper records. This is a crucial finding because medication documentation chronicles the treatment life of patients and serves as support in making treatment decisions. Although the study focussed on schizophrenia, it is plausible to hypothesize that these findings would generalize to other mental illnesses as well, which also require careful and extensive medication documentation. Yet, medication practices have often been found to be deviant from evidenced-based recommendations [21].
Medical errors are a serious problem in healthcare and are often a result of documentation errors made in paper records [1]. This study did not examine documentation errors per se; it used a proxy measure to assess the adequacy of documentation. So, it looked at whether items that should be documented were documented without determining accuracy or actual medication practices. But this study is a step towards documenting a critical strategy for ameliorating errors because omission of documenting items precludes correction of errors. The findings of this study suggest that implementing electronic medical records to replace paper records may be a fruitful avenue to advancing the quality of documentation for patients. The findings also suggest that the more complete documentation is the faster it is to retrieve; results found this relationship to be particularly true for electronic records. This is another argument for the use of electronic records, in that they may increase the retrieval time of documentation by being more complete. And yet the mental health centers in this study had only begun to use electronic records in the last few years compared to decades of use with paper records.
Certain medication documentation elements tended to be more complete than others, with similar items showing up for both electronic and paper records, namely, documentation of diagnosis, medication adherence and patient education. But electronic records had dramatically more complete documentation on two items related to current medications and rationale for their prescriptions. This finding has practical significance because it is essential for prescribers to know what medications patients are currently taking and why. Electronic records shared some similar items that were missing with paper records. These were documentation related to past psychotropic medications and documentation of glucose. But all 16 items, except one about weight documentation, showed improvement in completeness with using electronic records in place of paper records.
Despite the advantage of electronic medical records over paper records, the fact remains that there is still great need for improvement in medication documentation. Despite the evidence for items on the MedMAP Checklist, less than half were found to be documented in the charts in this study. Thus we can extend the findings of previous research [15, 16] that medication practices are still often at variance with evidence-based recommendations and more attention is needed in translating research into practice.
The results of this study corroborate several findings from a pilot study of the MedMAP fidelity scale [19]. In both studies, documentation about past medication treatments, side effects and outcomes were poorly documented. The MedMAP Checklist may have potential to be used as a shortened version of the original fidelity scale. As the fidelity scale was a measure that often took several days to complete with the prescriber portion averaging 36 min per chart, the MedMAP Checklist may be a viable alternative that takes considerably less time to use (7.60 min per chart for paper records, 6.08 min per chart for electronic records). However, it is also notable that the checklist is not as comprehensive as the fidelity scale as it does not measure as many items and does not measure them on a gradient.
We speculate that the findings of this study would generalize to other community mental health centers. This study found that the documentation in electronic records were more complete and faster to retrieve than paper records in ‘each’ of the three sites and that can be seen as three replications with the same results. It is possible that even greater advantages can be found for electronic records if sites were given even more time to adapt and develop their systems.
However, the value of electronic medical records in mental health services remains largely unexplored. There are many areas for future research. As this study was mainly an omnibus test of electronic records versus paper records, the mediators responsible for the higher level of completeness and faster retrieval time of documentation in electronic records were not systematically studied. Several untested hypotheses were formed during this study and may be fertile areas for future study. In interviews with program directors, they expressed the opinion that electronic records held staff more accountable for their documentation. The use of electronic records may create an organizational culture that changes attitudes about documentation. Another possible explanation may be in the forms themselves. Some form fields on the electronic records could be programmed to require staff to complete before progressing through other fields on the form. This was not examined in the current study because fields varied even within forms, let alone between agencies. Other possible explanations may be that staff find it easier or more agreeable to type than handwrite data, electronic records can be linked to billing systems, and electronic records exist in virtual space whereas paper records are organized by staff and have to be physically sifted through.
Study limitations
As the program director interviews revealed, there have been changes in Health Insurance Portability and Accountability Act regulations in the past few years that may have altered how and what was documented during the same time electronic medical records were being implemented at each of the sites. Some organizational changes were reported by some sites as well that may have influenced the results. Site A reported that mental health and addiction services were combined for the first time when the electronic medical record system was created; Site B reported minor organizational adjustments involving splitting their office management from two geographic areas to three.
Staff characteristics could have affected the completeness and retrieval time of documentation over time. Staff characteristics may have had a significant influence on medication documentation as some prescribers are more adept and diligent with documentation than others. The clinicians at each agency may have changed over time or documentation practices may have changed. A final limitation of this study is the lack of a gold standard to determine the accuracy of documentation in medical records, i.e. documentation may be complete but not accurate. Other studies have encountered the same problem and the ideal to capture the true nature of the patient is difficult, if not impossible to achieve [23]. However, this study points to areas of medication management that may improve with using electronic records instead of paper records, so that the mental health field can benefit from the advantages of technology and translate them into a higher quality of care for patients.
|
Funding |
|---|
|
TopAbstractMethodsResultsDiscussionFundingAppendix AReferences |
|---|
This study was partially funded by an Educational Enhancement Grant from the Indiana University-Purdue University Indianapolis (IUPUI) Graduate Student Organization.
|
Appendix A |
|---|
|
TopAbstractMethodsResultsDiscussionFundingAppendix AReferences |
|---|
|
|
|
References |
|---|
|
TopAbstractMethodsResultsDiscussionFundingAppendix AReferences |
|---|
- Institute of Medicine. The Computer-Based Patient Record: An Essential Technology for Health Care (1997) Washington, DC: National Academy Press.
- Young AS, Sullivan G, Burnam MA, Brook RH. Measuring the quality of outpatient treatment for schizophrenia. Arch Gen Psychiatry (1998) 55:611–7.
[Abstract/Free Full Text] - Dudman J. Electronic patient records. Facts at their fingertips. Health Serv J (2000) 110:9–13.
- Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA (1998) 280:1339–46.
[Abstract/Free Full Text] - Makoul G, Curry RH, Tang PC. The use of electronic medical records: communication patterns in outpatient encounters. J Am Med Inform Assoc (2001) 8:610–5.
[Abstract/Free Full Text] - Wager KA, Lee FW, White AW, Ward DM, Ornstein SM. Impact of an electronic medical record system on community-based primary care practices. J Am Board Fam Pract (2000) 13:338–48.[Medline]
- Ammenwerth E, Eichstadter R, Haux R, Pohl U, Rebel S, Ziegler S. A randomized evaluation of a computer-based nursing documentation system. Methods Inf Med (2001) 40:61–8.[Web of Science][Medline]
- Johnson M, Benbow SM, Baldwin RC. An electronic patient record system and geriatric psychiatry: Considerations and implications. Aging Ment Health (1999) 3:257–63.[CrossRef][Web of Science]
- Balas EA, Austin SM, Mitchell JA, Ewigman BG, Bopp KD, Brown GD. The clinical value of computerized information services: a review of 98 randomized clinical trials. Arch Fam Med (1996) 5:271–8.
[Abstract/Free Full Text] - Kaplan B, Lundsgaarde HP. Toward an evaluation of an integrated clinical imaging system: identifying clinical benefits. Methods Inf Med (1996) 35:221–9.[Web of Science][Medline]
- Romm FJ, Putnam SM. The validity of the medical record. Med Care (1981) 19:310–5.[CrossRef][Web of Science][Medline]
- Hippisley-Cox J, Pringle M, Cater R, Wynn A, Hammersley V, Coupland C, et al. The electronic patient record in primary care—regression or progression? A cross sectional study. BMJ (2003) 326:1439–43.
[Abstract/Free Full Text] - Hogan MF. New freedom commission report: the president's new freedom commission: recommendations to transform mental health care in America. Psychiatr Serv (2003) 54:1467–74.
[Free Full Text] - Drake RE, Teague GB, Gersing K. State mental health authorities and informatics. Commun Ment Health J (2005) 41:365–70.[CrossRef]
- Cradock J, Young AS, Sullivan G. The accuracy of medical record documentation in schizophrenia. J Behav Health Serv Res (2001) 28:456–65.[CrossRef][Web of Science][Medline]
- Lehman AF, Steinwachs DM. PORT Co-Investigators. At issue: translating research into practice: the schizophrenia patient outcomes research team (PORT) treatment recommendations. Schizophr Bull (1998) 24:1–10.
[Abstract/Free Full Text] - Drake RE, Goldman HE, Leff HS, Lehman AF, Dixon L, Mueser KT, et al. Implementing evidence-based practices in routine mental health service settings. Psychiatr Serv (2001) 52:179–82.
[Abstract/Free Full Text] - Substance Abuse Mental Health Services Administration. Evidence-based practices: Shaping mental health services toward recovery. 4 November 2007, date last accessed. Substance Abuse and Mental Health Services Administration's National Mental Health Information Center http://mentalhealth.samhsa.gov/cmhs/communitysupport/toolkits/community/workbook/appendixd.asp.
- Bond GR, Miller AL, Taylor A, Tsai J, Howard PB, El-Mallakh P, et al. A scale to assess fidelity to evidence-based principles of medication management for schizophrenia. Schizophr Bull (2007) 33:469.
- Gilbert DA, Altshuler KZ, Rago WV, Shon SP, Crismon ML, Toprac MG, et al. Texas medication algorithm project: definitions, rationale, and methods to develop medication algorithms. J Clin Psychiatry (1998) 59(7):345–51.[Web of Science][Medline]
- Lehman AF, Kreyenbuhl J, Buchanan RW, Dickerson FB, Dixon LB, Goldberg R, et al. The schizophrenia patient outcomes research team (PORT): updated treatment recommendations 2003. Schizophr Bull (2004) 30(2):193–217.
[Abstract/Free Full Text] - Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull (1979) 86(2):420–8.[CrossRef][Web of Science][Medline]
- Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc (1997) 4(5):342–55.
[Abstract/Free Full Text]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  C. P. Langlotz Structured Radiology Reporting: Are We There Yet? Radiology, October 1, 2009; 253(1): 23 - 25. [Full Text] [PDF]
|
|
|
|
 |
|
 P. B. Howard, P. El-Mallakh, A. L. Miller, M. K. Rayens, G. R. Bond, K. Henderson, and A. T. Cooley Prescriber Fidelity to a Medication Management Evidence-Based Practice in the Treatment of Schizophrenia Psychiatr Serv, July 1, 2009; 60(7): 929 - 935. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||