International Journal for Quality in Health Care 16:65-72 (2004)
© International Society for Quality in Health Care and Oxford University Press 2004; all rights reserved
Which data source in clinical performance assessment? A pilot study comparing self-recording with patient records and observation
Center for Quality of Care Research and
1 Department of Medical Informatics, University of Nijmegen, The Netherlands
Objective. A pilot study aimed to determine the extent to which each of three data sources could provide complete and reliable data for valid assessment of clinical performance.
Design. Clinical decisions taken in 168 consultations by seven family physicians were reviewed against guidelines for 15 clinical conditions. In total, 206 criteria were reviewed using three sources: medical records, observation in surgery, and structured self-recording by the physicians.
Setting and participants. Seven family practices in the Netherlands.
Main measures. Scores (%) of data recorded/total were obtained for each method. Kappa scores for the agreement between the three data sources were also obtained.
Results. Medical record examination provided 40%, observation 72%, and physician self-recording 95% of the data required for the review against guidelines. Nine per cent of the clinical decisions could be reviewed when using medical records, 46% when using observation data, and 69% when using data from prospective self-recording. In particular, decisions in the area of patient education and diagnostic examinations could not be reviewed validly using medical records only. Kappa agreements between the data available from the three sources as well as between the review results appeared to be 0.79.
Conclusions. Medical records alone only supply sufficient information for the review of a very limited set of clinical decisions. Physician self-recording has significantly more potential for valid review of a broad range of clinical decisions. Furthermore, self-recording seems a reliable data collection method that deserves further research.
Keywords: clinical decision making, clinical practice guidelines, data collection methods, performance assessment
Accepted for publication September 4, 2003.
There is an increasing consensus that care provision should be made transparent, both for external (accountability to society) and internal purposes (learning from mistakes and gaps in performance) [1]. In the UK, Clinical Governance demands the quality of clinical performance to be assessed and efforts have been made to produce useful indicators to this task [2,3]. Recently, there has been growing optimism about the potential of measuring quality [4]. The first statement of a National Roundtable on Health Care Quality in the United States was that quality could be defined and measured with a degree of scientific accuracy comparable to that of most measures in clinical medicine [5]. Schuster and colleagues demonstrated that it is possible to clearly describe many aspects of the quality of care [6]. Despite encouraging developments in the field of quality measurement, many methodological problems have not been solved satisfactory [7–10]. For instance, the data sources used to assess physicians’ performance are variously criticized for being biased, subjective, incomplete, or cumbersome and expensive [11–15].
Valid performance assessment requires data from everyday practice, including relevant clinical details. Commonly used ‘objective’ data sources, like medical records and databases from pharmacies or health insurance companies, are said to be incomplete and miss the link with relevant clinical details. Examples of alternative methods are observation and physician self-reports. Observation, however, is a time-consuming activity and it is not known whether it produces a complete set of data for performance assessment. Data may be missed, especially if video- or audiotape is used instead of direct observation. The physician, on the other hand, is aware of most aspects of performance during consultations, and may therefore be an ideal data source for performance assessment. However, performance assessment that makes use of the physicians themselves is usually regarded as subjective [11,16,17]. Adams and colleagues showed that physicians generally overestimate their guideline adherence [16]. This is explained by the phenomenon of social desirability, meaning that physicians feel a pressure to adhere to the guidelines and respond in a socially desired way. Another source of bias is the awareness of being observed or having your performance assessed, causing diversion from usual behaviour [18]. Observation is suggested to be a golden standard [19,20], but suffers from this bias too. Some authors indicate the subjectivity and suggestibility of implicit judgements by physicians and observers in performance assessment, and therefore stress the need for explicit review criteria [21]. Objectivity in performance assessment increases when a set of rigorously developed criteria is used. Data may be collected by physicians or observers, but performance is assessed by comparing the performance data with explicit review criteria, and not necessarily or preferably by the physicians or observers.
On behalf of a study assessing everyday performance of physicians with explicit medical review criteria, we aimed to perform apilot study to explore the use of physician self-recording as a method of data collection for performance assessment. Astructured method of self-recording was compared with medical record review and direct observation, because they are considered to be complete and objective [19,20]. We sought to see how self-recording compared with these two other methods in terms of data completeness and agreement.
 |
Materials and methods |
|---|
 Top Materials and methods ResultsDiscussionConclusionAppendixReferences |
|---|
A comparison was made between: (i) data collected for performance assessment by means of family physician self-recording; (ii) data collected routinely as part of the patient’s medical record; and (iii) data collected by an independent observer in the consulting room.
A convenience sample of family physicians practising in one district and known not to have previously participated in (quality of care) research was approached directly for participation. One physician refused because of lack of time, but seven others agreed. Their mean age was 47 years; none of them had academic interests; six were practising in cities, one in a small village; five were male; six had many years of practising experience; two had been GP trainers; and five used computerized medical records.
Clinical decisions made by these physicians in consultations were assessed over a period of 2 weeks. This assessment was carried out using explicit medical review criteria derived from national, evidence-based guidelines for family practice. In the Netherlands, the Dutch College of Family Physicians has produced these evidence-based practice guidelines since 1989 [22]. So far, just over 70 guidelines have been developed and updated, using a rigorous method that combines analysis of scientific literature, consensus discussions, and the testing ofthe guidelines by regular family physicians. However, no consideration has yet been given to how use of these guidelines could be monitored. Fifteen of these guidelines were selected in such a way that important areas of family practice were covered (see Appendix Table 1). Recommendations for appropriate clinical decision-making were extracted from these guidelines and used to define criteria for medical review or appropriateness. Criteria and recommendations were selected when they were found to be important to the purpose ofreview by a panel of five expert family physicians. Their expertise was based on their experience as final editors of the Dutch College’s guidelines. As recommendations are intended to guide clinical decisions by defining how to act under specific case conditions or existent clinical details, each criterion was constructed by relating clinical actions to relevant and specific clinical details using ‘if then’ algorithms [23,24]. In this way different cases meet different—yet more specified—criteria and thus certain aspects of case-mix are accounted for.
|
The process of selecting key recommendations from the guidelines and translating these into clinical actions and review criteria was also performed with the help of the expert panel. Thus, for the 15 clinical practice guidelines selected, 206 performance indicators were constructed for the assessment of various diagnostic and therapeutic decisions. Each indicator consists of a clinical action and none, one, or more relevant clinical details. For the condition angina pectoris, for instance, the following decision is being reviewed: did the physician have an electrocardiogram (ECG) performed (clinical action) for the right reasons? According to the national guideline, a clinical detail relevant to this decision is whether a recent infarction is suspected from the medical history. If this is the case, carrying out an ECG is the appropriate thing to do.
The indicators covered six areas, representing the field of decision making in family practice (see Appendix Table 1).
Procedures
Specific structured recording forms were carefully developed [25] to collect data on the decisions made and on the clinical details relevant to these decisions. Appendix Table 2 shows an example of such a recording form for the review of clinical decisions made in consultations with patients who suffer from angina pectoris. Both clinical details and decisions are listed, without any indication as to the appropriate performance in a specific case. Each form contained a description of the criteria for cases to be included; the physician and the observer/record reviewer had to decide independently whether the case matched the criteria.
|
Over a period of 2 weeks, performance was detailed on the recording forms by the seven family physicians as well as by two carefully instructed senior medical students serving as non-participating observers and medical record examiners. In order to limit the burden for the practices and because observation of practices is very time-consuming, the duration of recording was limited to 2 weeks.
The students were trained by scoring audiotaped consultations. A consensus procedure was applied using repetitive cycles of independent scoring, comparison of the scores, reaching consensus, and scoring again until 95% agreement was reached. This took two cycles, containing 29 consultations. The medical record was reviewed at the end of every day of observation.
The family physicians were instructed to perform and keep records as usual. In addition, they had to complete the structured recording form immediately after finishing the consultation. Only those cases were recorded where, firstly, the patient did not object to the presence of the observer and secondly, the reason for encounter matched the criteria of enclosure. Family physicians and observers were kept blind from the exact review criteria. All patients were informed prior to the consultation and an ethics committee was consulted.
Analysis
Complete data yield (100%) means that all items on the recording forms (actions or decisions and clinical details) have been scored. Completeness of data for the review of clinical decisions was therefore analysed by calculating the percentage of the number of items actually scored (meaning that they were not left open or scored with a ‘?’) and the maximum number of items on the form (Table 1). Moreover, scores (%) of decisions that could actually be reviewed were calculated. A clinical decision could be reviewed when both data on actions performed and data on relevant clinical details were available and complete for evaluation of adherence to the guideline’s recommendation (Table 2).
|
|
The reliability of data collection was analysed by calculating the Kappa agreement between data obtained from the three sources [26,27]. Kappa agreement between the review results derived from the three data sources was also calculated (Table 3).
|
Finally, all calculations were repeated for two subgroups of clinical conditions: diabetes mellitus and hypertension versus the remaining conditions. The aim of this procedure was to explore to what extent completeness of recording was condition-specific [28]. Diabetes mellitis and hypertension were chosen because they accounted for up to 58% of conditions encountered.
|
Results |
|---|
|
TopMaterials and methodsResultsDiscussionConclusionAppendixReferences |
|---|
Fifty-one out of 1015 patients objected to participation. In 168 cases of the remaining 964 patients, the physicians and observers agreed that the inclusion criteria were met (the observer identified four extra cases). This resulted in the recording of 1631 clinical actions or decisions and 1022 relevant clinical details.
Completeness of raw data
The availability of data varied with the source used (Table 1). Whereas medical records provided 40% of the required data for decision review, 72% was obtained by the observer and 95% by self-recording. Data on giving information and advice to patients was lacking from the medical records, as are data on diagnostic examinations such as history taking and physical or instrumental examinations. Information on the prescribing of medicine and on referral could be extracted from the medical records in 96% and 91% of cases, respectively. Twenty-eight per cent of the data on diagnostic decisions could be derived from the medical record, whereas the observer picked up twice as much.
With self-recording or observation, clinical details appeared to be somewhat less available than clinical actions or decisions. For the medical records, the reverse was the case. Patient-related details were not readily captured by the observer or from the medical record.
Completeness of decision review data
Using medical records, only 9% of decisions could be reviewed. In decision review based on observation and self-recording data, this was 46% and 69%, respectively (Table 2).
About one quarter of the decisions on referral and prescription could be reviewed validly with the use of data from the medical records. Less than 10% of the remaining types of decisions could be reviewed. The latter could be reviewed in about half the cases where data from observation were used. Sixty to eighty per cent of all types of clinical decisions could be reviewed using the data obtained by physicians’ self-recording (Table 2).
Scores for completeness of data and completeness of review showed no significant differences between cases of hypertension or diabetes (n = 97) and cases of the remaining conditions (n = 71).
Agreement concerning data collected and decision review results
Agreement between data obtained from self-recording and data obtained by observation could be computed for 1409 of 1631 clinical decisions, and for 445 of 1022 clinical details (Table 3). Agreement appeared to be good (Kappa scores were 0.67 and 0.79, respectively). For 222 decisions and 567 clinical details, agreement could not be computed because data from either observation or (less frequently) self-recording were lacking.
Calculations of Kappa scores between data from self-recording and data from medical records showed the same tendency: agreement was almost perfect (0.81 and 0.86), but this concerned only one-third of the data. The overall Kappa score of all data pairs involved for comparison of the three sources amounted to 0.79.
The calculation of Kappa scores between results of decision review based on data from self-recording compared with review results based on medical records or on observation and between medical records and observation were 0.75, 0.93, and 0.87, respectively (Table 3). These calculations could only be made from a small number of data pairs (620, 125, and 101, respectively), because only for these cases was all the necessary information available. The overall Kappa score was also 0.79.
|
Discussion |
|---|
|
TopMaterials and methodsResultsDiscussionConclusionAppendixReferences |
|---|
Principal finding
Physician self-recording clearly provides a more complete set of data elements than observation and medical record examination. Moreover, agreement between results from review using the three methods of data collection appears to be good. Therefore self-recording may be a valid and reliable source for performance review.
Potential weaknesses of the study
Our study design had some limitations. Firstly we used a small and select sample of family physicians, characterized by ready acceptance of observation of their practice. Theoretically, physicians that are at ease while being observed may be less afraid to show their performance and likewise are less inclined to fake a self recording-based performance review compared with physicians who object to observation. Secondly, observation bias was hard to avoid. The participating physicians understood that not only their clinical decisions were the focus of interest, but also their self-recordings and medical records. Therefore we cannot be sure whether physicians generally record similarly to the physicians in our study. However, in succession to the present study, self-recording without observation was performed by 200 family physicians. This allowed review of the same percentage of decisions as in the current pilot. Thirdly, for practical reasons the observers also performed the medical record review, whereas it would have been more prudent to have this review performed by others blinded to the observation. Blinding might have resulted in a less positive Kappa score between both data sources. In future, research-anonimized hard copies of the medical record could be taken from the practice and reviewed by an independent reviewer.
Although criteria and recommendations only covered a part of clinical practice in family medicine, it is unlikely that recording of decisions and clinical detail will differ much for conditions not studied in this pilot.
Since this was a first exploration of the method of self- recording, we acknowledge that the study needs to be replicated in a larger group of physicians. It needs no explaining, however, that observation of clinical practice on a large scale is a highly time consuming and very expensive activity. Additional methods may be considered in future studies, such as patient questionnaires in which patients report on the case received, or the observer questioning the physician about the justification for clinical decisions. We considered the last option, but did not include it in the comparison because we were afraid that it would influence performance.
Pros and cons of self-recording
The most significant advantage of self-recording is the increased data yield, leading to more complete clinical decision review. Compared with medical records, self-recording provides more information on more areas of clinical performance or decision making. Unlike medical records, self-recording offers potential to the assessment of diagnostic performance (58–74% of cases) and patient education (82–83% of cases). This contrasts sharply to the shortcomings of medical records, whether computerized or not, in performance review, as demonstrated previously by others [29,30,31].
Rethans and colleagues’ findings were very similar to ours [11]. In their study, 32% of all actions taken by Dutch general physicians were in the medical record, and again diagnostics and patient education scored lowest. Medication and therapy was recorded in 68% of cases, history in 29%, physical examination in 31%, and guidance and advice in 22%. Wilson and McDonald found that no more than 
30% of advice given could be retrieved from the patients’ notes [32].
In using self-recording, the validity of performance assessment is not only made stronger by increased data yield of clinical actions or decisions, but also by the gain of detailed clinical information. As many decisions depend on the clinical details of a case, they can only be reviewed validly when information on these details is available. Routinely collected data and even medical record data are often too global and unspecific to be related to the complex setting of doctor–patient contacts and decision making. Additionally collected relevant clinical details may be helpful to adjust for differences in case-mix and clinical severity, thus increasing the validity of performance scores or physicians’ profiles. Many of the concerns expressed with respect to physician profiling and public disclosure have to do with this aspect of validity [8,33].
It is not surprising that self-recording leads to increased data yield. Firstly, a preformatted form functioning like a data checklist guides recording, and secondly the family physician is considered to be an expert on the clinical data of the case he or she has just encountered. It is therefore more or less surprising that even with the use of self-recording, 30% of clinical decisions cannot be reviewed! This is probably due to the fact that most decisions depend on clinical details, some of which (6–13%) cannot be reproduced by the physician (Table 1). In other words, the physician does not recognize some of the clinical details that have essential positions in guideline recommendations.
There are some potential disadvantages of the method of self-recording. Firstly, the self-recording physician is aware of the fact that his or her performance is being studied, which is likely to influence both practice and recording, and probably to improve both [33–35]. It is, however, to be expected that this effect wears off with repeated or prolonged self-recording, in the same way as was suggested for video observation [36]. The effect needs to be studied and quantified. Probably only a selection of decisions that can be reviewed with the use of medical records would be most suitable to track the effects of self-recording (and observation) on the level of performance. A second disadvantage of self-recording is that the form itself may direct the physician’s behaviour by giving clues on how to adhere best to the current guideline. It is generally suggested that the use of close-ended scoring forms biases performance upwards. Moreover there could be learning effects. There is also the possibility of the physician gaming the score or being selective when deciding which cases to enclose [33,37]. Measures that can be taken to counteract these phenomena are: (i) to instruct physicians not to use the recording form until after the consultation; (ii) not to record more than two cases of the same clinical condition in order to reduce the learning effect; (iii) to record consequent cases, making sure no cases are inappropriately excluded from review; (iv) to construct recording forms that do not reveal the medical review criteria or the guidelines that underlie them, i.e. forms contain the lowest number of clues possible on how to adhere to the guideline; (v) to emphasize the confidentiality of the review results; and (vi) to reassure the physician that the review results serve no selective goals, only educational and scientific ones [38].
Improved medical recording for quality of performance ass essment
Data on the quality of clinical performance need to be valid and detailed. Physicians have first rank knowledge of what is decided and performed in relation to (details of) the presented conditions during consultations. They routinely keep records of their consultation as well. Unfortunately, medical records are not kept for the purpose of clinical decision review. Thus, if medical records are to be used in quality assessment, they need to be adapted in such a way that relevant information is recorded. The main barriers will be: (i) the physician and the quality controller have different perspectives of what the relevant clinical data are, i.e. the physician may be unaware of the data essential for record review; and (ii) the physician is not prepared to invest extra time in the recording of data that do not serve immediate and clear clinical or administrative goals.
The use of computerized record-keeping and standardized work sheets may help to obtain data for quality of care [12], but electronic medical recording in itself is no guarantee of substantial improvements in the quality of recording [39]. More directive interventions are probably required to prompt the physician to record relevant clinical actions and details. Menu-guided recording with pop-up screens triggered by diagnose or complaint codes may be useful and represent an alternative to the recording forms used in this study. As a reward for the recording of essential data, the physician could be supplied with specific recommendations aiding his or her decision making.
|
Conclusion |
|---|
|
TopMaterials and methodsResultsDiscussionConclusionAppendixReferences |
|---|
This pilot study suggests that structured self-recording allows fuller performance assessment than medical record review and observation. Future research, however, needs to replicate the study in larger numbers of physicians. Computerized self-recording instruments need to be developed and tested.
|
Appendix |
|---|
|
TopMaterials and methodsResultsDiscussionConclusionAppendixReferences |
|---|
The authors thank Chris Sluyter and Hans van Hoorn who contributed to the data collection in family practices. This study was supported by grant PAO/MPVV/955099 from the Dutch Ministry of Health and was achieved in close collaboration with the Dutch College of General Physicians.
|
References |
|---|
|
TopMaterials and methodsResultsDiscussionConclusionAppendixReferences |
|---|
- Solberg LI, Mosser G, McDonald S. The three faces of performance measurement: improvement, accountablility, and research. J Qual Improv 1997; 23: 135–147.
- NHS Executive. The New NHS Modern and Dependable: a National Framework for Assessing Performance. London: NHS Executive, 1998.
- McColl A, Roderick P, Smith H et al. Clinical governance in primary care groups: the feasibility of deriving evidence-based performance indicators. Qual Health Care 2000; 9: 90–97.
[Abstract/Free Full Text] - Brook RH, McGlynn EA, Cleary PD. Quality of health care, part 2: measuring quality of care. N Engl J Med 1996; 335: 966–970.
[Free Full Text] - Chassin MR, Galvin RW. The urgent need to improve health care quality. Institute of Medicine National Roundtable on Health Care Quality. J Am Med Assoc 1998; 280: 1000–1005.
[Abstract/Free Full Text] - Schuster MA, McGlynn EA, Brook RH. How good is the quality of health care in the United States? Milbank Q 1998; 76: 509, 517–563.[CrossRef][Web of Science][Medline]
- McKee M, Sheldon T. Measuring performance in the NHS. Good that it’s moved beyond money and activity but problems remain. Br Med J 1998; 316: 322.
[Free Full Text] - Hofer TP, Hayward RA, Greenfield S, Wagner EH, Kaplan SH, Manning WG. The unreliability of individual physician ‘report cards’ for assessing the costs and quality of care of a chronic disease. J Am Med Assoc 1999; 281: 2098–2105.
[Abstract/Free Full Text] - Giuffrida A, Gravelle H, Roland M. Measuring quality of care with routine data: avoiding confusion between performance indicators and health outcomes. Br Med J 1999; 319: 94–98.
[Abstract/Free Full Text] - Davies HTO, Lampel J. Trust in performance indicators? Qual Health Care 1998; 7: 159–162.[Abstract]
- Rethans JJ, Westin S, Hays RB. Methods for quality assessment in general practice. Fam Pract 1996; 13: 468–476.
[Abstract/Free Full Text] - Health Services Research Group. Quality of care: 1. What is quality and how can it be measured? Can Med Assoc J 1992; 146: 2153–2158.[Medline]
- Gerbert B, Hargreaves WA. Measuring physician behaviour. Med Care 1986; 24: 838–847.[Web of Science][Medline]
- McDonald CJ. Quality measures and electronic medical systems. J Am Med Assoc 1999; 282: 1181–1182.
[Free Full Text] - Iezzoni LI. Assessing quality using administrative data. Ann Intern Med 1997; 127: 666–674.[Web of Science][Medline]
- Adams AS, Soumerai SB, Lomas J, Ross-Degnan D. Evidence of self-report bias in assessing adherence to guidelines. Int J Qual Health Care 1999; 11: 187–192.
[Abstract/Free Full Text] - Saturno PJ, Palmer RH, Gascon JJ. Physician attitudes, self-estimated performance and actual compliance with locally peer-defined quality evaluation criteria. Int J Qual Health Care 1999; 11: 487–496.
[Abstract/Free Full Text] - Adair JG. The Hawthorne effect: a reconsideration of the methodological artifact. J Appl Psych 1984; 96: 334–345.[CrossRef]
- Donabedian A. The information. Medical record completeness. In Donabedian A (ed.) The Methods and Findings of Quality Assessment and Monitoring. An Illustrated Analysis. Ann Arbour, Michigan: Health Administration Press, 1985, pp. 350–375.
- Hermida J, Nicholas DD, Blumenfeld SN. Comparative validity of three methods for assessment of the quality of primary health care. Int J Qual Health Care 1999; 11: 429–433.
[Abstract/Free Full Text] - Lawthers AG, Palmer RH, Edwards JE, Fowles J, Garnick DW, Weiner JP. Developing and evaluating performance measures for ambulatory care quality: a preliminary report of the DEMPAQ project. Jt Comm J Qual Improv 1993; 19: 552–565.[Medline]
- Grol RPTM, Thomas S, Roberts S. Development and implementation of guidelines for family practice: lessons from the Netherlands. J Fam Pract 1995; 40: 435–439.[Web of Science][Medline]
- Field MJ, Lohr KN (eds) Institute of Medicine Guidelines for Clinical Practice. From Development to Use. Washington, DC: National Academy Press, 1992.
- Palmer RH, Banks NJ (eds) Using Clinical Practice Guidelines to Evaluate Quality of Care, Part 2. Rockville, MD: US Department of Health and Human Services, Public Health Service, Agency for Health Care Policy and Research, 1995.
- Lydeard S. The questionnaire as a research tool. Fam Pract 1991; 8: 84–91.
[Abstract/Free Full Text] - Cohen JA. A coefficient of agreement for nominal scales. Educ Psychol Measurement 1960; 20: 37–46.[CrossRef]
- Cohen JA. Weighted Kappa. Nominal scale agreement with provision for scale disagreement or partial credit. Psychol Bull 1968; 70: 213–220.[CrossRef][Medline]
- Rethans JJ, Martin E, Metsemakers J. To what extent do clinical notes by general physicians reflect actual performance? A study using simulated patients. Br J Gen Pract 1994; 44: 153–156.[Web of Science][Medline]
- Pringle M, Ward P, Chilvers C. Assessment of the completeness and accuracy of computer medical records in four practices committed to recording data on computer. Br J Gen Pract 1995; 45: 537–541.[Web of Science][Medline]
- Neal RD, Heywood PL, Morley S. Real world data—retrieval and validation of consultation data from four general practices. Fam Pract 1996; 5: 455–461.
- Harriss C, Pringle M. Do general practice computer systems assist in medical audit? Fam Pract 1994; 11: 51–56.
[Abstract/Free Full Text] - Wilson AE, McDonald P. Comparison of patient questionnaire, medical record, and audio tape in assessment of health promotion in general practice consultations. Br Med J 1994; 309: 1483–1485.
[Abstract/Free Full Text] - Bindman AB. Can physician profiles be trusted? J Am Med Assoc 1999; 281: 2141–2143.
- Casalino LP. The unintended consequences of measuring quality on the quality of medical care. N Engl J Med 1999; 341: 1147–1150.
[Free Full Text] - Bobrow RS. The unintended consequences of measuring qualityon the quality of medical care [letter]. N Engl J Med 2000; 342: 519.
- Pringle M, Stewart-Evans C. Does awareness of being video recorded affect doctors’ consultation behaviour? Br J Gen Pract 1990; 40: 455–458.[Web of Science][Medline]
- Eddy DM. Performance measurements: problems and solutions. Health Aff (Milwood) 1998; 17: 7–25.[Abstract]
- Wilkinson EK, McColl A, Exworthy M et al. Reactions to the use of evidence-based performance indicators in primary care: a qualitative study. Qual Health Care 2000; 9: 166–174.
[Abstract/Free Full Text] - Del Mar CB, Lowe JB, Adkins P, Arnold E, Baade P. Improving general physician clinical records with a quality assurance minimal intervention. Br J Gen Pract 1998; 48: 1307–1311.[Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  A. Klassen, A. Miller, N. Anderson, J. Shen, V. Schiariti, and M. O'Donnell Performance measurement and improvement frameworks in health, education and social services systems: a systematic review Int. J. Qual. Health Care, February 1, 2010; 22(1): 44 - 69. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. Tsasis and S. M Owen Using the balanced scorecard in the development of community partnerships Health Serv Manage Res, February 1, 2009; 22(1): 33 - 38. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. Tsasis and B. Harber Using the balanced scorecard to mobilize human resources in organizational transformation Health Serv Manage Res, May 1, 2008; 21(2): 71 - 80. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. Giesen, M. Willekens, H. Mokkink, J. Braspenning, W. Van Den Bosch, and R. Grol Out-of-hours primary care: development of indicators for prescribing and referring Int. J. Qual. Health Care, October 1, 2007; 19(5): 289 - 295. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. Y. Rowe, M. A. Olewe, D. G. Kleinbaum, J. E. McGowan Jr, D. A. McFarland, R. Rochat, and M. S. Deming The influence of observation and setting on community health workers' practices Int. J. Qual. Health Care, August 1, 2006; 18(4): 299 - 305. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||