International Journal for Quality in Health Care 15:7-13 (2003)
© 2003 International Society for Quality in Health Care
Paper |
Risk-adjusted sequential probability ratio tests: applications to Bristol, Shipman and adult cardiac surgery
1MRC Biostatistics Unit, Institute of Public Health, Cambridge
2Dendrite Clinical Systems, Henley-on-Thames, Oxfordshire
3Thoracic Surgery, Guy’s and St Thomas Hospitals, London, UK
Objective. To investigate the use of the risk-adjusted sequential probability ratio test in monitoring the cumulative occurrence of adverse clinical outcomes.
Design. Retrospective analysis of three longitudinal datasets.
Subjects. Patients aged 65 years and over under the care of Harold Shipman between 1979 and 1997, patients under 1 year of age undergoing paediatric heart surgery in Bristol Royal Infirmary between 1984 and 1995, adult patients receiving cardiac surgery from a team of cardiac surgeons in London, UK.
Main outcome measure. Annual and 30-day mortality rates.
Results. Using reasonable boundaries, the procedure could have indicated an ‘alarm’ in Bristol after publication of the 1991 Cardiac Surgical Register, and in 1985 or 1997 for Harold Shipman depending on the data source and the comparator. The cardiac surgeons showed no significant deviation from expected performance.
Conclusions. The risk-adjusted sequential probability test is simple to implement, can be applied in a variety of contexts, and might have been useful to detect specific instances of past divergent performance. The use of this and related techniques deserves further attention in the context of prospectively monitoring adverse clinical outcomes.
Keywords: adverse clinical outcomes, general practitioners, monitoring, mortality, paediatric and adult cardiac surgery, risk-adjustment, sequential probability ratio tests
The need for systems to monitor clinical performance has been brought into particular focus by the report of the Bristol Royal Infirmary Inquiry [1] and the finding that general practitioner Harold Shipman murdered over 200 of his patients [2]. Rarer adverse events may require cumulative monitoring over time rather than, for example, examination of annual data. Historical industrial quality control procedures have been recommended, such as Shewhart control charts and cumulative sum (CUSUM) techniques [3,4]. The medical context does, however, add an additional complexity in the need to adjust for case-mix in an attempt to avoid clinicians or trusts being unfairly penalized for treating higher-risk patients. A common suggestion is to plot the accumulating difference between the observed number of adverse events, and the number expected according to an established risk-adjustment procedure [5,6]. If concerned with mortality, for example, this means the cumulative ‘excess deaths’ (or conversely ‘lives saved’) can be monitored [7].
The problem then arises of setting appropriate ‘thresholds’ on such plots to indicate the need for further scrutiny. These can be set for a single time point using standard methods for confidence intervals [8]. However, this does not allow for the well-known problem of repeated testing in which a true null hypothesis is certain to be eventually rejected [9], which in this context could correspond to an unreasonable number of false accusations of poor performance. The ‘risk-adjusted CUSUM’ has been suggested as a technique for dealing with both risk-adjustment and sequential testing, but setting appropriate thresholds is not straightforward [10].
In this paper we investigate a risk-adjusted version of the classic sequential probability ratio test (SPRT) that was developed for quality control of military supplies in the Second World War: this is shown to take the form of a simple adaptation of a cumulative ‘observed – expected’ plot with horizontal thresholds. Examples are provided in three monitoring contexts: annual surgical mortality (paediatric cardiac surgery in Bristol), adverse events in a population (Harold Shipman’s practice) and individual operations (cardiac surgery by a group of surgeons). We conclude that the risk-adjusted SPRT is a simple technique that could be useful within a clinical monitoring system.
Materials and methods
Formal statistical methods for sequential analysis were developed in 1943 independently by Barnard in the UK and Wald in the US [11,12]. The SPRT is the most powerful method for discriminating between two hypotheses [12], and was recommended well over 40 years ago in a medical context for clinical trials and clinical experiments [13,14].
The formal procedure has three components: a running test statistic, thresholds for the statistic that determine statistical significance, and actions to be taken on crossing a threshold.
The test statistic
Suppose we have two hypotheses: a null hypothesis H0 corresponding to performance as expected, and H1 to a level of performance deemed importantly divergent. Wald showed that the most powerful sequential comparison based on an accumulating set of data takes the form of a running ‘log-likelihood ratio’ (LLR) [12], which is increased or decreased after each observation by a quantity depending on the event observed and, in its risk-adjusted form, the expected outcome if that event were performance ‘as normal’, i.e. under H0. This procedure is described in the Appendix. Here, we have set our example charts to detect a doubling in the relevant process parameter, purely for illustration. In practice, one should set a chart to detect the lowest of the range of values deemed unacceptable.
Thresholds
Wald showed that sampling should continue if the LLR lies between two thresholds denoted a and b: when LLR exceeds b, stop and reject H0 in favour of H1, and vice versa when LLR is less than a [12]. Thus, the boundaries take the form of horizontal lines. Using the traditional language of statistical hypothesis testing, let
 | (1) |
 | (2) |
and ß, we can use the following equations, developed by Wald [12], to calculate (approximately) appropriate thresholds a and b
 | (3) |
 | (4) |
and ß should reflect the relative ‘costs’ of making the two types of error. For example, if we wish to avoid falsely identifying an adequate surgeon as ‘higher-risk’, then should be made very small, whereas if we consider it a very serious mistake to miss a poorly performing surgeon, then ß should be made very small. We have adopted a convention of equal and ß, with illustrative examples given in Table 1 of the Appendix.
|
Instead of choosing a single value for
and ß, a set of horizontal lines can be drawn on the chart to indicate different degrees of urgency: for example, a single centre might use = ß = 0.1 as an ‘alert’ threshold and = ß = 0.01 for ‘alarm’. When monitoring many individuals or centres more stringent boundaries may be appropriate because of the many comparisons being made: of 10 centres performing normally, we would expect one to cross the ‘alert’ boundary by chance alone. A Bonferroni-like adjustment might suggest, when monitoring n individuals or institutions, using 0.10/n and 0.01/n for ‘alert’ and ‘alarm’ respectively.
Action
Strictly speaking, we should pre-specify values for and ß and stop monitoring once either threshold has been crossed. An alternative is to restart the procedure when, say, we cross the lower boundary and so are confident there is no increase in mortality. This has the advantage that it is not possible to build up excessive ‘credit’ and so gains sensitivity to changes in performance, but also has the disadvantage that the strict interpretation of and ß is lost. However, as we take the view that any such procedure is only an aid to clinical monitoring, we shall allow for restarts when crossing the ‘reject H1’ boundary in our examples. However, when divergent performance has been detected (‘reject H0’) we shall not assume a restart takes place.
Three datasets are used for illustration.
Bristol
As part of the Bristol Royal Infirmary Inquiry [1], annual mortality rates for open-heart surgery on children under 1 year of age were made available from the Cardiac Surgical Register between 1985 and 1995, and from Hospital Episode Statistics between 1991 and 1995 [15]. The observed mortality is for operations performed in the Bristol Royal Infirmary, whereas the expected mortality rates are the median rates in 11 other centres; no adequate risk-adjustment procedure was available [16].
Shipman
We consider two analyses of mortality rates for men and women aged 65 years and over in the practice list of Dr Harold Shipman using data provided by Baker [17]: firstly, death between 1977 and 1998 taking place in the patient’s home or in practice premises and having certificates signed by the GP himself, as compared with the rate of such certificates signed by a sample of local GPs, and secondly, all deaths in Shipman’s practice between 1987 and 1998, as compared with age-adjusted expected rates for England and Wales.
Cardiac surgeons
There are many available data sets of mortality rates for adult cardiac surgery [18]. We took a set analysed previously using the risk-adjusted CUSUM technique [10]. These comprise the 30-day mortality from individual coronary artery bypass graft operations carried out between 1994 until 1998, risk-adjusted using the Parsonnet system which has been recalibrated on surgery in the same unit between 1992 and 1993 [10]. Data for two surgeons are provided.
Results
Bristol
The cumulative excess mortality in Bristol from the two data sources is displayed in Figure 1A, which is transformed in Figure 1B to the SPRT plot for monitoring. There are 12 centres in England performing such surgery, so if such a procedure were to have been used for prospective monitoring it may have been reasonable to demand ’s of around 0.01 for ‘alert’ and 0.001 for ‘alarm’. The Cardiac Surgical Register data crossed the = 0.05 boundary in 1989, and the 0.001 boundary in 1991, whereas the Hospital Episode Statistics results crossed the 0.01 boundary in 1993 and passed 0.001 in 1994.
|
Shipman
The cumulative excess of deaths certified by Shipman at home or in practice premises from 1977 is shown in Figure 2A, revealing a steadily increasing pattern for females, particularly accelerating after 1995. The final totals closely mirror the conclusion of the public inquiry [2], that Shipman killed, or probably killed, 180 women and 55 men aged 65 years or over. There are around 9000 general practices involving 27 000 doctors in England, suggesting that any monitoring procedure would want to use stringent criteria such as = ß = 0.000001. Figure 2B shows that the graph for females dying at home or in the practice crossed this limit in 1985, at which point there were 41 excess female deaths. At the point where Shipman was arrested, in 1998, this excess had grown to 174.
|
However, compared with England and Wales, mortality rates irrespective of place of death, excess total mortality on Shipman’s list was broadly in line with national rates between 1988 and 1994 and only started accumulating in 1995 for women (Figure 3A). Figure 3B shows that the cumulative LLR thus builds up substantial credit and therefore it is not until 1997 that an ‘alarm’ threshold of 0.000001 is crossed. However, by 1992 the test would have confidently concluded that Shipman did not have a raised mortality rate and the monitoring procedure could therefore have been restarted, and in Figure 3C we see this could have led to an alarm being raised at the end of 1996, when there were 67 excess deaths in females aged over 65 years, compared with 119 by 1998.
|
Cardiac surgeons
In the data sets selected for illustration, surgeon 2 has some excess mortality early on in the series but made up for this later on (Figure 4A), whereas surgeon 1 showed some small improvement in mortality over that expected. The SPRT plot in Figure 4B shows no evidence for increased risk and an eventual confident rejection of that hypothesis for both surgeons. Figure 4C, similarly, shows no evidence for substantially decreased risk: in fact there is a confident rejection of a decrease in risk at around case 100 for surgeon 2, at which point it might be reasonable to restart the monitoring procedure.
|
Discussion
The techniques discussed in this paper can be seen as a natural development from control charts and CUSUMs, and naturally complement intuitively attractive plots of cumulative observed–expected mortality [5,6]. They are easily implemented in spreadsheet programs.
Bristol
The analysis suggests that the divergent performance for Bristol might have been detected earlier using a simple statistical technique applied to routinely collected data. However, we note that in practice the Cardiac Surgical Register results were not available to centres until at least a year has elapsed, whereas Hospital Episode Statistics were not used for monitoring purposes.
Shipman
Our first analysis may well over-estimate the mortality attributable to wrongdoing, as there are many innocuous reasons why a GP might sign more certificates for deaths occurring at home, say because of his care policy for terminally ill patients. In addition, there were only between four and six control GPs in any year, so the control group is local but limited and possibly non-representative. The second comparison uses national death rates as controls, but could be insensitive if Shipman’s victims tended to be in poor health and hence his actions did not increase overall death rates. This appears to have been the pattern until around 1995. It is important to note that the data used in our analysis were specially put together after Shipman’s conviction and no such routine monitoring was in place: in particular, death certificates do not carry details of the GP and linkage must be made through the NHS Central Register. Nevertheless, the results suggest that a simple monitoring procedure could have led to earlier detection of Shipman’s crimes and a substantial saving of lives.
Cardiac surgeons
This example demonstrates the importance of adjustment for pre-operative risk in surgical monitoring. The SPRT that assumes constant baseline risk, that is, that does not adjust for risk, suggests that the surgeons are performing divergently. However, the risk-adjusted version of the SPRT the SPRT, which does take into account factors that are beyond a surgeon’s control, indicates that the surgeons are in fact performing similarly.
The issue of ‘restarts’ is quite complex. Crossing the lower boundary confirming ‘performance as expected’ is reassuring but otherwise is not of great interest, and it seems natural to begin a new monitoring session at that stage in order to retain sensitivity to changes in performance. However, a series of such restarts increases the overall chance of a Type I error eventually to 1 and, because in effect we are avoiding ‘accepting’ the null hypothesis, decreases the Type II error eventually to 0. If we further encourage restarts by making ß larger, then the behaviour of the SPRT begins to closely resemble that of the risk-adjusted CUSUM [10,19]. This latter technique uses the same form of LLR as the SPRT for monitoring, but has no concept of accepting the null hypothesis: the scheme is restarted each time it crosses 0, and the boundaries are set in a pragmatic way. Both SPRTs and risk-adjusted CUSUMs warrant further research, particularly as the approximation underlying the boundaries of the SPRT may be questionable for Poisson data.
Implementation of such monitoring systems requires good-quality data, as well as specification of thresholds and expected performance. If a SPRT is being used within an institution then setting thresholds requires consideration of the seriousness of each type of error. However, if a central audit body is examining multiple SPRTs from a number of institutions then they need to additionally take into account the multiple comparisons being made. As a rule-of-thumb we have suggested the simple Bonferroni procedure of dividing the error rates by the number of institutions being monitored, but this requires further investigation.
The risk-adjustment scheme used to derive expected performance can cause difficulties. For example, if monitoring GPs, should comparison only be made with other practitioners in the immediate neighbourhood, or those with similar socio-economic practice lists? An appropriate baseline is essential for fair comparisons. Finally, Frankel and colleagues claim that a monitoring system may only detect really extreme divergence [20]. We have somewhat arbitrarily identified a doubling of risk as being ‘important’, and shown sensitivity of the procedure even allowing for multiple comparisons.
The SPRT is one of the simplest prospective monitoring schemes: slightly more sophisticated developments might include the risk-adjusted CUSUM [10], systematic downweighting of historical cases or shrinkage estimation of rates towards the average [4,21,22]. Such formal statistical methodology may aid routine monitoring of clinical performance.
Appendix
Here we describe calculation of the LLR weights for charts detecting a doubling in the odds or risk ratio (such as the charts illustrated in this paper), with reference to the cardiac surgery and Shipman examples. Also, a table giving the values of the boundaries a and b for various equal and ß is provided, to illustrate the use of the Wald equations given in the Materials and methods section of this paper.
Suppose we observe data x1,x2,...,xn and wish to compare two hypotheses H0, H1. Then the most powerful test is based on the ‘log-likelihood ratio’ LLR = Ziln(P1(xi)/P0(xi)) where P0(x) is the probability of observing the data under H0 and so on.
Consider patient by patient monitoring, such as in the cardiac surgery example (Bernoulli data). Given that the pre-operative risk is P, a doubling in the odds on death is optimally detected by adding the score –ln(1 + P) to the running score total if the patient survives, and 0.69 – ln(1 + P) if the patient dies. The value 0.69 is equal to ln(2). For detecting a halving of the odds on death, this value would be replaced by ln(1/2). To adjust for risk, we can allow the value of P to depend on individual patient risk factors such as age, sex and diabetes status.
For monitoring of aggregate data, such as the Shipman data (Poisson data), a doubling in risk of death is optimally detected by adding 0.69O – E to the running LLR, where O is the observed number of deaths and E the expected number of deaths over a fixed time interval. Note that this is a simple adjustment to the cumulative ‘excess mortality’ plot of O – E. To adjust for risk, we simply use expected counts based on a suitable comparator (England and Wales averages, for example) for the risk categories that we are interested in.
We are grateful to Professor Vern Farewell for many useful discussions.
Address reprint requests to Olivia Grigg, MRC Biostatistics Unit, Institute of Public Health, University Forvie Site, Robinson Way, Cambridge CB2 2SR, UK. E-mail: olivia.grigg{at}mrc-bsu.cam.ac.uk 
Accepted for publication August 28, 2002.
References
- BRI Inquiry Panel. Learning from Bristol: The Report of the Public Inquiry into Children’s Heart Surgery at the Bristol Royal Infirmary 1984–1995. London, UK: The Stationery Office, 2001. Available from http://www.bristol-inquiry.org.uk/final_report/.
- Shipman Inquiry: The First Report. London, UK: The Stationary Office, 2002. Available from http://www.the-shipman-inquiry.org.uk/reports.asp.
- Mohammed M, Cheng K, Rouse A, Marshall T. Bristol, Shipman, and clinical governance: Shewhart’s forgotten lessons. Lancet 2001; 357: 463–467.
- de Leval MR, Francois K, Bull K, Brawn W, Spiegelhalter DJ. Analysis of a cluster of surgical failures: application to a series of neonatal arterial switch operations. J Thorac Cardiovasc Surg 1994; 107: 914–924.
- Lovegrove J, Valencia O, Treasure T, Sherlaw-Johnson C, Gallivan S. Monitoring the results of cardiac surgery by variable life-adjusted display. Lancet 1997; 350: 1128–1130.
- Poloniecki J, Valencia O, Littlejohns P. Cumulative risk-adjusted mortality chart for detecting changes in death rate: observational study of heart surgery. Br Med J 1998; 316: 1697–1700.
- Lawrance R, Dorsch M, Sapsford R et al. Use of cumulative mortality data in patients with acute myocardial infarction for early detection of variation in clinical practice: observational study. Br Med J 2000; 323: 324–327.
- Sherlaw-Johnson C, Lovegrove J, Treasure T, Gallivan S. Likely variations in perioperative mortality associated with cardiac surgery: when does high mortality reflect bad practice? Heart 2000; 84: 79–82.
- McPherson CK. Statistics: the problem of examining accumulating data more than once. N Engl J Med 1974; 290: 501–502.
- Steiner SH, Cook RJ, Farewell VT, Treasure T. Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics 2000; 1: 441–452.
- Barnard GA. Sequential tests in industrial statistics (with discussion). J R Statist Soc 1946; 8(suppl.): 1–26.
- Wald A. Sequential tests of statistical hypotheses. Ann Math Statist 1945; 6: 117–186.
- Armitage P. Sequential tests in prophylactic and therapeutic trials. Q J Med 1954; 23: 255–274.
- Bartholomay AF. The sequential probability ratio test applied to the design of clinical experiments. N Engl J Med 1957; 256: 498–505.
- Spiegelhalter DJ, Evans S, Aylin P, Murray GD. Overview of statistical evidence presented to the Bristol Royal Infirmary Inquiry concerning the nature and outcomes of paediatric cardiac surgical services at Bristol relative to other specialist centres from 1984 to 1995. Bristol Royal Infirmary Inquiry 2000. Available from http://www.bristol-inquiry.org.uk/final_report/annex_b/images/Spiegelhalteretal_O_statev1.pdf (Accessed August 10 2001).
- Aylin P, Alves B, Best N et al. Comparison of UK paediatric cardiac surgical performance by analysis of routinely collected data 1q984-96: was Bristol and outlier? Lancet 2001; 358: 181–187.
- Baker R. Harold Shipman’s Clinical Practice 1974–1998: A Review Commissioned by the Chief Medical Officer. London: The Stationery Office, 2001.
- Keogh BE, Kinsman R. National Adult Cardiac Surgical Database Report 1999–2000. The Society of Cardiothoracic Surgeons of Great Britain and Ireland.
- Grigg O. A Comparison of Methods for Monitoring Surgical Performance. Master’s thesis, Department of Statistical Science, University College London, 2001.
- Frankel S, Sterne J, Smith G. Mortality variations as a measure of general practitioner performance: implications of the Shipman case. Br Med J 2000; 320: 489.
- Goldstein H, Spiegelhalter DJ. Statistical aspects of institutional performance: league tables and their limitations (with discussion). J R Stat Soc Series A 1996; 159: 385–444.
- Christiansen C, Morris C. Improving the statistical approach to health care provider profiling. Annal Intern Med 1997; 127: 764–768.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  M. E. Matheny, D. A. Morrow, L. Ohno-Machado, C. P. Cannon, M. S. Sabatine, and F. S. Resnic Validation of an Automated Safety Surveillance System with Prospective, Randomized Trial Data Med Decis Making, March 1, 2009; 29(2): 247 - 256. [Abstract] [PDF]
|
|
|
|
 |
|
 J. M. Ansermino, J. P. Daniels, R. T. Hewgill, J. Lim, P. Yang, C. J. Brouse, G. A. Dumont, and J. B. Bowering An Evaluation of a Novel Software Tool for Detecting Changes in Physiological Monitoring Anesth. Analg., March 1, 2009; 108(3): 873 - 880. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 G. L. Grunkemeier, R. Jin, and Y. Wu Cumulative sum curves and their prediction limits. Ann. Thorac. Surg., February 1, 2009; 87(2): 361 - 364. [Full Text] [PDF]
|
|
|
|
|
|
V. T. Tsang, K. L. Brown, M. J. Synnergren, N. Kang, M. R. de Leval, S. Gallivan, and M. Utley Monitoring risk-adjusted outcomes in congenital heart surgery: does the appropriateness of a risk model change with time? Ann. Thorac. Surg., February 1, 2009; 87(2): 584 - 587. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 B. Kunadian, J. Dunning, A. P Roberts, R. Morley, D. Twomey, J. A Hall, A. G C Sutton, R. A Wright, D. F Muir, and M. A d. Belder Cumulative funnel plots for the early detection of interoperator variation: retrospective database analysis of observed versus predicted results of percutaneous coronary intervention BMJ, April 26, 2008; 336(7650): 931 - 934. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. Coory, S. Duckett, and K. Sketcher-Baker Using control charts to monitor quality of hospital care with administrative data Int. J. Qual. Health Care, February 1, 2008; 20(1): 31 - 39. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 V. A. Ferraris, F. H. Edwards, D. M. Shahian, and S. P. Ferraris Risk Stratification and Comorbidity Card. Surg. Adult, January 1, 2008; 3(2008): 199 - 246. [Full Text]
|
|
|
|
 |
|
 A. Patwardhan and D. Patwardhan How Transferable is Deming's Approach to a Health and Social Care Setting? Journal of Health Management, December 1, 2007; 9(3): 443 - 457. [Abstract] [PDF]
|
|
|
|
 |
|
 R. Wolfe, S. Bolsin, M. Colson, and P. Stow Monitoring the rate of re-exploration for excessive bleeding after cardiac surgery in adults Qual. Saf. Health Care, June 1, 2007; 16(3): 192 - 196. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. J Biau, M. Resche-Rigon, G. Godiris-Petit, R. S Nizard, and R. Porcher Quality control of surgical and interventional procedures: a review of the CUSUM Qual. Saf. Health Care, June 1, 2007; 16(3): 203 - 207. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. P. McGuirk, J. Stickley, M. Griselli, O. F. Stumper, S. J. Laker, D. J. Barron, and W. J. Brawn Risk assessment and early outcome following the Norwood procedure for hypoplastic left heart syndrome Eur. J. Cardiothorac. Surg., May 1, 2006; 29(5): 675 - 681. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
N. Kang, V. T. Tsang, S. Gallivan, C. Sherlaw-Johnson, T. J. Cole, M. J. Elliott, and M. R. de Leval Quality assurance in congenital heart surgery. Eur. J. Cardiothorac. Surg., May 1, 2006; 29(5): 693 - 697. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
G. J. Murphy, C. A. Rogers, M. Caputo, and G. D. Angelini Acquiring Proficiency in Off-Pump Surgery: Traversing the Learning Curve, Reproducibility, and Quality Control Ann. Thorac. Surg., November 1, 2005; 80(5): 1965 - 1970. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 G Leandro, N Rolando, G Gallus, K Rolles, and A K Burroughs Monitoring surgical and medical outcomes: the Bernoulli cumulative SUM chart. A novel application to assess clinical interventions Postgrad. Med. J., October 1, 2005; 81(960): 647 - 652. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. Billett, N. Kendall, and P. Old An investigation into GPs with high patient mortality rates: a retrospective study J. Public Health Med., September 1, 2005; 27(3): 270 - 275. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. A. Rogers, J. S. Ganesh, N. R. Banner, R. S. Bonser, and On behalf of the steering Group Cumulative risk adjusted monitoring of 30-day mortality after cardiothoracic transplantation: UK experience Eur. J. Cardiothorac. Surg., June 1, 2005; 27(6): 1022 - 1029. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. A. Rogers, B. C. Reeves, M. Caputo, J. S. Ganesh, R. S. Bonser, and G. D. Angelini Control chart methods for monitoring cardiac surgical performance and their interpretation J. Thorac. Cardiovasc. Surg., December 1, 2004; 128(6): 811 - 819. [Full Text] [PDF]
|
|
|
|
|
|
D.J. Spiegelhalter Monitoring clinical performance: A commentary J. Thorac. Cardiovasc. Surg., December 1, 2004; 128(6): 820 - 822. [Full Text] [PDF]
|
|
|
|
|
|
M. Caputo, B. C. Reeves, C. A. Rogers, R. Ascione, and G. D. Angelini Monitoring the performance of residents during training in off-pump coronary surgery J. Thorac. Cardiovasc. Surg., December 1, 2004; 128(6): 907 - 915. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. Sherlaw-Johnson, S. Gallivan, T. Treasure, and S.A.M. Nashef Computer tools to assist the monitoring of outcomes in surgery Eur. J. Cardiothorac. Surg., November 1, 2004; 26(5): 1032 - 1036. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M A Mohammed Using statistical process control to improve the quality of health care Qual. Saf. Health Care, August 1, 2004; 13(4): 243 - 245. [Full Text] [PDF]
|
|
|
|
|
|
N. Kang, T. Cole, V. Tsang, M. Elliott, and M. de Leval Risk stratification in paediatric open-heart surgery Eur. J. Cardiothorac. Surg., July 1, 2004; 26(1): 3 - 11. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. A Mohammed, A. Rathbone, P. Myers, D. Patel, H. Onions, and A. Stevens An investigation into general practitioners associated with high patient mortality flagged up through the Shipman inquiry: retrospective analysis of routine data BMJ, June 19, 2004; 328(7454): 1474 - 1477. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. P. Morton The use of statistical process control methods in monitoring clinical performance Int. J. Qual. Health Care, August 1, 2003; 15(4): 361 - 362. [Full Text] [PDF]
|
|
|
|
|
|
S. Bolsin and M. Colson Making the case for personal professional monitoring in health care Int. J. Qual. Health Care, February 1, 2003; 15(1): 1 - 2. [Full Text] [PDF]
|
|
|
|
|
|
T. O. Lim Statistical process control tools for monitoring clinical performance Int. J. Qual. Health Care, February 1, 2003; 15(1): 3 - 4. [Full Text] [PDF]
|
|
|
|
|
|
J. C. Benneyan and A. D. Borgman Risk-adjusted sequential probability ratio tests and longitudinal surveillance methods Int. J. Qual. Health Care, February 1, 2003; 15(1): 5 - 6. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||