Case-mix adjustment of NHS patient outcomes

Recently Edge health was retained by a large NHS programme in the UK to support the development of an approach to risk adjusting surgical outcome data. The client in question currently reports surgical outcome data back to practicing surgeons (confidentially) to aid them with their revalidation as well as quality improvement.

Why was risk adjustment of surgical outcome data important for our client?

Consider the following scenario, comparing two consultants. Consultant A has a very low mortality rate (0.2% of patients on which this consultant operated on sadly died), while Consultant B has a higher mortality rate (3.2% of patients undergoing the same procedure with this consultant sadly died). These “raw” rates do not consider the patient characteristics, or underlying comorbidities (case-mix) of the patients treated by each consultant.

For example, Consultant A may have seen very young, healthy patients, while Consultant B has seen older patients with more comorbidities such as heart conditions or diabetes. Despite undertaking the same surgical procedure in theory it would be unfair to say Consultant B actually performed worse.

A proper risk adjustment approach therefore has to take into account the underlying patient characteristics to adjust important outcome measures. Here, in our example, given the underlying patient characteristics, Consultant B actually performed better than Consultant A (after risk adjustment).

The resulting model

We then developed and validated a risk adjustment model that used regression based methods to predict expected outcomes based on the observed data and used this to adjust the outcomes at surgeon level. We also implemented machine learning methods to determine the performance of our model. This was especially important in this instance, as the smaller consultant level samples make it harder to evaluate performance. A key risk for us was that at individual surgeon level, risk adjustments can quickly become unstable or unreliable due to the low number of observations. It was important to present only statistically stable, reliable, risk-adjusted rates. We thus had to determine a dynamic lower bound on the number of observations required per consultant for risk adjustment, balancing consultant coverage and statistical validity.

For this project, we presented the risk adjusted rates in funnel plots with control limits (labelled alert and alarm for research purposes only) (Figure 2 below). The data is now used with selected clinicians on a confidential and voluntary basis with the goal to undertake detailed evaluation and testing and with clinician support ultimately make it available to all consultants in the country.

We are working on making the code publicly available if our client is supportive as soon as the pilot phase concludes. If you are interested in more details please do not hesitate to contact us at [email protected]