Integrating surgeon-derived risk estimates to improve clinical data-based prediction of post-operative complications
*Margaret T. Berrigan MD, Brendin Beaulieu-Jones MD, MBA, MBI, *Alma Burbano MD, *Gabriel Brat MD, MPH
Surgery, Beth Israel Deaconess Medical Center, Boston, MA
Background: Several validated risk calculators have been developed to assist clinicians in predicting risk of postoperative complications. These calculators leverage multiple discrete clinical datapoints to generate consistent predictions. Surgeon risk estimates rely on clinical judgement, and, when measured after surgery, take into account intraoperative events. We sought to evaluate how integrating surgeon risk estimates made before and after surgery might impact predictive performance of these risk scores. Study Design: From 2022-2023 surgeons at a single academic medical center were provided with patient-specific risk scores before surgery. Risk scores were generated using the ACS NSQIP Surgical Risk Calculator or, for emergency general surgery cases, the POTTER Calculator. Surgeons were asked to estimate their patient’s risk of any complication relative to the provided risk score. Surgeons were surveyed again immediately after surgery to assess for changes in their risk estimate. Thirty-day clinical outcomes were collected and binomial regression models predicting any post-operative complication, as defined by ACS NSQIP, were trained using patient-specific risk scores and surgeon risk adjustment data. Results: Pre- and post-surgery surgeon risk estimates were collected for 575 cases (representing 100 procedure types) from surgeons in four divisions with 0-27 years in practice post-training. Half of cases (49.04%) were urgent or emergent and half (51.65%) were associated with the acute care surgery service. The overall rate of any complication was 30.61%. Before surgery, 32.17% of patients were estimated to be at higher risk than reflected by their risk score. After surgery, 61.08% of risk estimates for these high-risk patients did not change, 21.08% increased further. Model performance is shown in table 1. Conclusions: Pre- and post-surgery risk estimates offer valuable clinical insight not otherwise accounted for in objective data-based risk calculators. Post-surgery risk adjustments in particular offer the opportunity to account for intraoperative events that may significantly alter a patient’s risk. The ability to provide high-quality, personalized postoperative care depends on our understanding of a patient’s risk profile. Further study is required to understand the scenarios in which surgeon judgment is especially valuable and when it may be misleading. Creating a system in which clinical data and surgeon judgement are integrated to produce a reliable and accurate risk score would allow targeting of surveillance and intervention mechanisms to promote early detection of, and appropriate response to, high-risk patients.
Model performance for prediction of any post-operative complication using calculated risk scores, pre-surgery surgeon risk estimates, and immediate post-surgery surgeon risk estimates
Model | AUC (95% CI) | Sensitivity | Specificity |
Pre-surgery reference score | 0.77 (0.72-0.81) | 0.86 | 0.55 |
Pre-surgery reference score + pre-surgery risk estimate | 0.79 (0.74-0.83) | 0.78 | 0.62 |
Pre-surgery reference score & risk estimate + post-surgery risk estimate | 0.80 (0.76-0.84) | 0.79 | 0.65 |
Back to 2024 Abstracts