Replacing NHS England’s legacy theatre data pipeline: one cloud metric engine powering Model Hospital, SPaedIT and HVLC

Solution: Digital Products & Data transformation

Broad topic: Elective recovery, theatre productivity, surgical hubs

Team: Edge Health data engineering and analytics team, working with NHS England’s Theatres team

Methodology: Legacy migration, metadata-driven pipeline engineering, automated data quality validation, metric engine design, parallel-run metric assurance, opportunity modelling, clinical co-design, engineer upskilling and handover

Programmes used: UDAL (Azure Data Lake), Databricks, PySpark, SQL, Azure DevOps (Git), Power BI, Excel-based metadata configuration

Situation

Reducing the number of patients waiting for surgery is one of the NHS’s most pressing priorities, and raising theatre throughput is one of the main ways to do it. NHS England collects theatre data from every trust in England every two weeks. It processes, analyses and benchmarks that data to help Trusts drive performance improvement and allocate resources, including for hub-based working, where high-volume, low-complexity (HVLC) procedures are grouped so they can be delivered more efficiently.

The infrastructure behind this was reaching the end of its life. NHS England was replacing its Data Collections Framework, and ingestion of provider submissions was moving to Arden & GEM. The workflows that turned raw submissions into national metrics ran on a legacy Alteryx system, and they had to move to a modern cloud environment without disrupting the national products that relied on them.

Task

NHS England and GIRFT asked Edge Health to migrate the theatre data collection and metric-building workflows to Azure, on the national UDAL platform. The new pipeline had to reproduce the existing national metrics faithfully, run reliably on a fortnightly reporting cycle, and be transparent enough for teams across NHSE to trust.

We were also asked to upskill the data engineers on the GIRFT team so they could own the pipeline after handover. Finally, we used the new data to analyse the national opportunity from HVLC and hub-based working.

Action

Design principles. We built the pipeline around four principles:

• Metadata-driven: cleaning rules, derivations and metric logic are held in Excel-based configuration rather than hard-coded, which standardises processing and makes every rule visible to non-engineers.
• Modern tools (Databricks and PySpark): chosen to overcome the limits of SQL, lower the barrier between analytics and engineering, and align with NHS strategic architecture.
• Version-controlled: Git through Azure DevOps sits at the heart of the pipeline, so every change to code or metadata is auditable.
• Modular: code is written as reusable functions, like building blocks that can be pieced together to answer different questions.

Ingest. Providers submit CSVs to the AGEM Data Landing Platform, where schema validation catches structural errors at the point of entry. Files are processed to pseudonymise identifiable data and are then delivered to UDAL.

Daily processing. An automated Databricks pipeline loads each day’s data from the lake and runs it through several stages:

• Metadata-driven cleaning rules are applied.
• Data quality checks are logged for every row.
• Duplicate submissions are removed.
• The data is normalised into files, theatres, sessions, cases and patients, each with pseudonymised identifiers.
• Metadata-driven derivations then add flags, calculations and aggregations to each table.
• Theatre estate questionnaire data is curated alongside the submissions.

Fortnightly metrics. A metric engine applies the metric logic in the metadata to the normalised and derived tables and aggregates results by trust, site, month and week. A single metrics store serves multiple national products including Model Hospital, SPaedIT and HVLC analytics. Secondary data quality checks run across the whole submission, producing a DQ dashboard and feedback reports shared with providers. Each cycle therefore gives trusts specific information to improve their next submission.

Metric assurance. Before switching over, we ran the new metrics in parallel with the legacy Alteryx outputs and compared them trust by trust in Power BI dashboards. We tuned the metric engine until it matched or improved on the existing metrics, repeating the process for each of Model Hospital, SPaedIT and HVLC.

Insight. Using the curated data, we worked with clinical leads, theatre experts and trusts to model the opportunity across every theatre in the country. The central scenario asked what would happen if trusts scheduled HVLC cases routinely, at the activity levels achieved by early hubs and by clinicians trialling the approach.

Capability transfer and challenges. We delivered the work in close collaboration with NHSE’s team and upskilled newly joining data engineers in Databricks, Git and Power BI. Turnover in the NHSE engineering team and a learning curve on several new tools made planning harder. The modular, metadata-driven design and full version control meant that new joiners could understand the pipeline quickly and take it on.

Result

NHS England has replaced its legacy Alteryx workflows with an automated, version-controlled cloud pipeline on UDAL. It processes theatre data from every trust in England daily and produces national metrics every two weeks. One metric engine now powers Model Hospital, SPaedIT and HVLC reporting, so all three products work from the same assured data. SPaedIT, one of those products built by Edge health, went on to win the 2025 HSJ Data Integration Project of the Year.

Trusts receive automated data quality feedback every cycle, which improves the data at source. The opportunity analysis showed that routine HVLC scheduling could significantly increase throughput and reduce waiting lists. This work fed into new funding models for surgical hubs and helped speed up national roll-out.

The pipeline was handed over to NHSE’s own data engineers. The NHS now has a lasting, transparent capability to track theatre productivity as it works towards elective recovery.