Data Engineering as the Backbone of Modern Healthcare Analytics: Experience from Our Work in the NHS

May 4, 2023 • Reading time 3 minutes

We are known for our analytics

As a healthcare technology firm, we are known for our expertise in analytical work for the NHS. This includes operational improvement or benchmarking delivered visually via dashboards, and digital tools to improve scheduling, bed planning or empower better decision-making on the ground.

But people are realising that analytics need good engineering

Until recently, we were seldomly asked about how it all works under the hood. However, we are noticing that as trusts and national bodies have developed more and more dashboards, analytical tools and data collection systems, they are noticing errors, outages, delays that are due to back-end processes not being scaled and maintained in line with the increased demands of the analytics teams. In turn, questions that we are asked are pivoting from “can you help us build a dashboard?” to “can you help us make sure our dashboard works and is usable?”.

Example of a data engineering failure

New tools make it easier than ever to do it well

At the same time, the tools that are available for data engineering, especially in the cloud with Azure, AWS or Gcloud, are making it easier and easier to get it right. And when data engineering is done well, it can ensure that analysts have the right data, refreshed without delay, displaying the right things in the right place.

Data engineering is like plumbing: data flowing to where you need it at the right time.

To deliver high-quality analytics to our clients we have invested heavily in data engineering, both in terms of tools and talent. Our team of data engineers is skilled in building efficient and robust data pipelines using cloud technologies such as Azure, which allows us to process, store, and analyse vast amounts of data in real-time and get our data plumbing right first time.

Example: Work with a National Programme on processing a national data collection

We recently worked with a national programme that collected theatre data from every Trust in the country. The data needed to be organised, flow together into a coherent schema and ultimately be validated, checked and organised into templates and tables. All of this required to be done in a stable, scalable and reproducible fashion. We achieved this using Azure tools, which allowed the data to power high quality analytics and feed into performance improvements for the entire country.

  • For a detailed case study of this work, click here.

Data engineering is the foundation of analytics

As data and dashboards proliferate, data engineering is becoming increasingly important and more senior leaders are noticing that problems with dashboards may lie under the hood. At Edge, we believe that getting the data right requires even more than engagement on the ground and high quality analysts. It needs the right plumbing to ensure that integrity, scale and reliability of any data products are guaranteed. Talk to us about our data engineers, and rest reassured that any data products we deliver have the highest quality plumbing underneath.

George Caws

George is a Senior Analyst at Edge. He has experience in data analysis, visualisation, and building models and dashboards to automate workflows. He is Edge's lead in data engineering including SQL and Azure.