Between 2005 and 2009, hundreds of patients at the Mid-Staffordshire NHS Foundation Trust died needlessly. A public inquiry into the scandal in 2013 concluded that it was essential that the Care Quality Commission (CQC), the independent regulator of all health and social care services in England, conduct ‘risk-related monitoring’ of healthcare providers.
In response, a big data monitoring system, known as Intelligent Monitoring, was created to calculate the risk of poor care and help the CQC identify which hospitals should be prioritised for inspection.
The use of patient information data to monitor and regulate healthcare is part of a wider, global trend of risk-based governance. With Economic and Social Research Council funding, King’s College London (KCL) researchers Henry Rothstein, Professor of Risk and Regulation, and David Demeritt, Professor of Geography, investigated the construction and effectiveness of these approaches.
They discovered that the CQC’s Intelligent Monitoring could not accurately predict the findings of on-site inspections, meaning inspection teams were not being sent where they were most needed.
The research led the CQC to redesign its statistical surveillance system, addressing the problems highlighted by the KCL team. The findings also have implications for the use of big data in risk-based governance tools by international healthcare regulators, as well as other sectors.
About the project
In recent years, government agencies have increasingly adopted risk-based approach to regulation. This centres on the idea that interventions should not try to prevent all possible harms but instead focus on controlling the greatest potential threats according to their consequence and probability.
Professor Demeritt explains:
Risk-based governance tools have the potential to reduce costs and bureaucracy. However, very important decisions are being made based on these systems, so we wanted to understand more about how and why they are being used, and how successful they are at improving regulation.
The research team compared risk predictions from the CQC’s statistical surveillance system against the findings of on-site inspections. They found the system’s risk scores not only failed to identify the worst performing trusts, but its predictions were more often wrong than they were right.
To understand why the system failed to serve as a reliable ‘smoke alarm’ of poor hospital healthcare, the team then undertook qualitative research, interviewing senior policy makers and drawing on historical policy analysis.
Professor Rothstein explains:
There were a number of causes for the discrepancy. Data is often reported at aggregate level for each trust, which may have multiple premises.
So, you might have a hospital with world-class stroke care but poor standards of maternity care, and this is given just one score.
Then there are bigger questions over indicators. What constitutes, good care: is it timely, is it safe, does it factor in patient experience?
If you’re making decisions based on data, then that data has to be meaningful and there must be an agreement on what it stands for.
Impact of the project
The research improved understanding within the CQC and the wider healthcare community of how to design and use statistical surveillance systems to detect poor quality hospital care.
A new monitoring system for the CQC
To address the problems identified by the KCL research and help direct on-site inspection teams more appropriately, the CQC replaced its Intelligent Monitoring system with a new statistical surveillance system.
In a written submission to the Health and Social Care Committee it stated:
[We have] conducted extensive evaluation internally, looking in detail at the relationship between quality ratings and individual indicators and combinations of indicators to identify those with the strongest relationship, as well as the overall risk score used in the paper submitted by Professor Demeritt and Dr Rothstein. The new model builds on what we have learned from this evaluation.
Shaping international understanding of statistical surveillance
As part of their research, the team also looked at systems in France, Germany and the Netherlands. Their findings revealed fundamental differences in the way healthcare quality indicators are constructed, measured and used.
The team also presented the research to the Organisation for Economic Co-operation and Development (OECD) Working Party on Health Care Quality Outcomes. This highlighted the challenges of using statistical surveillance methods to its 37 country representatives.
Professor Demeritt explains:
Many healthcare systems across world are using data to improve their healthcare systems but every country thinks about quality of care in different ways. For example, in Germany, it’s about surgery success, whereas France has more emphasis on patient experience.
We need to understand that these variations could be impeding international efforts to benchmark quality and identify best practice. It’s not a one-size-fits-all approach.
Find out more
Read more about the team’s research on the CQC’s Intelligent Monitoring system (KCL)
Subsequent research on the relevance for international healthcare regulators (KCL)
Full article: Accounting for failure: risk-based regulation and the problems of ensuring healthcare quality in the NHS (tandfonline.com)
British Medical Journal article on assessing the ability of the CQC’s statistical surveillance tool
Top image: Credit: da-kuk, E+ via Getty Images