News feature: How the Health and Social Care Information Centre approached the transition of the data backbone of the health service
It was a big call for the Health and Social Care Information Centre (HSCIC). Not only did it decide to take the management of NHS Spine – the infrastructure that connects more than 28,000 healthcare IT systems – in-house, but to redevelop it from a proprietary system to one that runs on open source software.
But it has been sufficiently pleased with the result to blow its trumpet over the transition, declaring that it pulled it off in 18 months, kept downtime to minimal levels, has made it 100% available since last August, and is on course for serious operational savings and improved performance.
Andrew Meyer, head of the Digital Delivery Centre at HSCIC, describes it as “a tremendous piece of work” and gives credit not just to the in-house team and supporting contractors, but BT Global Services, which had run the Spine for 10 years in its previous incarnation.
The figures illustrate the scale of the Spine and how much was at stake: it has 900,000 registered users, serves 21,000 organisations, holds more than 500 million records and documents and handles 6 billion messages a year, hitting 1,500 per second at peak times.
It brings together a collection of national applications, services and directories, the big ones being the Personal Demographics Service, Electronic Prescription Service, Summary Care Record, Secondary Uses Service and Care Identity Service.
Flexibility factor
Meyer says the reasons for the dual migration were largely down to cost and flexibility.
“The Spine had evolved over the 10 years that BT ran the service, with the original requirement moving on, so there was no clear documentation we could give to a supplier to ask them to build something,” he says. “All the knowledge was within HSCIC, we understood how the services should work, so it made sense for us to do it.
“When you balance that against the Government ICT strategy, which started to say no more contracts over £100 million and where possible use open source, it seemed like an obvious step. That started the idea.”
He makes no criticism of BT and emphasises that open source does not mean free, but mentions its practitioners' eagerness to help, how HSCIC is establishing a presence in the community and its potential to work with others in further developing the underlying software.
It took a few months to win support for the idea, involving the development of a proof of concept, drawing up of a business case and conversations with the Department of Health and Government Digital Service. Then planning began for the move with an eye on the details of each stage.
For each service the transition took two days running on the old and new systems and a third to monitor and pick up any issues. Things were done differently for the Secondary Uses Service with a 'lift and shift' approach, where about 80Tb of data was transferred to new hardware.
Logistics demand
“It was a logistics challenge more than anything,” Meyer says. “We had to get the technology right, but over the transition weekend we were confident of this and had to be sure we were switching things on and off at the right point, monitoring the services and making sure that clinically they were still safe.
“We had to coordinate the things happening within BT, have lines open to the US where work was being done for us, and coordinate with our own teams to ensure we were not missing anything.
“We paid close attention to everything. A couple of months before the transition weekend we started having rehearsals. They started as a paper exercise, then went to technical rehearsals, making sure everything was in place for the real event.”
It required a lot of coordination, involving a plan broken into 10-minute increments and an understanding of the function and impact of each element.
“When you think that the Spine is used 24/7/365 by the NHS and at any point there are 250,000 concurrent users, taking it down for any period at all will have a significant impact on patient safety.
“We started looking at transition from the point of view of how to minimise that. It had been out for 24-48 hours for database upgrades, and there had been instances of the Demographics Service slowing down, which caused difficulties in emergency departments when checking patient details. So we were aware of need to mitigate that transition.
“We started out with a 24-hour transition that was not good enough, and worked to get it down to five minutes. That was a tremendous piece of work by BT and the staff in HSCIC.”
Move to NoSQL
Moving to open source made it even more complex, creating what Meyer describes as “extreme technical challenges”. The legacy service had used a heavily bespoke version of an Oracle database, but the new one would use a Riak NoSQL system, effectively breaking the limitations of structured query language to provide more flexibility and speed in searches.
This had to be done over two stages, moving to a later version of Oracle before the switch to Riak, with a rewriting of all the underlying code. This took HSCIC into areas beyond its own capabilities, especially in DevOps, which brings together the software developers with other IT professionals.
A search for outside support led it to contract a Leeds based consultancy, BJSS, for the majority of the work, and enterprise application specialist Mastek for work on the Care Record Service. Meyer says their staff formed good relationships with the in-house team, to the extent that “when you walked into the office you couldn't tell who was from where”.
It was also seen as important to keep the team as small as possible to minimise the number of hand-offs in the work, and HSCIC managed to keep the number of people at 65 each for the Spine and Care Record Service projects.
The companies are now continuing to provide support as HSCIC takes time to build up its in-house skills.
Weekly evolution
“We're evolving the service on a weekly basis,” Meyer says. “We have agile teams and DevOps team that develop the Spine, and in that environment there's an incentive to get the code absolutely right, because if it goes wrong in the middle of the night it's them who are called out to fix it.”
This relates to one of the prime benefits from the transition, the scope to add new functions on anything up to a weekly basis – a sharp contrast to the 9-12 month gaps between any changes in the legacy service.
“The Spine service does not stand still and one of the key architectural decisions was to automate as far as we can and make it simple to operate, but also have the flexibility to change the system to meet the needs of the NHS and social care. Those things were built in from the start.”
In short, he is confident that the Spine is now fit for a constant evolution to meet the demands of the NHS. But he also points to more immediate financial savings, estimated at £21 million per year on operations, and a reduction of the time needed for a lot of processes dependent on the system. He cites the example of a clinician now being issued with a new smartcard in five minutes rather than the previous half an hour.
More important, however, is the faster transmission of patient care information. It is impossible to be precise about this as the speed depends heavily on the local broadband infrastructure of organisations using the Spine, but Meyer claims that they are already seeing clear improvements, and this can make a big difference in the quality of care.
“Saving a few seconds for everybody will have a huge cumulative effect,” he concludes.
Image from gov.uk under Open Government Licence 3.0
