A new AI tool for summarising government consultation responses has been used for the first time and produced nearly identical results to when humans carried out the process.
The UK Government’s Incubator for AI (i.AI) said that Consult has been used on a live consultation by the Scottish Government seeking views on how to regulate non-surgical cosmetic procedures such as injecting lip fillers and hair removal by laser.
It reviewed over 2,000 responses to six qualitative questions using generative AI and sorted them into key themes it identified, which were checked and refined by experts in the Scottish Government.
As this was the first use of the tool, the experts also manually reviewed the responses and in the majority of cases agreed with the choices made by the AI. i.AI said the differences in view had a negligible impact on how themes were ranked overall.
Its evaluation – details of which have been published – gave it an F1 score for its performance of 0.76.
Theme Finder
Consult has been built on Theme Finder Analysis, another AI tool developed by i.AI, the code for which has been made available as open source on GitHub.
The Consult algorithms have been trained on past government consultations and synthetic data, using the Open AI GPT4 system on Microsoft Azure. It is used in a process that keeps humans ‘in the loop’ through experts reviewing the themes identified and how responses are sorted into them through an interactive dashboard that allows them to filter and search for insights.
It is part of the Humphrey bundle of AI tools under development to support the work of civil servants.
The Department for Science, Innovation and Technology (DSIT) – the parent department of i.AI – said that Consult is still in trial but there are plans to roll it out across the Civil Service.
It said this could save officials from around 75,000 days of analysis across its average of 500 consultations per year. These cost around £20 million in staffing costs.
Time saver
Technology Secretary Peter Kyle said: “No one should be wasting time on something AI can do quicker and better, let alone wasting millions of taxpayer pounds on outsourcing such work to contractors.
“After demonstrating such promising results, Humphrey will help us cut the costs of governing and make it easier to collect and comprehensively review what experts and the public are telling us on a range of crucial issues.
“The Scottish Government has taken a bold first step. Very soon, I’ll be using Consult, within Humphrey, in my own department and others in Whitehall will be using it too – speeding up our work to deliver the Plan for Change.”
The response to the consultation, which is informing the content of Scotland’s Non-Surgical Cosmetic Procedures Bill, is expected to be published before the end of June.
