The Government Digital Service (GDS) is working on a project using generative AI with a chatbot to improve the user experience of the GOV.UK platform.
Programme lead Robin Pembrooke outlined the initiative at this week’s UKAuthority Automation & Bots4Good conference, saying it is part of an effort to develop new forms of interaction with GOV.UK to reduce uneven access across the country.
“One of the ways we’re thinking about doing that is how can use generative AI and large language models (LLMs) to improve user experience and to run the platform better, to make it easier to publish content or manage the quality of that content,” he said.
“We think there’s huge potential in this technology and it will have major impact on how people use GOV.UK, but we also know new technology comes with significant risks.”
The chatbot has been developed with an LLM to provide answers to questions using only content published the website – which includes around 800,000 pages – and without using any personal information.
It has been tested internally with variables to manage the tone, brevity and nature of interactions between the user and chatbot, with rules to prevent it answering in any unwanted languages, and in a style that reflects that of GOV.UK.
Time saver
“The current version does and quickly summarises the answer to what can be quite complex questions,” Pembrooke said. “Without using the tool it might mean users will have to visit several pages.”
A second phase involved controlled, lab based testing with 10 internal business users to assess the quality of answers provided through the model. This has now been scaled up to 1,000 invited users, with a set of live dashboards providing the development team with indications of how they rate the quality of answers.
Pembrooke said the results of this are scheduled to be published next month, but that generally users found it easier and quicker to use than existing solutions.
He added, however, that more work is needed on issues around trust and the quality and reliability of answers.
“A known issue with LLMs that consistency and inaccuracy a major issue; these models do make things up,” he said. “We found that by focusing on our own controlled dataset we could improve the accuracy.
“You would need very high levels of accuracy before a service like this could go into a full, unmoderated customer service. But by doing these experiments we are learning how to improve quality and with future experiments we’ll improve it further.”
He said that GDS is setting up a steering group for future work on LLMS with the Central Digital and Data Office, the Department of Business and Trade and the 10DS data science team, and has been reaching out to other government organisations.
It is also aiming to build an LLM sandbox to experiment with different models and cloud providers.
