The team working on a chatbot powered by generative AI for the GOV.UK platform has reported encouraging results as it extended the test service to more users.
This marks progress in the development of GOV.UK Chat – reported by UKAuthority in November of last year – by the team within the Government Digital Service. The service uses a large language model (LLM) to find information within GOV.UK.
After stages involving the use of the chatbot among a small number of people internally, it was scaled up to around 1,000 users in central government within a live private pilot for the next phase.
A newly published blogpost says, that when using the bot to search for information, nearly 70% found the responses were useful and just under 65% were satisfied with the experience.
Insights for improvements
The pilot also provided insights on room for improvement, notably that sometimes a query could not be answered because the GOV.UK page was too long. This has prompted work to “chunk up” the content without losing important context before it is passed to the LLM.
In addition, the team identified a need to redirect users if their questions could not be answered using GOV.UK.
Another finding was that some users underestimated or dismissed the risks of any inaccuracies because of the credibility of the GOV.UK brand, and that some have a misplaced confidence on a system that could sometimes be wrong. This has prompted work to ensure users understand the relevant risks.
“These findings validate why we’re taking a balanced, measured and data driven approach to this technology — we’re not moving fast and breaking things,” according to the blog.
“We’re innovating and learning while maintaining GOV.UK’s reputation as a highly trusted (according to internal polling) information source and a ubiquitously recognised symbol in the UK.”
More experiments
In another blogpost, Chris Bellamy, director of GOV.UK, has said that upcoming experiments will involve evaluating ideas from users and various government departments.
“We are continuing to develop the accuracy of the chat while partnering with other government organisations, aiming to establish rapid feedback loops to assess and iterate going forward,” he said.
