The Data Science Unit of the Welsh Government has developed a proof-of-concept tool that uses generative AI in editing text.
It has carried out the project to support content designers in the Welsh Government by providing better drafts for them to work from, and to help a wider range of officials in producing better content.
A member of the unit named Joe said the current version of the tool can: improve readability by using shorter and simpler words and sentences; improve the accessibility of content by following Easy Read guidance from the UK Association for Accessible Formats; edit the structure of content to make it easy to understand and provide a logical flow, with headings, bullet points and shorter descriptions; and provide consistency to style guidance across a range of platforms and formats.
It uses the Flesch Kincaid Grade level readability formula, and so far three large language models (LLMs) have been tested and the unit is experimenting with others in a test environment.
Content produced with the tool is also being assessed manually on whether it has been grouped into sections for a coherent flow, and whether all the relevant information has been included.
Successful tasks
“Manual reviews have shown that the model has successfully performed some of the tasks with which it has been prompted, like removing certain expressions,” Joe said.
“One example of this is changing ‘carbon monoxide is a silent killer’ to ‘carbon monoxide is a dangerous gas that can’t be seen or smelled. It can be deadly’.”
He added that the project has discovered some limitations with the LLMs, such as sometimes missing the logical glow of content and difficulties in following the Welsh Government style guidance.
“It demonstrates how important it is for content designers to oversee the content edited by AI and be involved in the design process,” he said.
The Data Science Unit is now aiming to deploy the proof-of-concept with some updates in response to the feedback it has had so far.
