The London Office of Technology and Innovation (LOTI) has begun a research project on the role of ‘humans in the loop’ in deployments of AI in local government.
It is based on the widely held view that human judgement should remain part of any sensitive processes that are automated by AI systems, and is aimed at providing practical advice and guidance for authorities designing services that use the technology.
A LOTI blogpost on the issue says there is a trend for councils to designate an official as the human in the loop for a process to evaluate and review the output of an AI system, and being ready to correct its decisions.
But there are questions around what constitutes meaningful human involvement and how effective the relevant person is mitigating the ethical risks of AI.
LOTI has highlighted critical issues including: the accountability of the person involved; whether they have the skills, time and capacity within their organisation to make good decisions; whether there are technical factors such as a lack of transparency that undermine oversight of the systems; where the relevant legal liabilities lay; and the danger of the human oversight being undermined by human flaws.
There are also issues around humans being accountable for their mistakes in a way that cannot apply to AI models, and whether people prefer other people rather than machines making decisions that affect them.
Three questions
These have prompted three initial questions for the research: How effective can humans be at mitigating the risks of AI, automation and algorithms? What are the relevant critical dimensions to understand if a human in the loop does constitute a meaningful human involvement? How should local authorities design and account for humans in the loop?
The project will include workshops and interviews with local government and external experts, desk research and examining case studies, with the aim of producing a report with recommendations and practical resources.
“Our enquiry is to fundamentally better understand the dimensions of how humans and machines should co-exist in our organisations,” LOTI says.
“I want to understand what those conditions are: what are the roles that humans should be playing or will naturally exist in, what are the skills that different people will need, how do we manage these people, who do we hold accountable in situations where different types of decisions are made with AI or automation or algorithms in conjunction with humans?”
