Department’s data science team is looking at the potential of the open source platform for big data processing
Data scientists at the Department for Work and Pensions (DWP) are exploring the use of the Apache Spark open source platform for processing big data.
Calvin Dudeck (pictured), head of data science innovation at the DWP, told yesterday’s Public Sector Enterprise ICT conference, staged by Whitehall Media, that the department now has a team working with the technology as part of its investigations of new uses of data, artificial intelligence and machine learning.
“We just introduced our analytics environment where we are looking at using Apache Spark to improve our customer service, to help our case workers in the Jobcentres, to help people get to the next best action,” he said.
Speaking to UKAuthority, Dudeck said a team of more than 20 people has been using the platform for about six months. The work has been largely experimental so far with no application to a specific service, but he hopes it could be valuable in the future.
“The idea behind Apache Spark is to be able to do things in a distributed way and move away from an emphasis on batches of data to a real time approach,” he said, adding that it could be a year or so before a more focused use takes shape.
“My goal is to build up the team, build up some of our capabilities and look at the analysis we can do with an Apache Spark capability,” he said.
Apache Spark is a fast engine for large scale data processing that can power different libraries including SQL and DataFrames, and its own MLlib for machine learning. It has been developed by an open source community through the Apache Software Foundation, one of the prime developers of open source technology.
AI shortcomings
Dudeck also told the conference that, while artificial intelligence has immense potential, it currently has limitations that could restrain its use for many functions. These include: a lack of the equivalent of human hands, which provide feedback to the brain with microscopic precision; an absence of creative intelligence to come up with something original, new solutions to new problems; and a lack of social intelligence.
The latter can be particularly significant in the provision of services that depend on assessing the circumstances of individuals or groups.
