The difference between this approach and its predecessors is that DeepMind hopes to use “dialogue in the long term for safety,” says Geoffrey Irving, a safety researcher at DeepMind.
“That means we don’t expect that the problems that we face in these models—either misinformation or stereotypes or whatever—are obvious at first glance, and we want to talk through them in detail. And that means between machines and humans as well,” he says.
DeepMind’s idea of using human preferences to optimize how an AI model learns is not new, says Sara Hooker, who leads Cohere for AI, a nonprofit AI research lab.
“But the improvements are convincing and show clear benefits to human-guided optimization of dialogue agents in a large-language-model setting,” says Hooker.
Douwe Kiela, a researcher at AI startup Hugging Face, says Sparrow is “a nice next step that follows a general trend in AI, where we are more seriously trying to improve the safety aspects of large-language-model deployments.”
But there is much work to be done before these conversational AI models can be deployed in the wild.
Sparrow still makes mistakes. The model sometimes goes off topic or makes up random answers. Determined participants were also able to make the model break rules 8% of the time. (This is still an improvement over older models: DeepMind’s previous models broke rules three times more often than Sparrow.)
“For areas where human harm can be high if an agent answers, such as providing medical and financial advice, this may still feel to many like an unacceptably high failure rate,” Hooker says.The work is also built around an English-language model, “whereas we live in a world where technology has to safely and responsibly serve many different languages,” she adds.
And Kiela points out another problem: “Relying on Google for information-seeking leads to unknown biases that are hard to uncover, given that everything is closed source.”