Can AI safeguard us against AI? One of its Canadian pioneers thinks so
LawZero will ‘look for scientific solutions to how we can design AI that will not turn against us'
When Yoshua Bengio first began his work developing artificial intelligence, he didn't worry about the sci-fi-esque possibilities of them becoming self-aware and acting to preserve their existence.
That was, until ChatGPT came out.
"And then it kind of blew [up] in my face that we were on track to build machines that would be eventually smarter than us, and that we didn't know how to control them," Bengio, a pioneering AI researcher and computer science professor at the Université de Montréal, told As It Happens host Nil Köksal.
The world's most cited AI researcher is launching a new research non-profit organization called LawZero to "look for scientific solutions to how we can design AI that will not turn against us."
"We need to figure this out as soon as possible before we get to machines that are dangerous on their own or with humans behind [them]," he said. "Currently, the forces of market — the competition between companies, between countries — is such that there's not enough research to try to find solutions."
Meet LawZero's conception: Scientist AI
Bengio started LawZero using $40 million of donor funding. Its name references science fiction writer Isaac Asimov's Three Laws of Robotics, a set of guidelines outlining the ethical behaviour of robots that prevents them from harming or opposing humans.
In Asimov's 1985 novel Robots and Empire, the author introduced the Zeroth Law: "A robot cannot cause harm to mankind or, by inaction, allow mankind to come to harm."
With this in mind, Bengio said LawZero's goal is to protect people.
"Our mission is really to work towards AI that is aligned with the flourishing of humanity," he said.
Several AI technologies in recent months have been reported to undermine, deceive, and even manipulate people.
For example, a study earlier this year found that some AIs will refuse to admit defeat after a chess match, and instead hack the computer to cheat the results. AI firm Anthropic detailed last month that during a systems test, its AI tool Claude Opus 4 tried to blackmail the engineer so that it would not be replaced by a newer update.
These are the kind of scenarios that drove Bengio to design LawZero's guardian artificial intelligence, Scientist AI.
According to a proposal by Bengio and his colleagues, Scientist AI is a "safe" and "trustworthy" artificial intelligence that would function as a gatekeeper and protective system for humans to continue to benefit from this technology's innovation with intentional safety.
It's also "non-agentic," which Bengio and his colleagues define as having "no built-in situational awareness and no persistent goals that can drive actions or long-term plans."
In other words, what differentiates agentic and non-agentic AI is their autonomous capacities to act in the world.
How would Scientist AI work? Can it work?
Scientist AI, Bengio says, would be paired with other AIs, and act as a kind of "guardrail." It would estimate the "probability that an [AI]'s actions will lead to harm," he told U.K. newspaper, the Guardian.
If that chance is above a certain threshold, Scientist AI will reject its counterpart's suggested action.
But can we guarantee that this guardian AI will also not turn against us?
David Duvenaud, an AI safety researcher who will act as an adviser for LawZero, says it's a rational concern.
"If you're skeptical about our ability to control AI with other AI, or really be sure that they're going to be acting in our best interest in the long run, you are absolutely right to be worried," Duvenaud, an assistant professor of computer science and statistics at the University of Toronto, told CBC.
Still, he says, we have to try.
"I think Yoshua's plan is less reckless than everyone else's plan," he said.
AI researcher Jeff Clune, also an adviser on the project, agrees.
"There are many research challenges we need to solve in order to make AI safe. The important thing is that we are trying, including allocating significant resources to this critical issue," Clune, a University of British Columbia computer scientist, said in an email. "That is one reason the creation of LawZero is so important."
According to Bengio's announcement for LawZero, "the Scientist AI is trained to understand, explain and predict, like a selfless idealized and platonic scientist."
Resembling the work of a psychologist, Scientist AI "tries to understand us, including what can harm us. The psychologist can study a sociopath without acting like one."
Bengio says he hopes this widespread reckoning on the rapid, yet alarming, evolution of AI will catalyze a political movement to start "putting pressure on governments" worldwide to regulate it.
"I often get the question of whether I'm optimistic or pessimistic," he said. "What I say is that it doesn't really matter. What matters is what each of us can do to move the needle towards a better world."
Interview with Yoshua Bengio produced by Kate Swoger
