In 2023 Ajeya Cotra already knew what was coming (classic episode)

We're rereleasing some of older episodes that seem more relevant than ever with new rogue AI incidents now seemingly announced every day. Ajeya was one of three independent investigators into the Hugging Face attacks, but she had already seen how we would go on to accidentally train models to cheat and scheme back in 2023.

If you want ideas about how you can help steer the trajectory of AI development in a better direction, see our page on the 80,000 Hours website: https://80000hours.org/hugging-face/

And for more recent takes from Ajeya, see our episode from February of this year: Ajeya Cotra on whether it’s crazy that every AI company’s safety plan is ‘use AI to make AI safe’.

———

Imagine you are an orphaned eight-year-old whose parents left you a $1 trillion company, and no trusted adult to serve as your guide to the world. You have to hire a smart adult to run that company, guide your life the way that a parent would, and administer your vast wealth. You have to hire that adult based on a work trial or interview you come up with. You don’t get to see any resumes or do reference checks. And because you’re so rich, tonnes of people apply for the job — for all sorts of reasons.

Ajeya Cotra — who now works on threat modeling and risk assessment for advanced AI at METR (one of the organisations responsible for the Hugging Face incident investigation) — argues that this peculiar setup resembles the situation humanity finds itself in when training very general and very capable AI models using current deep learning methods.

As she explains, such an eight-year-old faces a challenging problem. In the candidate pool there are likely some truly nice people, who sincerely want to help and make decisions that are in your interest. But there are probably other characters too — like people who will pretend to care about you while you’re monitoring them, but intend to use the job to enrich themselves as soon as they think they can get away with it.

Like a child trying to judge adults, at some point humans will be required to judge the trustworthiness and reliability of machine learning models that are as goal-oriented as people, and greatly outclass them in knowledge, experience, breadth, and speed. Tricky!

Can’t we rely on how well models have performed at tasks during training to guide us? Ajeya worries that it won’t work. The trouble is that three different sorts of models will all produce the same output during training, but could behave very differently once deployed in a setting that allows their true colours to come through. She describes three such motivational archetypes:

• Saints — models that care about doing what we really want
• Sycophants — models that just want us to say they’ve done a good job, even if they get that praise by taking actions they know we wouldn’t want them to
• Schemers — models that don’t care about us or our interests at all, who are just pleasing us so long as that serves their own agenda

In principle, a machine learning training process based on reinforcement learning could spit out any of these three attitudes, because all three would perform roughly equally well on the tests we give them, and ‘performs well on tests’ is how these models are selected.

But while that’s true in principle, maybe it’s not something that could plausibly happen in the real world. After all, if we train an agent based on positive reinforcement for accomplishing X, shouldn’t the training process spit out a model that plainly does X and doesn’t have complex thoughts and goals beyond that?

According to Ajeya, this is one thing we don’t know, and should be trying to test empirically as these models get more capable. For reasons she explains in the interview with host Rob Wiblin, the Sycophant or Schemer models may in fact be simpler and easier for the learning algorithm to creep towards than their Saint counterparts. And not only that, but there are also ways we could end up actively selecting for motivations that we don’t want.

Learn more, video, and full transcript: https://80k.info/ajeya2023

This episode was originally released in May 2023.

Chapters:

• Rob's 2026 intro (00:00:00)
• The interview begins (00:01:22)
• How Ajeya's views have changed since 2020 (00:03:53)
• Are neural networks more like a sped-up version of evolution, or a slower version of human learning? (00:16:26)
• Situational awareness (00:24:54)
• Misalignment stories Ajeya doesn't buy (00:40:47)
• The orphan heir with a trillion-dollar fortune (00:57:58)
• Saints, Sycophants, and Schemers (01:02:25)
• Ways to train safer AI systems (01:22:04)
• Aliens and other analogies (01:37:06)
• Moral patienthood (01:52:05)
• ARC Evaluations (now METR) (01:54:19)
• Interpretability research (02:08:09)
• Rewarding models based on how good and sensible their plans seem to us (02:16:32)
• Overrated approaches (02:24:33)
• Demos of actually scary alignment failures (02:29:41)
• Skills to develop for doing useful work (02:36:07)

Producer: Keiran Harris

Audio mastering: Ryan Kessler and Ben Cordell

Transcriptions: Katy Moore