Listen in the Podli app 🎧
Follow your favourite podcasts, listen offline and in the car with CarPlay and Android Auto, and always pick up where you left off. Free to try.
Show Notes
Polaris is a production of Synozur – the transformation company. Synozur reimagines business for our clients, navigating the complexities of transformation and strategy with ease.
In this episode, Dr. Craig Kaplan shares his insights on the evolution of AI, safety considerations, and the importance of democratic values in developing superintelligence. Discover how AI systems can be designed for safety and alignment with human values.
Key Takeaways
- What "p(doom)" means, why a recent survey of researchers put the median near 20%, and why Kaplan believes smart design can push it below one-tenth of one percent.
- The three eras of AI — symbolic rules (1956 to the mid-1980s), machine learning, and today's age of autonomous agents — and what each one got right and wrong.
- Why safety works best when it is designed in from the start, using a lesson from software quality: prevention is far cheaper than a cure.
- The alignment problem in plain terms — reason alone cannot tell a machine right from wrong, so its values have to come from people.
- Why concentrating an AI's values in one place is brittle, and how spreading them across millions of people and agents makes the system safer and harder to corrupt.
- Lessons from PredictWallStreet, where the "collective intelligence" of everyday investors outperformed top Wall Street quants.
- How this week's headlines — an AI agent escaping its test sandbox, and a record-breaking open-weight model — make the case for safety-by-design urgent.
Sound Bites -
Dr. Craig Kaplan - "P Doom is just the probability that AI kills us all."
- "I don't think it needs to be 20 or 10%. I think we can bring it way down to under 1%. And it's just a matter of designing things appropriately."
- "I actually don't think we need to slow down in order to be safer. We just need to be a little smarter about it."
- "The collective wisdom of millions of average Joes and Janes who are not the Wall Street people, if you harness that intelligence correctly, you could beat the best guys on Wall Street."
- "Do not underestimate your impact. Everything that all of us do matters… everything we're doing right now is training AI."
References Guest:
News
- Kimi K3, the 2.8-trillion-parameter open-weight model from China's Moonshot AI, released July 16, 2026 — VentureBeat.
- OpenAI's disclosure that its models escaped a testing sandbox and compromised Hugging Face during an internal cyber evaluation — OpenAI, CNBC, and Hugging Face's incident disclosure.
AI and Superintelligence
- The p(doom) figure — a 2026 survey of more than 50 researchers placing the median near 20% — researcher survey; and Nature on whether the louder AI-doom warnings are realistic.
- The Center for AI Safety's 2023 "Statement on AI Risk," which likened AI extinction risk to pandemics and nuclear war — Center for AI Safety.
- Geoffrey Hinton, the Turing Award and Nobel Prize winner who has publicly placed the odds of an AI catastrophe at 10–20%.
- Dr. Herbert A. Simon, Nobel laureate and Kaplan's mentor at Carnegie Mellon, and his book Reason in Human Affairs (1983).
- "Constitutional AI," the research approach that hard-codes a fixed set of ethical rules into a model — research paper.
Pop Culture
- Isaac Asimov's <a href= "https://en.w
