Nick Bostrom joins Theo Jaffee and Sofia Puccini on MTS to revisit the ideas about superintelligence and existential risk that he helped bring into the mainstream, now that increasingly capable AI systems actually exist. Nick discusses why reinforcement learning may be pushing models toward more goal-seeking behavior, what recent agent experiments tell us about instrumental convergence, and why he favors preserving the ability to slow frontier development rather than committing now to a broad or indefinite AI pause. He also makes the case that the risks of AI need to be weighed alongside its potential upside, from advances in medicine and longer lifespans to entirely new ways of being human. They also get into whether today's models could already have some form of consciousness, the tradeoffs of open-source AI, whether machines can do original philosophy, mind uploading, and what the simulation hypothesis might mean for the superintelligence we create.
Nick Bostrom joins Theo Jaffee and Sofia Puccini on MTS to revisit the ideas about superintelligence and existential risk that he helped bring into the mainstream, now that increasingly capable AI systems actually exist.
Nick discusses why reinforcement learning may be pushing models toward more goal-seeking behavior, what recent agent experiments tell us about instrumental convergence, and why he favors preserving the ability to slow frontier development rather than committing now to a broad or indefinite AI pause. He also makes the case that the risks of AI need to be weighed alongside its potential upside, from advances in medicine and longer lifespans to entirely new ways of being human.
They also get into whether today's models could already have some form of consciousness, the tradeoffs of open-source AI, whether machines can do original philosophy, mind uploading, and what the simulation hypothesis might mean for the superintelligence we create.
Resources:
Nick Bostrom / Macrostrategy Research Initiative: https://nickbostrom.com/
Follow Theo Jaffee on X: https://x.com/theojaffee
Follow Sofia Puccini on LinkedIn: https://www.linkedin.com/in/sofiapuccini/