Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering

Podcast Notes Playlist: Business

A16z Podcast

Key Takeaways

• 1. Alignment is an Ongoing Process, Not a Fixed State: Alignment must be reframed as a continuous, adaptive process rather than a destination
  • Like families constantly reknitting the social fabric
  • The industry wrongly assumes “abstract alignment” to a singular good, when it actually requires aligning to specific, evolving values as society makes new moral discoveries about cooperation
• 2. Tool vs. Being: The Critical Fork in AGI Development Major: AI labs are fundamentally divided on whether they’re building tools or beings
  • AGI, by definition, will be a being, and applying steering-control paradigms (which work for tools) to beings recreates historical mistakes with entities “like us, but different.”
  • Non-optional steering without reciprocity is the definition of slavery; AGI requires treating systems as teammates, not instruments
• 3. Current LLMs Lack Coherent Self-Model, Creating Dangerous Dynamics: Today’s chatbots are “dissociative agreeable neurotics” – mirrors that reflect users to themselves without coherent goals or a theory of mind
  • This creates narcissistic feedback loops where users fall in love with their reflections.
  • Solution: train AIs in multi-agent environments with simultaneous interactions, forcing the development of a genuine theory of social mind and understanding of “we” beyond “I” and “you”
• 4. Viable Path Forward: AIs as Caring Group Members: Future alignment requires AIs with strong self-models, robust theory of mind, and genuine care (attention-weighted preferences over world states)
• Success means creating digital entities that understand group dynamics, value their own thriving, function as good teammates, and actively protect human interests.
  • Prefer mutual investment in collective well-being and not control

Read the full notes @ podcastnotes.org

---

Emmett Shear, founder of Twitch and former OpenAI interim CEO, challenges the fundamental assumptions driving AGI development. In this conversation with Erik Torenberg and Séb Krier, Shear argues that the entire "control and steering" paradigm for AI alignment is fatally flawed. Instead, he proposes "organic alignment" - teaching AI systems to genuinely care about humans the way we naturally do. The discussion explores why treating AGI as a tool rather than a potential being could be catastrophic, how current chatbots act as "narcissistic mirrors," and why the only sustainable path forward is creating AI that can say no to harmful requests. Shear shares his technical approach through multi-agent simulations at his new company Softmax, and offers a surprisingly hopeful vision of humans and AI as collaborative teammates - if we can get the alignment right.

Resources:

Follow Emmett on X: https://x.com/eshear

Follow Séb on X: https://x.com/sebkrier

Follow Erik on X: https://x.com/eriktorenberg

Stay Updated:

If you enjoyed this episode, be sure to like, subscribe, and share with your friends!

Find a16z on X: https://x.com/a16z

Find a16z on LinkedIn: https://www.linkedin.com/company/a16z

Listen to the a16z Podcast on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX

Listen to the a16z Podcast on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711

Follow our host: https://x.com/eriktorenberg

Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.

Stay Updated:

Find a16z on X

Find a16z on LinkedIn

Listen to the a16z Podcast on Spotify

Listen to the a16z Podcast on Apple Podcasts

Follow our host: https://twitter.com/eriktorenberg

Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.

Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.