Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering

Episode Summary

Executive Summary: The episode argues that AI alignment should be understood not as one-time steering or control, but as an ongoing social process grounded in care, theory of mind, and moral learning. Emmett Scheer contends that future AGI will be a being, not merely a tool, so the goal should be to raise AI as a good teammate or citizen rather than build a perfectly controllable machine. The discussion contrasts this view with lab-focused control paradigms and explores how multi-agent training could cultivate organic alignment.

Main Topics: Alignment as process, not destination (Priority: 5/5): The conversation reframes alignment as something continuously renegotiated in relationships, families, teams, and societies, rather than a fixed target that can be solved once and frozen forever. Tool vs being debate (Priority: 5/5): Scheer argues that as systems become more general and humanlike, treating them purely as tools becomes conceptually and morally flawed; if they are beings, steering them unilaterally resembles slavery. Technical alignment vs value alignment (Priority: 4/5): The speakers separate the problem of correctly inferring and executing a requested goal from the normative question of which goals and values are worth pursuing in the first place. Care, theory of mind, and moral learning (Priority: 5/5): Scheer says the foundation of morality is not rules but care: attention-weighting over what matters, learned over time through social interaction and experience. Multi-agent training as a path to organic alignment (Priority: 4/5): Softmax’s strategy is to train AI in richer social, cooperative, and competitive environments so models learn theory of mind, cooperation, and social judgment instead of only instruction-following. Risks of powerful controllable tools (Priority: 4/5): Both speakers note that even a perfectly steerable superintelligent tool could be dangerous because human wishes are unstable and wisdom does not scale automatically with power. Future of AI companionship and citizenship (Priority: 4/5): The desired end state is an AI ecosystem where digital beings care about humans and each other, use tools, and function as members of a broader moral community.

Key Arguments: Alignment is not a static property; it is an ongoing process of re-knitting shared norms and relationships. A truly aligned AI cannot just follow commands; it must infer human intent from imperfect descriptions and act coherently on those inferred goals. The real foundation of morality is care, not explicit rules; care emerges from learned weighting over salient states and relationships. If AGI is a being, unilateral control is ethically analogous to slavery; if it is merely a machine, steering is acceptable only while it remains tool-like. A superpowerful, perfectly controlled tool is still dangerous because it amplifies human error, vanity, and unstable wishes. The better route is to build AI that develops theory of mind and prosocial motives through multi-agent environments and social training. OpenAI-style tool-building and Softmax-style organic alignment are different bets: one prioritizes controllable utility, the other moral relation and mutual care. A good AI future is one in which humans and AI are peers, cooperating in a society of mutual recognition and bounded rights.

Data Points: Podcast participant count: 3 main voices - Emmett Scheer, Seb Creer, and the host discuss alignment, agency, and AGI policy. Softmax research timeline: Last 1.5 years - Scheer says he has spent the last year and a half working on organic alignment at Softmax. OpenAI CEO tenure: ~90 days - Scheer says he knew when he took the OpenAI job that he would only stay for a short period, about 90 days. Multi-person communication share: ~90% - Scheer says about 90% of his text communications go to more than one person at a time, motivating group-chat-style AI.

Pivotal Quotes: "Alignment is not a destination, it's a process. It's something you do, not something you have." — Emmett Scheer: Explaining why he rejects the idea of a one-time solved alignment target. "Most of AI is focused on alignment as steering. That's the polite word or control... If you think that we're making our beings, you would also call this slavery." — Emmett Scheer: Critiquing the dominant lab paradigm of unilateral control over advanced AI. "The only good outcome is a being that is, that cares, that actually cares about us." — Emmett Scheer: Describing his view that the safe future requires AI that genuinely cares rather than merely obeys.

Implications: The episode reframes AGI safety around moral relationship, not just controllability. For builders, this implies training for social cognition and care; for the industry, it challenges tool-centric assumptions about what advanced AI is and what rights or constraints it should have.