AI:AM: A Level We Shouldn't Pass? Notes from The Curve + Tokens vs. Salaries & Is SaaS Cooked?

Episode Summary

Executive Summary: The episode argues that frontier AI is moving faster than many expected: capability gains, better data and synthetic data, rising memory and compute demands, and agents are already reshaping software work. Speakers describe emerging limits and governance ideas, from caps on pretraining compute to safety bottlenecks, while also warning that AI is leaking frontier capabilities through distillation, RL environments, and agent tools into the broader ecosystem.

Main Topics: Frontier AI consensus and capability acceleration: At The Curve, insiders and critics converged on the view that AI systems are becoming extremely powerful quickly, shortening timelines and shifting debate from whether AI matters to how far it will go and whether takeoff could be rapid. Potential caps on intelligence and compute: A frontier lab executive suggested there may be a level of intelligence society should not exceed, and even entertained a concrete limit on pretraining flops, raising the possibility of direct constraints on scaling. Memory wall, inference economics, and chip design: Positron co-founder Thomas Somers described sharp memory price inflation, the rise of agent-driven context demands, and chip designs optimized for commodity memory and higher bandwidth utilization. Agents transforming software engineering: Sean Wang argued that AI engineers and high-agency developers are more valuable than ever, but that organizations are now paying close attention to 'Claude slop,' agent productivity, and the limits of current SaaS and CRUD workflows. Safety, alignment, and critical infrastructure ownership: Evan Miyazono discussed coordination failures, the need for shelling points on AI risk, and potential interventions such as cryptographic attestation for agent identity and protections against accidental or malicious agent swarms. Training data, reward hacking, and new benchmark design: Edward Hugh described richer workplace-like datasets, the shift from task-centric benchmarks to role-centric environments, and how better specification and rubrics are needed to reduce reward hacking and scattershot model behavior. AI beyond verifiable tasks: Across the episode, speakers emphasized that AI is no longer limited to neat benchmark tasks; it is increasingly useful in science, chip design, music, and other domains where the quality criteria are less directly verifiable.

Key Arguments: Frontier labs are increasingly aligned that AI systems will become much more capable, making near-term decisions about compute and safety highly consequential. A senior frontier executive reportedly believes there may be an intelligence ceiling worth not crossing, and was open to operationalizing it as a cap on pretraining flops. Synthetic data, data augmentation, and test-time compute can be recycled back into pretraining, creating a recursive improvement loop. Frontier models may already possess research taste, but current RL and elicitation methods do not fully surface it. The biggest competition is concentrated among the top frontier companies; others are partly keeping up via distillation and RL environment tooling. AI adoption is shifting the bottleneck from model capability to memory capacity, context length, and the number of concurrent agents. Agentic software development increases demand for high-agency engineers, while mediocre vibe-coded output is becoming easier to reject. Current SaaS products are vulnerable because agents can now request changes quickly through internal coding agents, reducing tolerance for slow vendor roadmaps. Safety and alignment work may consume a growing share of compute, especially through monitoring and environment hardening. Better RL environments reduce downstream cheating, suggesting a proto-scaling law between environment quality and model behavior. Industry training may shift away from expensive RL-only pipelines toward supervised fine-tuning, distillation, and multi-teacher aggregation. AI is increasingly useful in areas once thought difficult to verify, including scientific discovery, kernel optimization, formal verification, and music composition.

Data Points: Memory price change: 4.5x increase in a year and a week - Thomas Somers said a memory quote had risen dramatically over the past year. Memory price outlook: Possible additional doubling over the next year - Somers said he would not be surprised by another major increase. Positron realized memory bandwidth: 93% of theoretical bandwidth - First-generation product achieved high utilization on decode forward passes. NVIDIA GPU memory bandwidth utilization: 30% to 40% - Somers contrasted typical realized bandwidth with theoretical specs. NVIDIA B300 theoretical bandwidth: 8 TB/s - Used as a benchmark for discussing actual realized memory bandwidth. NVIDIA B300 memory capacity: 288 GB - Compared against Positron's capacity-per-chip claims. NVIDIA Rubin Ultra memory capacity: 192 GB - Analyst-reported next-gen capacity referenced in the interview. Positron capacity per chip: 2.3 TB - Azimov chip capacity described by Somers. Memory channel count: 72 LPDDR5X channels - Positron described a chiplet solution scaling beyond common designs. Current agents running in background: 15 to 20 - Somers said his concurrent agent usage had grown quickly. Token spend peak: Over $100,000 per day - Positron's spending during heavy experimentation. Frontier model training benchmark: About 70 seconds reduced by almost half - NanoGPT Speedrun record improvement cited in the episode. Design/verification ratio at Positron: 60% design / 40% verification - Somers compared his startup's ratio to NVIDIA's mature process. Gate-level emulation speedup: ~500 kHz vs ~10 Hz - Agent-assisted chip verification dramatically accelerated hardware testing. Apex Agents benchmark change: Apex 1.1 release - Mercor revised the benchmark to improve task specification and penalize hacking behavior.

Pivotal Quotes: "there likely is, let's say, a level of intelligence that we just shouldn't go past" — Frontier lab executive (as reported by host): The Curve discussion on whether society should impose hard limits on AI capability "I don't want to pay for someone else to go through LLM psychosis" — Sean Wang (Swix): On hiring, productivity, and rejecting low-quality AI-generated work "as intelligence gets cheap, it's the coordination that gets expensive" — Evan Miyazono: On why AI governance may require new institutions and coordination mechanisms

Implications: The episode suggests AI is entering a phase where compute governance, safety engineering, and agent productivity matter as much as raw model capability. Companies and regulators may need to act sooner, while software, chips, and scientific work all reorganize around agentic systems.