🔬 Automating Science: World Models, Scientific Taste, Agent Loops — Andrew White

Episode Summary

Executive Summary: Andrew White discusses his path from molecular dynamics and biomaterials research into founding Future House and Edison Scientific to automate science. He argues that AI’s biggest near-term impact is not replacing all scientists but accelerating hypothesis generation, literature/data analysis, and lab-in-the-loop discovery. The conversation centers on Cosmos, scientific taste, provenance, and why natural language and verifiers matter more than brute-force simulation.

Main Topics: From academia to AI-for-science entrepreneurship: White traces his transition from a tenure-track professor to co-founding Future House and Edison Scientific, driven by the belief that automating scientific discovery is both feasible and too important to do slowly inside academia. What ‘automating science’ actually means: He defines the goal as automating the discovery loop: generating hypotheses, choosing experiments, analyzing results, and updating world models—rather than only building domain-specific predictors. Cosmos, taste, and verification: The discussion explains how Cosmos evolved from separate tools into a workflow that uses literature search, data analysis, and human feedback to rank hypotheses, revealing that scientific ‘taste’ is a key bottleneck. Why simulation is overrated for many scientific problems: White criticizes MD and DFT as highly expensive, heavily parameterized methods that often simulate simplified systems well but fail on real-world complexity; he contrasts this with ML successes like AlphaFold. Natural language as the connective tissue of science: He argues that language is the best current representation for linking papers, code, data, and ideas across scientific domains, though multimodal and mathematical representations still matter. Dual-use risk and AI safety in science: The conversation addresses CBRN concerns, concluding that AI has not dramatically lowered the barrier for most misuse cases, but may accelerate logistics, protocol execution, and other second-order risks. The future role of scientists: White suggests scientists will increasingly act as wranglers, interpreters, and taste-makers for AI systems, with human involvement still needed to judge what counts as good science.

Key Arguments: The core bottleneck in science is increasingly not idea generation alone, but verification, interpretation, and access to experimental or analytical constraints. Scientific taste is hard to model directly; systems work better when they optimize for downstream consequences such as experiment outcomes, literature consistency, or data-analysis success. Human preference-based RLHF on hypotheses was less informative than expected; downstream verifiers and real data are better signals than subjective rankings alone. MD and DFT are powerful but often overused and overtrusted; they tend to reproduce limited classes of systems rather than yield robust, de novo discovery for messy real-world problems. AlphaFold demonstrated that some problems assumed to require massive custom infrastructure can be solved with ML on ordinary hardware, overturning assumptions about compute-heavy first-principles approaches. Provenance matters: scientific agents should cite sources, show code lineage, and make their reasoning traceable so humans can audit the result. Natural language remains the best integration layer because it can connect papers, code, data, and expert intuition across scientific domains. AI is more likely to augment than eliminate scientific labor in the near term, because science has effectively unlimited demand and still needs human judgment to decide what is meaningful. Safety risks exist, but current evidence suggests AI mostly accelerates known workflows and logistics rather than dramatically lowering the expertise needed for harmful CBRN work.

Data Points: PhD lab composition: 19 experimental researchers and 2 simulation researchers - White described his PhD group at the University of Washington Tenure resignation timing: June (recently) - He said he resigned his tenured position in June after a long sabbatical Scientific agent benchmark accuracy: 60%–70% correctness - He said Bixbench is reaching this range and human disagreement appears at a similar level Human agreement on data analysis: ~70% - White cited that humans only agree about 70% of the time on some analyses Human agreement on interpretation: ~50%–55% - He described Cosmos results where interpretation of analysis had only about half-level agreement Timeframe of major progress: 2023–2025 - He said the last two years produced enormous progress in AI-for-science Protein folding training budget: ~10,000 GPU hours - He estimated this as the scale to train a good protein-folding model now Computation for water simulation: ~1,000,000 CPU hours - He described a large DFT/quantum water simulation project as one of the biggest of his career Compute share anecdote: ~20% of the world’s computing power - He estimated that, pre-ChatGPT, this went to simulating water Lab-loop experiment duration: 4 weeks - He said the Robin experiment involved roughly four weeks of wet-lab testing Breadth of hypothesis evaluation: 100 runs - He described running an agent many times and taking consensus-style analyses to reduce uncertainty Forecast horizon for the original mission: 10-year mission - He referred to Future House’s early framing of automating science as a 10-year goal Red-team access to GPT-4: ~9 months before public release - White said he used GPT-4 long before release as a red teamer Conference anecdote about MD: Special-purpose machines could fold only 1–2 proteins/day - He contrasted old assumptions about protein folding with AlphaFold’s desktop accessibility

Pivotal Quotes: "The point of this podcast is to bring together AI engineers and scientists." — R.J. Haneke: Opening framing of the show’s mission "We decided to automate science as the goal." — Andrew White: Explaining the shift from tools and agents to a broader mission at Future House "The frontier right now is scientific taste." — Andrew White: Summarizing where current AI-for-science systems are still weakest

Implications: AI-for-science is shifting from simulation-heavy, domain-specific tooling toward traceable, verifier-driven discovery systems. Scientists may increasingly supervise AI workflows, but human judgment, taste, and provenance remain central to turning outputs into real knowledge.