Automating Scientific Discovery, with Andrew White, Head of Science at Future House

Episode Summary

Executive Summary: Andrew White argues that automating science will remain fundamentally empirical: models can improve hit rates and planning, but only lab measurements resolve biological and chemical complexity. He traces Future House’s mission to build semi-autonomous AI systems for science, and details Paper QA and Aviary as tools to make literature search, protocol reasoning, and training of scientific agents more effective.

Main Topics: Andrew White’s path into AI for science: White explains his background in statistical mechanics, his 2019 sabbatical exposure to ML in physical sciences, early work on language models for chemistry, and his involvement in OpenAI’s GPT-4 red teaming, which shaped his view of AI’s power and limitations. Why biology is empirically driven: White argues that biology’s multi-scale complexity, noisy measurements, and long feedback loops make it resistant to pure in silico replacement; AI can help, but wet-lab experimentation remains the ultimate arbiter. Future House’s mission and funding model: He describes Future House as a mission-focused, Eric Schmidt-backed organization inspired by focused research organizations (FROs), aiming to accelerate discovery through semi-autonomous scientific AI rather than full lab automation. Paper QA: literature intelligence at scale: Paper QA uses retrieval, full-text search, contextual summarization, and LLM filtering to answer questions across scientific literature, detect contradictions, and generate citation-backed summaries with superhuman performance on several tasks. Aviary: agents, environments, and training: Aviary formalizes scientific tasks as trainable agent-environment systems, distinguishes memory as part of the agent, and enables end-to-end training even when parts of the system use black-box commercial models. Limits of foundation models in biology and chemistry: White is skeptical that multi-omic foundation models or purely generative biology models will be silver bullets; he sees them as useful tools, not substitutes for experimental feedback loops and domain-specific workflows. Product, infrastructure, and future directions: He discusses cost/latency tradeoffs, figure understanding, commercialization tensions, and the need for better web access, better memory, and more diverse hypothesis generation for future scientific systems.

Key Arguments: Biology cannot be reduced to simple cartoons or first-principles in silico models; complexity appears at every level, so empirical measurement is indispensable. AI mainly raises hit rates and reduces experimental iterations; it does not eliminate the need for lab validation. Future House’s strategy is to build focused, mission-driven systems for one core problem: automating science, not general robotics or fully autonomous labs. Paper QA worked because it spent compute liberally, used full-text search, and inserted an intermediate relevance/summarization step that removed distracting information before final answering. Aviary’s agent-environment split is practical for training: anything trainable belongs in the agent, while tools and interfaces belong in the environment. Black-box gradient estimation is possible via surrogate modeling over sampled completions, but it is more a proof of concept than a broadly transformative optimization method. Biology and drug discovery have long latency between idea and feedback, making it hard to know whether a platform approach is truly better until many years later. The most important bottlenecks in drug discovery may be operational ones like patient enrollment and access to data, not just better predictive models. Current foundation-model approaches in biology are promising for annotation and metagenomics, but unlikely to be the dominant path to deeper disease understanding. The internet is becoming less machine-accessible due to anti-bot measures, which could hinder future AI scientific workflows unless new access mechanisms emerge.

Data Points: Future House funding scale: $20 million to $50 million - White describes FRO-style organizations as being funded at this scale over a few years. Future House time horizon: 5 years (moonshot, not guaranteed) - He says the project is not a straightforward five-year goal but a moonshot that may require more funding. Paper QA runtime cost: $0.15 to $1.00 per question - He gives the approximate cost for using the hasanyone.com/Paper QA system depending on question complexity. Paper QA runtime latency: about 3 to 4 minutes - He notes the system spends time on multiple searches and processing to maximize answer quality. Protein gene ordering cost: about $25 for a 100-amino-acid protein gene - He uses this to illustrate how cheap some biological hypothesis testing has become. Free energy calculation cost: $10 to $15 - He compares cloud compute cost for binding calculations to wet-lab experiment costs. Equivalent experiment cost: about $20, potentially down to $5 - He estimates the cost of making/testing organic molecules against proteins. Virtual molecule catalogs: 50 to 100 billion molecules - He says searchable catalogs of synthesizable molecules now reach tens of billions. ChemCrow result error: ~15 nanometers off - The designed molecule’s measured absorption was close to the target wavelength. WikiCrow articles generated: ~18,000 total articles - He says Future House generated tens of thousands of Wikipedia-style articles on human genes/diseases, with some genes lacking literature. Human genome article coverage added: ~16,000 articles added - He explains that they extended coverage far beyond what Wikipedia previously had. Contradiction detection throughput: ~75 per minute - He cites high-throughput contradiction checking across literature. LabBench / O1 performance: above human level - He says OpenAI’s report showed O1 exceeded human performance on a laboratory protocol QA benchmark from Future House. CQA improvement: 50 to 70 - He reports a roughly 20-point gain on a cloning-related benchmark. Aviary completion branching: K = 8 - He describes generating multiple completions and selecting among them with a Q model. Tree-search branching: branching factor of 2 - He says they duplicate the environment during search and often use smaller branching factors to control explosion. Clinical trial enrollment: ~60 days to enroll first patient - He highlights this as a major bottleneck in drug discovery innovation.

Pivotal Quotes: "You will never be able to reduce biology to like these cartoon diagrams." — Andrew White: He is arguing that biological systems are too multi-scale and context-dependent for purely reductionist modeling. "The only thing that does work 100% of the time is measuring it in the lab." — Andrew White: He emphasizes that experimental validation remains the final source of truth in biology and chemistry. "I think it's a domain which is driven by observations and empirical measurements rather than a domain that's driven by like some sort of virtual in silico model that you can derive." — Andrew White: He contrasts empirical science with the idea that an AI system could infer cures purely from reasoning.

Implications: AI for science is likely to look like high-throughput, semi-autonomous experimentation support rather than fully autonomous discovery. Near-term wins will come from better retrieval, planning, and prioritization, not replacing the lab or human judgment.