#120 – François Chollet: Measures of Intelligence

Episode Summary

Executive Summary: François Cholet argues that intelligence is not the possession of skills, but the efficiency of acquiring new skills in truly novel situations. He critiques benchmark culture, the Turing test, and pure scaling narratives in deep learning, and presents ARC as a more rigorous way to measure generalization using explicit priors, core knowledge, and unseen tasks.

Main Topics: Defining intelligence as skill acquisition efficiency (Priority: 5/5): Cholet distinguishes intelligence from skills or memorized outputs, defining it as how efficiently a system learns new tasks it was not prepared for and adapts to genuinely novel situations. Language, memory, and cognition (Priority: 4/5): He frames language as an operating system for the mind: a tool for querying and programming memory, while cognition itself precedes language and also relies on visual, spatial, and topological representations. Deep learning, GPT-3, and the limits of scaling (Priority: 5/5): Cholet sees large language models as powerful pattern generators that improve with scale, but argues they remain constrained by training data, plausibility, and weak guarantees of factual consistency or true reasoning. Why ARC was created (Priority: 5/5): The Abstraction and Reasoning Corpus is presented as a benchmark designed to measure fluid intelligence by requiring novel problem solving built on explicit core knowledge priors, with no external world knowledge. Priors, psychometrics, and human intelligence (Priority: 4/5): He draws on developmental psychology and psychometrics to explain innate priors, the G-factor, and the need to control for priors and experience when comparing human and machine intelligence. Generalization and the taxonomy of AI capability (Priority: 5/5): Cholet proposes a spectrum from robustness to flexibility to extreme generalization, emphasizing that real intelligence means adapting to unknown unknowns and developer-unforeseen cases. Meaning, culture, and impact (Priority: 3/5): The conversation closes with a broader philosophical view: meaning comes from creating cultural ripples that influence future minds, human and artificial alike.

Key Arguments: Intelligence should be measured by the efficiency of acquiring new skills in novel environments, not by static skill performance or benchmark scores. Language is important but not fundamental; it is a layer on top of cognition and acts like an operating system and memory-query interface. Deep learning provides strong perception and pattern completion, but current models do not reliably support explicit reasoning, consistency, or factual grounding. GPT-style models may imitate reasoning through massive pattern matching, yet scaling alone does not solve novelty, long-tail edge cases, or out-of-distribution adaptation. A valid intelligence benchmark must make priors explicit, separate priors from experience, and prevent the creators from baking in solutions. ARC is meant to force human-like abstraction over a tiny set of core knowledge priors while excluding language and world knowledge. Psychometrics is most useful at scale because correlations across tasks reveal latent cognitive structure such as the G-factor. The Turing test is philosophically interesting but scientifically weak because it outsources measurement to biased human judges and rewards deception over understanding. Human intelligence is highly general relative to biology, but not universal; it is constrained by evolved priors and the human condition. Cognition is not compression itself; compression is only one tool in the cognitive toolkit, while intelligence must preserve flexibility for the uncertain future.

Data Points: ARC test set performance at launch: 0% - Cholet says machine performance on ARC started at zero, while humans found the tasks easy. ARC current state-of-the-art: ~20% - He notes ARC had improved to around 20% test-set solved at the time of the conversation. GPT-3 parameters: 175 billion - Used as an example of scale in large language models. Human brain synapses vs GPT-3: ~1,000x more - He gives a rough comparison that the human brain has about a thousand times more synapses than GPT-3 has parameters, while noting the systems are very different. Wikipedia / culture scaling: External cognition at civilization scale - He argues that tools like Wikipedia and Google search extend human intelligence beyond individual brains. Image model training set: 350 million labeled images - He recalls working on a Google image classification model trained on a very large noisy dataset. Noisy label source: Social tags / page keywords - The image labels were derived from weak signals rather than manual human annotation. Autonomous driving training data example: 30 million road situations - He cites a paper suggesting this still was insufficient for self-driving generalization. Brain processing timescale: ~50 milliseconds to seconds - He uses this to argue that brain input/output bandwidth is not the main bottleneck for augmentation. Physical money share of currency: ~8% physical, 92% digital - Mentioned during the Cash App sponsor read, illustrating digitization of money. Masterclass annual price: $180/year - Sponsor mention for the all-access course platform.

Pivotal Quotes: "the measure of intelligence is the ability to change." — François Cholet / quoting Einstein: Used to support the idea that intelligence is about adaptation to new situations. "I see language as the operating system of the brain, of the human mind." — François Cholet: Explaining language as a layer on top of cognition and a mechanism for memory access. "the efficiency with which you acquire new skills, tasks that you did not previously know about, that you did not prepare for." — François Cholet: His core definition of intelligence.

Implications: For AI, progress should be judged by novel task learning, not benchmark imitation or text fluency alone. ARC-style tests and explicit priors may better reveal real general intelligence and guide systems toward robust adaptation.