Language, Cognition, and the Limits of LLMs - with Tal Linzen (NYU/Google)

We host Tal Linzen, Associate Professor at NYU and Research Scientist at Google, for a conversation on the intersection of cognitive science and large language models.

We discussed why children can learn language from around 100 million words while LLMs need trillions, and the surprising finding that as models get better at predicting the next word, they become worse models of how humans actually process language. Tal walked us through how his lab uses eye-tracking and reading-time data to compare model behavior to human behavior, and what that reveals about prediction, working memory, and the limits of current architectures.

We also got into nature versus nurture and how inductive biases can be instilled by pre-training on synthetic languages, world models and whether transformers actually use the geometric structure they encode, the BabyLM challenge and data-efficient language learning, and what mechanistic interpretability can offer cognitive science beyond just fixing model bugs. The conversation closed on academia versus industry, the role of PhDs in the current AI moment, and how AI coding tools are changing the way Tal teaches and evaluates students at NYU.

Timeline

• 00:13 — Intro and what cognitive science means
• 02:16 — Using computational simulations to understand how humans learn language
• 05:26 — How children learn language vs. how LLMs are pre-trained
• 07:53 — Why mainstream LLMs are not good models of humans
• 10:07 — Comparing humans and models with eye-tracking and reading behavior
• 13:52 — Sensory modalities, smell, and how much you can learn from language alone
• 16:03 — Animal cognition and decoding animal communication
• 17:00 — Nature vs. nurture, inductive biases, and what transformers can and can't learn
• 21:21 — Instilling inductive biases through synthetic languages
• 27:34 — The bouba/kiki effect and cross-linguistic sound symbolism
• 28:33 — Latent causal structure in language and whether models discover it
• 31:13 — Does knowing linguistics help build better models?
• 35:07 — World models: what they mean, and why transformers encode geometry but don't use it
• 39:13 — Tokenization, and why Tal doesn't like it
• 41:35 — Scaling laws and the inverse-U curve of model quality vs. human fit
• 44:34 — Where the human–model mismatch comes from: architecture, memory, and data
• 47:08 — Diffusion language models and sentence planning
• 48:21 — Data quality, synthetic data, and curriculum effects
• 50:54 — Comparing models at different training stages to human development; BabyLM
• 54:40 — What level of the model should we actually probe? Representations vs. behavior
• 1:01:04 — Mechanistic interpretability, Deep Dream, and human dreaming
• 1:02:11 — Cognitive neuroscience, intracranial recordings, and working memory
• 1:10:31 — Should you still do a PhD in 2026?
• 1:12:31 — Will software engineers lose their jobs to AI?
• 1:17:43 — Teaching in the age of coding agents: what changes in the classroom
• 1:20:54 — What's next: human-like LLMs as user simulators, and recruiting

  Music:

  • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
  • "Palms Down" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
  • Changes: trimmed

  About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.