COMPLEXITY Nature of Intelligence, Ep. 5: How do we assess intelligence?
145 snips
Nov 20, 2024 
In this discussion, Erica Cartmill, a cognitive science professor, and Ellie Pavlick, a computer science and linguistics expert, dive into the intricate nature of assessing intelligence. They critique traditional IQ tests while exploring historical perspectives on intelligence. The duo highlights communication differences between humans and animals, shedding light on social cognition. They also discuss the challenges in evaluating large language models, questioning conventional assessments and redefining what it means to understand intelligence across species.
AI Snips
Chapters
Books
Transcript
Episode notes
Animal Language Abilities
- Past research focused on teaching human language to animals like apes and parrots.
- While they could learn and use words, they didn't form complex sentences, suggesting limitations in syntax acquisition.
Evaluating LLMs
- Current LLM evaluations rely on convenient but shallow metrics like repurposed human tests (SAT, MCAT).
- These tests do not accurately reflect the depth of human intelligence and are inappropriate for measuring AI.
LLM Test Performance vs. Human
- Human test performance reflects both underlying ability and test-taking strategies.
- LLMs likely learn test-taking patterns rather than demonstrating genuine understanding, making comparisons flawed.
