The Evolution of Benchmarks for AI Systems

The benchmarks by which we are measuring what AI systems do and how they do it are extremely narrow. These are effectively, in most cases, kind of tinker toy assessments that allowed academic models to compare themselves to each other. We have slipped into a pattern of being comfortable making claims about intelligence or capability," he says.