AI Benchmark

An AI benchmark is a standardized evaluation — typically a curated dataset of tasks with known correct answers — used to measure how well an AI model performs on a particular capability and to compare models on a common yardstick. Benchmarks let researchers and practitioners answer questions like "which model is best at coding?" or "how much did reasoning improve between versions?" with reproducible numbers rather than anecdotes.

Common benchmarks target different skills: MMLU measures broad academic knowledge, HumanEval and SWE-bench measure coding ability, GSM8K and MATH measure mathematical reasoning, and GPQA tests graduate-level science. Because static benchmarks can leak into training data and become 'saturated', the field increasingly relies on harder, contamination-resistant tests and on human-preference evaluations like Chatbot Arena, where users vote on head-to-head model responses. No single benchmark captures real-world usefulness, so models are best judged on a portfolio of evaluations plus task-specific testing.