A benchmark for testing language model knowledge with multiple-choice questions across many subjects.
MMLU is one huge exam day for AI. Math sits by law. Medicine brings a clipboard.
It appears on leaderboards to check school knowledge and expert knowledge. A high score does not mean real job skills.
LLM
MMLU measures an LLM’s knowledge across many subjects.
Leaderboard
Leaderboards often use MMLU scores to compare models.
Benchmark contamination
If test questions leak into training, MMLU scores can look too high.
QA
MMLU mostly uses multiple-choice QA questions.