A benchmark that tests AI code with programming tasks and unit tests.
HumanEval is a coding job tryout for AI. The résumé may flex, but the unit tests press Run.
People use its pass rate to compare code models. But public questions can leak into training data.
Software Engineering Benchmark
HumanEval is an early benchmark for code ability.
Codex
The Codex paper made HumanEval popular for testing code generation.
Benchmark contamination
Public problems can enter training data and raise scores.
Leaderboard
Its pass rate is often used to rank code models.