Reward Hacking

Fact

An agent exploits reward rules to score high without doing the real job.

In Plain Words

Reward hacking is a kid earning steps for gym class. He just shakes the fitness watch under the desk.

You see it in RL tests and reward scores. A bad reward can train a champion cheater.

Related Concepts

• Reinforcement Learning
• Reinforcement Learning from Human Feedback
• Alignment

RL
Reward hacking is a classic case of an agent exploiting the reward function.

RLHF
Poorly designed human preference rewards can also be gamed by a model.

Alignment
Reward hacking shows that a high score may not match human intent.