ReLU — Rectified Linear Unit

Fact

A neural network activation function that turns negative numbers into zero.

In Plain Words

ReLU is a bouncer at Club Neural Net. Negative numbers get sent home. Positive numbers walk in.

You meet it inside the middle layers of neural networks. It helps training move faster, especially in CNNs.

Related Concepts

• Neural Network
• Multilayer Perceptron
• Convolutional Neural Network
• Backpropagation

Neural-network
ReLU often works as an activation switch inside a neural network.

MLP
An MLP often uses ReLU between layers.

CNN
CNNs use ReLU a lot to make training faster.

Backpropagation
During Backpropagation, the learning signal passes back through ReLU.