Data Parallelism

Fact

A way to train copies of one model on many devices at once.

In Plain Words

Data parallelism is like a cafeteria line with the same lasagna recipe. Each cook handles a different tray, then they compare notes before the next batch.

In AI training, each GPU gets a copy of the model and different data. It speeds up big model training, but syncing updates costs time.

Related Concepts

• Distributed Computing
• Model parallelism
• GPU Compute
• Stochastic Gradient Descent

Distributed Computing
Data parallelism is one of the most common ways to do distributed training.

Model parallelism
Data parallel splits the data, while model parallel splits the model.

GPU
Data parallelism usually copies the same model onto many GPUs.

SGD
Each device computes gradients first, then they combine them for one update.