A way to split one big computing job across many machines that work together.
Distributed computing is a school cafeteria at lunch rush. One cook would melt, so the whole crew saves the pizza line.
It splits a huge job across many computers. You meet it in big AI training. It also powers giant data work and busy online services.
Data Parallelism
Data Parallelism is the most common way to split training work in distributed computing.
Model parallelism
Model parallel shares a model across devices when one machine cannot fit it.
GPU
Distributed computing often makes many GPUs work as one team.
AI data center
Large distributed computing usually runs inside an AI data center.