The amount of data memory can send to a chip each second.
A GPU is like a race car. Memory bandwidth is the fuel pipe. If the pipe is narrow, the car cannot go fast.
It often limits inference speed. It matters a lot when reading model weights and making each new token.
VRAM
VRAM is the storage room. Memory bandwidth decides how fast data leaves it.
Inference
Inference does not depend on compute alone. Memory bandwidth often sets the pace.
TPS
When bandwidth is tight, TPS hits a ceiling fast.
Flash Attention
Flash Attention reduces memory traffic to get around bandwidth limits.