Temporarily moving some model data to other hardware when memory is too small.
Offloading is like cooking a holiday dinner on a tiny kitchen counter. You stash pans in the hallway, then keep cooking.
It helps run a big model when VRAM is too small. The trade-off is speed. Data has to travel back and forth.
VRAM
Offloading moves some model data elsewhere when VRAM is too small.
Local-LLM
Offloading can help a local device run a model it could not fit before.
Quantization
Quantization cuts size, and Offloading moves data to another place.
Inference
Offloading often happens during inference and trades speed for the chance to run.