A way to store model numbers with lower precision so the model is smaller.
Quantization is like packing for a trip with a tiny carry-on. You keep the outfits, but ditch the giant shampoo.
It stores model numbers in a smaller form. This saves GPU memory and helps big models run on your laptop.
Parameter
Quantization compresses parameters from high precision to lower precision.
GPU
Quantization cuts GPU memory use, so the same GPU has an easier job.
Local-LLM
Quantization is a common way to shrink a model for local use.
Inference
Quantization is mostly used during inference to save cost and gain speed.