A speed trick: a small model drafts text, then a big model quickly checks it.
SpecDecode is like a drive-thru with a speedy rookie. The rookie lines up likely orders, and the manager just says yes or no.
It speeds up answers in online AI services. It makes words appear faster, not smarter.
Inference
SpecDecode happens during inference and aims to make text generation faster.
SLM
An SLM often drafts the next tokens before the bigger model checks them.
LLM
SpecDecode reduces the LLM’s token-by-token work.
TPS
SpecDecode often shows up as higher TPS.