A measure of how long the slowest few requests take to finish.
The pizza arrives for everyone else. Yours is still “out for delivery.” That last sad wait is tail latency.
It shows whether a few users get stuck at busy times. It matters a lot for AI replies.
Continuous batching
Continuous batching can change the wait time for slow requests.
Inference engine
An inference engine must control slow replies, not only average speed.
TPS
High average TPS does not mean every request finishes quickly.