A scheduling method for adding new requests to a batch already running.
Continuous batching is a diner grill at lunch rush. New burgers slide on while the cook flips the first batch.
You meet it in online LLM inference. It keeps the GPU busy. Answers can arrive faster.
Inference
Continuous batching is used during inference to run requests together more efficiently.
GPU
Continuous batching cuts GPU waiting time and keeps compute packed tighter.
Token
LLMs produce tokens step by step, so new requests can join during generation.
LLM
Continuous batching is common in live LLM services, especially with many users at once.