VLLM — vLLM

Fact

An open-source tool for running and serving large language models.

In Plain Words

vLLM runs a GPU like a busy hotel. Guests take any open room. Nobody blocks a whole floor.

It helps one graphics card handle more chat requests with less memory. Developers use it to run AI chat services.

Related Concepts

• LLM inference engine
• PagedAttention
• Continuous batching
• Application Programming Interface

Inference engine
vLLM is an open-source inference engine for serving large language models.

PagedAttention
vLLM uses PagedAttention to manage model memory more efficiently.

Continuous batching
vLLM uses continuous batching to keep the GPU busy with requests.

API
vLLM can run a server that works like the OpenAI API.