vLLM
High-throughput inference server for open models
- Open source
- Infrastructure
What it is
vLLM is an inference engine whose PagedAttention memory manager treats the key-value cache like virtual memory pages, letting it hold many concurrent sequences without fragmenting GPU memory. Continuous batching keeps the GPU busy across requests of very different lengths, and the server speaks an OpenAI-compatible API. Tensor and pipeline parallelism cover multi-GPU serving.
Best for
Self-hosting open-weight models where concurrent throughput per GPU is the number you are optimising.
Where it falls short
It assumes you have GPUs and the appetite to operate them — capacity planning, drivers and upgrades are entirely yours.
Characteristics
- inference-server
- paged-attention
- continuous-batching
- gpu
This page carries no score, star rating or review count, and nothing about its placement in the directory was paid for. The outbound links above go to the product’s own domain with no referral parameters. Read the directory methodology for what that means in practice.
More in Infrastructure
Other tools solving the same problem, so you can see what vLLM is actually competing with.
Ollama
Open sourceRun open-weight models locally
Modal
FreemiumServerless GPU compute defined in Python
Replicate
PaidRun and fine-tune models behind an API