vLLM
vLLM is an open-source high-throughput inference and serving engine for deploying large language and multimodal AI models efficiently on GPUs, CPUs, and other accelerator hardware.
vLLM is an open-source AI inference and model-serving framework designed for AI developers, machine learning engineers, infrastructure teams, researchers, and organizations that want to deploy large language models with high throughput and efficient hardware utilization. Originally developed at UC Berkeley’s Sky Computing Lab, it has evolved into a community-driven project focused primarily on production inference rather than providing an end-user chatbot or hosted AI service.
The engine uses technologies such as PagedAttention, continuous batching, optimized scheduling, prefix caching, speculative decoding, chunked prefill, and optimized CUDA kernels to improve throughput and memory efficiency. It also supports several quantization approaches including GPTQ, AWQ, INT4, INT8, and FP8, allowing organizations to reduce model memory requirements depending on their hardware and model.
vLLM integrates closely with Hugging Face and currently supports more than 200 model architectures, including decoder-only LLMs, Mixture-of-Experts models, hybrid architectures, multimodal models, embedding models, rerankers, reward models, and classification models. It can expose deployed models through OpenAI-compatible APIs as well as an Anthropic Messages-compatible API and gRPC interfaces. Features include streaming, structured outputs, tool calling, reasoning parsers, and efficient LoRA adapter serving.
For larger deployments, vLLM supports tensor, pipeline, data, expert, and context parallelism across multiple accelerators and nodes. Hardware support includes NVIDIA and AMD GPUs, x86, ARM, and PowerPC CPUs, plus accelerator integrations for Google TPU, Intel Gaudi, Huawei Ascend, Apple Silicon, and other platforms. Deployment options include Python, Docker, Kubernetes through Helm, and distributed environments.
vLLM itself does not have a subscription pricing model. The project is distributed under the Apache 2.0 license, while users are responsible for the cost of GPUs, servers, cloud infrastructure, storage, and the models they deploy.