vLLM
Open-source engine for running and serving large language models quickly and cheaply
Profile
- First released
- June 2023
- Created at
- Sky Computing Lab, UC Berkeley, USA
- Maintained by
- Hosted project of the PyTorch Foundation (part of the Linux Foundation), with a contributor community
- License
- Apache 2.0
- Type
- Open-source LLM inference and serving engine (software library)
vLLM is an open-source program for ‘serving’ large language models (LLMs). Serving means taking users' questions and having a model produce and send back answers. It was developed at the Sky Computing Lab of UC Berkeley in the United States and released on 20 June 2023. Researchers including Woosuk Kwon and Zhuohan Li took part in building it, together with Ion Stoica, a co-founder of Databricks. For two months before the release it was already in real use behind Chatbot Arena, run by the research team LMSYS, and its code is published under the Apache 2.0 licence, which lets anyone use and modify it.
Its core technique is PagedAttention. A language model keeps information about the words it has already read in a memory called the ‘KV cache’, and the developers found that existing systems wasted 60–80% of this memory. Borrowing the way operating systems manage memory in small units (pages), PagedAttention splits the KV cache into blocks and cuts the waste to under 4%. The team said this gives up to 24 times higher throughput than Hugging Face Transformers (team-reported). The research was presented at SOSP, an operating-systems conference, in 2023. In January 2025 an alpha version of the ‘V1’ engine with a redesigned core came out, and on 7 May 2025 vLLM became a hosted project of the PyTorch Foundation, part of the Linux Foundation.
In January 2026, vLLM's creators and core maintainers announced that they had founded a company called Inferact. The company said it had raised $150 million in its first round, led by Andreessen Horowitz (a16z) and Lightspeed, at a valuation of $800 million. Its CEO is Simon Mo, one of vLLM's original creators. Inferact said vLLM supports more than 500 model architectures and more than 200 types of accelerators (chips for AI calculations), with more than 2,000 contributors (company-reported). a16z said companies such as Meta, Google and Character.ai use vLLM in their real services. As of October 2026, vLLM is an Apache 2.0 project governed by the PyTorch Foundation, and that month ‘vLLM TorchTPU’, for Google's TPU chips, was released.
Sources for this profile (10) · as of Oct 31, 2026
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention · vLLM Blog
- vllm-project/vllm · GitHub (vLLM project)
- Efficient Memory Management for Large Language Model Serving with PagedAttention · arXiv
- vLLM V1: A Major Upgrade to vLLM's Core Architecture · vLLM Blog
- PyTorch Foundation Expands to Umbrella Foundation and Welcomes vLLM and DeepSpeed Projects · PyTorch Foundation
- PyTorch Foundation Welcomes vLLM as a Hosted Project · PyTorch Foundation
- Introducing Inferact: Building the Future of AI Inference · Inferact
- Inference startup Inferact lands $150M to commercialize vLLM · TechCrunch
- Investing in Inferact · Andreessen Horowitz
- Introducing vLLM TorchTPU · vLLM Blog
AIMade by AIThis profile was written by AI (Claude) from the 10 sources above (official pages, filings, news reports).How it's made
Records 9
Newest first. Open one to see it with its source.
Disaggregated serving has come down from a research claim to an operational guide. The key is using p99 latency rather than average throughput as the yardstick, since agent loops are directly exposed to tail latency. It
An inference-architecture shift in which, as long multi-turn and agent workloads become standard, the KV cache extends beyond GPU memory into the storage hierarchy.
A record, in concrete numbers, of AMD GPUs being optimized in a major open inference engine into a real alternative to NVIDIA.
As agentic coding becomes the representative workload of inference serving, the goal of benchmarks itself has shifted from one-shot throughput to the economics of long multi-turn sessions.
An attempt to handle contexts in the 1 million-token class on a single node; solutions to the memory bottleneck of long-context agents are converging on a combination of sparse attention and hierarchical memory.
Multi-node model loading and weight swapping (RL rollouts) are promoted to first-class problems of the serving stack
The second rewrite of the execution layer since the V1 redesign. Serving competition moved from kernels to schedulers and execution cores
A sign that open-source serving infrastructure, once text-only, is expanding to cover non-autoregressive and multimodal generation
Made prefix caching the default; up to 1.7× faster than V0