| Documentation | Paper | Developer Slack |
--- ## About the Llumnix V1 Upgrage (Mar. 2026) We have refactored and upgraded Llumnix to Llumnix v1, a new architecture designed to be more modular and cloud-native, and better suited for production environments. We will keep iterating Llumnix v1 in the future. See the [new repository](https://github.com/llumnix-project/llumnix) for details! This repository, i.e., the Ray-based architecture (Llumnix v0), is a better choice for local deployments and quick prototyping and experimentation of scheduling ideas. --- ## 🔥 Latest News - [2025.5] We created a developer [slack](https://llumnix.slack.com) for discussion and collaboration. - [2025.1] We updated vLLM to version v0.6.3.post1. - [2024.11] Llumnix v0.1.0 launched! - [2024.7] We officially released the first version of Llumnix. - [2024.6] We released our OSDI '24 [research paper](https://arxiv.org/abs/2406.03243) on arxiv. ## 🚀 Why Llumnix Llumnix is a cross-instance request scheduling layer built on top of LLM inference engines such as [vLLM](https://github.com/vllm-project/vllm). Llumnix provides optimized multi-instance serving performance in terms of: - *Low latency* - **Reduced time-to-first-token** (TTFT) and queuing delays with less memory fragmentation - **Reduced time-between-tokens** (TBT) and preemption stalls with better load balancing - *High throughput* - Integration with state-of-the-art inference engines - Support for techniques like prefill-decode disaggregation Llumnix achieves this with: - Dynamic, fine-grained, KV-cache-aware scheduling - Continuous **rescheduling** across instances - Enabled by a KV cache migration mechanism with near-zero overhead - Exploited for continuous load balancing, de-fragmentation, and prefill-decode disaggregation Llumnix is easy to use with: - Minimal code changes required for vanilla vLLM deployments - Seamless integration with existing multi-instance deployment platforms - Fault tolerance, elasticity, and high service availability - Extensibility to more inference engines and scheduling policies ## Getting Started Llumnix provides two entrypoints `api_server` and `serve` for deploying Llumnix. The `api_server` entrypoint provides a compatible deployment method with the default single-instance vLLM. By contrast, using the `serve` entrypoint, user can easily deploy Llumnix via Ray job submission API. If you are already utilizing vLLM for multi-instance LLM serving deployments, simply replace the vLLM serving deployment command `python -m entrypoints.vllm.api_server ...` for each instance with the command provided below: ``` python -m llumnix.entrypoints.vllm.api_server \ --host $HOST \ --port $PORT \ ... ``` During the serving deployment execution, Llumnix will automatically configure itself and serve as the request scheduling layer on top of the multiple vLLM engine instances. For deploying Llumnix using the `serve` module, please refer to [QuickStart](./docs/Quickstart.md). Visit our [documentation](./docs/) to get started: - [Quick Start](./docs/Quickstart.md) - [Supported Models](./docs/Supported_Models.md) - [Fault Tolerance](./docs/Fault_Tolerance.md) - [Simulator](./docs/Simulator.md) - [Prefill-decode Disaggregation](./docs/Prefill-decode_Disaggregation.md) ## Performance We evaluate the performance of the KV-cache-aware load-balancing scheduler and migration mechanism of Llumnix with 16 Qwen2.5-7B instances (each using an A10-24GB GPU) and 16 Llama2-13B instances (each using an A800-80GB GPU). We use Poisson distributions with different request rates to generate request arrivals. For the input/output lengths of requests, we use ShareGPT dataset.