Last updated: November 4, 2026
SGLang 0.5.23, released November 2, 2026, lets MoE models larger than GPU memory run on one consumer GPU with Paged Experts, adds an opt-in low-latency mode built on NVIDIA and AMD fused kernels, runs on Google TPU7x, serves LoRA on FP8 and NVFP4 MoE models, and supports ten new models.
Contents
- What’s new at a glance
- Paged Experts: MoE models larger than VRAM
- Low-latency mode with megakernels
- Google TPU7x support
- Faster MoE prefill and quantized MoE LoRA
- New models
- How to upgrade and try the new features
- How 0.5.23 compares with 0.5.22 and other engines
- Limitations
- Frequently asked questions
SGLang is an open-source serving engine for large language and diffusion models, built for high-throughput GPU inference. The official v0.5.23 release notes list 1,007 merged pull requests from 257 contributors. This guide covers the changes that matter most to people running models themselves.
What’s new at a glance
| Change | What it does | Why it matters |
|---|---|---|
| Paged Experts (opt-in) | Pages MoE experts between host RAM and the GPU | Run MoE models larger than VRAM on a single consumer GPU, up to 30x faster than CPU offload |
| Low-latency mode | NVIDIA megakernel and AMD monokernel plugins for DeepSeek-V4.1-Flash decode | Up to 44% faster single-user decode on GB300 and 75% on MI350X |
| Google TPU7x | SGLang-TorchTPU, a PyTorch-native TPU plugin | A new hardware target with its own Docker image |
| MoE prefill tuning | Wider FlashInfer CUTLASS MoE autotuning | First token 30% to 58% sooner on 8K prompts on H200 |
| Quantized MoE LoRA | New opt-in LoRA engine for FP8 and NVFP4 MoE models | Serve fine-tuned adapters on quantized MoE checkpoints |
| DeepSeek-V4.1 on GB300 | Prefill CUDA graph and replay improvements | Up to 1.5x prefill throughput and about 28% lower first-token latency on 32K to 64K prompts |
| HiCache L3 for MLA | File-based L3 prefix cache with decode context parallelism | Larger reusable prefix caches for DeepSeek-style models |
All figures in this guide come from the release notes and linked pull requests, measured by contributors on specific hardware.
Paged Experts: MoE models larger than VRAM
Mixture-of-experts models route each token to only a few experts per layer, and consecutive tokens reuse most of them. The Paged Experts pull request uses that pattern. It keeps K of a layer’s experts in a slot table on the GPU, holds every expert in pinned host memory, and copies in only the experts a step needs that are not already resident. The computation stays on the GPU.
Before this change, the main option was --cpu-offload-gb, which streams every offloaded weight to the GPU on every forward pass. The pull request reports 12x (int4) and 30x (FP8) faster single-stream decode on a 16 GB card, and 27x on an 8 GB laptop. It also states that results are unchanged, with bit-identical log probabilities for bf16 and FP8.
Two new server flags control it: --enable-paged-experts and --paged-experts-num-resident. K is sized automatically from the memory budget unless you set it. A smaller K leaves more memory for the KV cache and longer context; a larger K reduces paging and speeds up decoding.
Low-latency mode with megakernels
The new low-latency mode targets single-user speed. Opt-in plugins, the NVIDIA megakernel and the AMD monokernel, run DeepSeek-V4.1-Flash decode as fused persistent kernels. The release reports up to 44% faster single-user decode on GB300 and 75% on MI350X. On NVIDIA, the megakernel pull request adds the --enable-nvidia-megakernel flag. The plugin itself ships as a separate package, sgl-nvidia-mega, and SGLang refuses to launch with the flag if the plugin, GPU, model or settings do not match what the plugin supports, listing every reason.
Google TPU7x support
SGLang now runs on Google TPU7x through SGLang-TorchTPU, a PyTorch-native plugin. It uses its own image, lmsysorg/sglang:v0.1.0-tpu, rather than the main version tag, so check that repository for supported models before planning a TPU deployment.
Faster MoE prefill and quantized MoE LoRA
A MoE autotuning change extends FlashInfer CUTLASS tuning to prefill chunks. The release says MoE models such as DeepSeek-V4.1-Flash, GLM-5.3-Flash and gpt-oss reach the first token 30% to 58% sooner on 8K prompts on H200. Separately, a new opt-in LoRA engine lets you serve LoRA adapters on quantized MoE models in FP8 and NVFP4.
New models
The language and vision-language additions are Muse Spark Open, Beam-1.0, MiniMax-M3.1-Flash, MiniCPM-V 4.7 and Kev. The diffusion additions are Cosmos3-Nano-Sim (Bimanual, Transfer and Auto), FastH3 Omni, FLUX.3 Image Klein, HY-Nomas and MOSS-Image. Each has a cookbook page in the SGLang documentation. See our Muse Spark Open guide and MiniMax M3.1 Flash guide for the models themselves.
How to upgrade and try the new features
- Install with
uv pip install --prerelease=allow sglang==0.5.23, or pulllmsysorg/sglang:v0.5.23for NVIDIA CUDA 13. Since 0.5.22, SGLang requires Python 3.11 or newer. - On AMD, Intel or TPU hardware, use the matching image:
v0.5.23-rocm10-mi35x,v0.5.23-rocm10-mi30x,v0.5.23-xpu,v0.5.23-xeonorlmsysorg/sglang:v0.1.0-tpu. - To run an MoE model that does not fit in VRAM, add
--enable-paged-expertsto your launch command. Start with automatic sizing, then tune--paged-experts-num-residentif you need more context or more speed. - For single-user DeepSeek-V4.1-Flash decode on GB300, install the sgl-nvidia-mega plugin and launch with
--enable-nvidia-megakernel. Read the startup log line that confirms the megakernel is active. - Re-run your own latency and throughput benchmarks before moving production traffic, because contributor figures come from specific GPUs and models.
How 0.5.23 compares with 0.5.22 and other engines
The 0.5.22 release focused on data-center throughput: up to 2.3x more DeepSeek-V4.1 throughput per GPU on H200, a loader that made model loading 10x faster, better prefill-decode disaggregation and Apple Silicon support. Version 0.5.23 widens the audience, with Paged Experts aimed at laptops and gaming PCs and a low-latency mode for single users.
The Paged Experts pull request compares itself with llama.cpp tuned with --n-cpu-moe. The two were close when most of the model fit on the GPU. Paging decoded faster with FP8 on a 16 GB card over PCIe 5.0 (87.7 versus 45.2 tokens per second), while llama.cpp kept lower time to first token on short prompts. For other options, see our llama.cpp 0.9.0 guide, vLLM 0.32 guide and Ollama guide. The DeepSeek V4.1 Flash guide covers the model that gains most from this release.
Limitations
- Paged Experts currently works on a single GPU with bf16, fp16, block-quantized FP8 and GPTQ experts. NVFP4, AWQ, MXFP4, tensor parallelism and expert parallelism are listed as follow-ups.
- With several concurrent requests and a small K, paging can fall behind llama.cpp, according to the pull request’s own tests.
- The megakernel low-latency mode covers DeepSeek-V4.1-Flash decode on specific GPUs and refuses unsupported setups.
- The new LoRA engine and Paged Experts are opt-in, so default behavior is unchanged.
Frequently asked questions
What is new in SGLang 0.5.23?
SGLang 0.5.23, released November 2, 2026, adds opt-in Paged Experts for MoE models larger than GPU memory, a low-latency mode using NVIDIA megakernel and AMD monokernel plugins, Google TPU7x support through SGLang-TorchTPU, LoRA on FP8 and NVFP4 MoE models, faster MoE time to first token and ten new models.
What is Paged Experts in SGLang?
Paged Experts is an opt-in feature enabled with –enable-paged-experts. It keeps a set number of experts per MoE layer on the GPU, holds all experts in pinned host memory, and copies in only the experts a step needs. The release notes report it as up to 30x faster than CPU offload.
How do I install SGLang 0.5.23?
Run uv pip install –prerelease=allow sglang==0.5.23, or pull the Docker image lmsysorg/sglang:v0.5.23 for NVIDIA CUDA 13. The release also lists ROCm images for AMD MI35x and MI30x, XPU and Xeon images for Intel, and lmsysorg/sglang:v0.1.0-tpu for Google TPU7x.
Which new models does SGLang 0.5.23 support?
The release lists Muse Spark Open, Beam-1.0, MiniMax-M3.1-Flash, MiniCPM-V 4.7 and Kev as language or vision-language models, plus the diffusion models Cosmos3-Nano-Sim, FastH3 Omni, FLUX.3 Image Klein, HY-Nomas and MOSS-Image.
Does SGLang run on Google TPUs?
Yes. As of 0.5.23, SGLang runs on Google TPU7x through SGLang-TorchTPU, a PyTorch-native plugin maintained in a separate sgl-project repository, with its own Docker image tagged v0.1.0-tpu.
Is SGLang faster than vLLM or llama.cpp?
It depends on the workload. The Paged Experts pull request found SGLang decoded faster than tuned llama.cpp when many experts sit in host RAM on a fast PCIe link, while llama.cpp had lower time to first token on short prompts. Benchmark your own model and hardware.
Sources
- SGLang v0.5.23 release notes (GitHub)
- SGLang v0.5.22 release notes (GitHub)
- Paged Experts pull request #42557 (GitHub)
- NVIDIA megakernel pull request #44945 (GitHub)
- SGLang-TorchTPU repository (GitHub)
Image: SGLang logo from the SGLang repository, Apache License 2.0, copyright the SGLang project authors (resized). FreeAnswers is not affiliated with or endorsed by the SGLang project.
Related guide: ERNIE 5.5 Flash Base guide: Baidu’s open 280B-A14B MoE checkpoint.