llama.cpp 0.9.0: FP8, Multi-Node and New Models Guide

Last updated: November 4, 2026

llama.cpp 0.9.0, released November 4, 2026, adds an FP8 E4M3 tensor type, a faster attention path for running many sequences on a shared KV cache, NCCL-based tensor parallelism across machines, four new model architectures, Chatterbox TTS, Nemotron 3.5 Transcribe and predefined Pocket TTS voices.

Contents

What’s new at a glance
How llama.cpp versions work now
FP8 E4M3 support
Faster multi-sequence inference
Multi-node tensor parallelism
New models and audio support
How to upgrade and try the new features
How 0.9.0 compares with 0.8.0 and other runtimes
Limitations
Frequently asked questions

llama.cpp is the open-source C/C++ inference engine behind many local AI tools. Version 0.9.0 arrives one week after 0.8.0. This guide covers what changed, based on the official v0.9.0 release notes and the linked pull requests.

What’s new at a glance

Change	What it does	Why it matters
FP8 E4M3 tensor type	OCP FP8 E4M3 in saturating mode, with conversion helpers and a new file type	Lower-precision weights in a format widely used for inference
flash_attn_ext_rows	Each query slice attends only to its own list of KV rows	Unified KV cache stays fast with many parallel sequences
Multi-node tensor parallelism	NCCL-based collectives in the CUDA backend, used by the RPC server	Split one model across several machines
New architectures	Clover-35B-A3B-Preview, Limite 1B Violetto, Metis-1.6, Mistral Large 4	More models run without conversion workarounds
Audio	Chatterbox TTS, Nemotron 3.5 Transcribe, predefined Pocket TTS voices	Local speech generation and transcription
Gemma 4 E2B and E4B	Graph optimizations described as major prefill gains	Faster prompt processing on small Gemma models
ggml 0.29.0	Bundled core library update with backend performance work	CPU, CUDA, SYCL, Metal, Vulkan, OpenCL and Hexagon improvements

How llama.cpp versions work now

For most of its history, llama.cpp shipped only build numbers such as b12110. The ggml-org releases and versioning notes now describe two tracks. Nightly builds from the master branch use b-number tags. Official releases use tags such as v0.9.0 and are created only when llama.cpp’s internal copy of ggml is identical to a released ggml version, here ggml v0.29.0.

That rule lets Homebrew link llama.cpp to a system-wide ggml package. The llama.app installer still publishes nightly builds, and the v0.9.0 notes name b12110 as the matching nightly.

FP8 E4M3 support

The FP8 pull request adds the Open Compute Project FP8 E4M3 format to ggml in saturating mode. Its author notes that E4M3 is the FP8 variant most often used for inference, while E5M2 is more common in training. The public API gains LLAMA_FTYPE_MOSTLY_F8_E4M3, which reports as the F8_E4M3 file type.

The pull request lists KV cache, CUDA and Vulkan support as separate follow-up work, so check backend support before relying on FP8.

Faster multi-sequence inference

A unified KV cache lets parallel requests share one memory pool but used to slow prompt processing. The new flash_attn_ext_rows operation gives each sequence a compact list of the KV rows it needs. In the pull request benchmarks on an RTX Pro 6000, four parallel prompts of about 100,000 tokens processed at 8,626 tokens per second with Gemma 4 26B-A4B, up from 2,982, and at 2,778 tokens per second with Qwen3.8-27B, up from 1,106.

Multi-node tensor parallelism

ggml 0.29.0 adds a communicator interface for ranks running in separate processes. The NCCL pull request implements it in the CUDA backend and uses it in the RPC server. On two DGX Spark nodes the author reports Qwen3.8-27B (Q4_K_M) generation rising from 13.02 tokens per second on one node to 23.51 across both. The test command used -sm tensor, --rpc NODE-1-IP:50052,NODE-2-IP:50052 and -dev RPC0/RPC1.

New models and audio support

The four new architectures are Clover-35B-A3B-Preview, a diffusion model run in autoregressive mode; Limite 1B Violetto; Metis-1.6, a hybrid Mamba-2 and attention design in Praxis (about 2.9B) and Logos (about 10.2B) sizes; and Mistral Large 4. LFM3 models gain NVFP4 GGUFs and tensor split, and Qwen3-VL video prompts are now handled accurately.

On the audio side, the Nemotron 3.5 Transcribe pull request adds offline transcription in 40 languages with prompt context and speaker attribution. Word timestamps and streaming are out of scope. Chatterbox TTS joins the speech models, and Pocket TTS can now use predefined voices without voice cloning.

How to upgrade and try the new features

Install or update with the README script: curl -LsSf https://llama.app/install.sh | sh on macOS or Linux, or irm https://llama.app/install.ps1 | iex in PowerShell. For the exact v0.9.0 tag, download binaries from the GitHub release or build from source.
Start a server from a Hugging Face GGUF repository with llama serve -hf <user>/<repo>. Version 0.9.0 fixes -hf tag matching inside longer quantization names.
For a TTS model, call GET /props to read the voices list, then send POST /tts with input and a voice from that list. A predefined voice cannot be combined with a speaker reference.
For Nemotron 3.5 Transcribe on the server, add --reasoning off. The pull request notes that the model’s template enables thinking by default, which otherwise returns reasoning and no transcript.
If you call /v1/responses, you can now pass sampling options such as seed and min_p; they carry through to the converted chat request.
To test multi-node inference, run the RPC server on each CUDA machine and launch with -sm tensor and an --rpc host list, as in the pull request.

The server documentation lists the full /tts options, including response_format, seed and streaming. Our Hugging Face hf CLI guide covers downloading GGUF files.

How 0.9.0 compares with 0.8.0 and other runtimes

The 0.8.0 release on October 28, 2026, was larger on features: the LFM3 family, a /v1/realtime endpoint following the OpenAI realtime protocol, a POST /tts endpoint, selectable reasoning effort levels, and Hugging Face model browsing in the web UI. Version 0.9.0 focuses on performance and scale. Our LFM3 guide and Nemotron 3.5 guide cover the models these releases support.

If you want a desktop app or a simpler model manager on top of a local engine, see our Ollama October 2026 guide and LM Studio Bionic guide. For high-throughput GPU serving, compare with our vLLM 0.32 guide. For the newly supported Mistral model, see the Mistral Large 4 guide.

Limitations

FP8 E4M3 arrives as a core type, with more backend work described as pending in the stacked pull requests.
Multi-node tensor parallelism depends on CUDA with NCCL; other backends were not part of that change.
Nemotron 3.5 Transcribe has no word timestamps or streaming yet, and its author called Chatterbox support complicated and a work in progress when the pull request opened.
Performance figures come from contributor tests on specific hardware, so results on your setup will differ.

Frequently asked questions

What is new in llama.cpp 0.9.0?

llama.cpp 0.9.0, released November 4, 2026, adds an OCP FP8 E4M3 tensor type, a flash_attn_ext_rows operation that speeds up the unified KV cache for multi-sequence inference, NCCL-based multi-node tensor parallelism through ggml 0.29.0, four new architectures, Chatterbox TTS, Nemotron 3.5 Transcribe and predefined Pocket TTS voices.

What is the latest llama.cpp version?

As of November 4, 2026, the latest official release is v0.9.0, published the same day. The project also publishes nightly builds tagged b plus a commit number, such as b12110, which is the nightly build that matches v0.9.0.

What is the difference between llama.cpp v0.x releases and b-numbered builds?

Official releases use semantic version tags such as v0.9.0 and are only cut when llama.cpp’s internal ggml copy matches a released ggml version, so they can link against a system-wide ggml. Nightly builds use b-number tags and come straight from the master branch.

Which new models does llama.cpp 0.9.0 support?

The release notes list four new architectures: Clover-35B-A3B-Preview (a diffusion model run in autoregressive mode), Limite 1B Violetto, Metis-1.6 (Praxis at about 2.9B and Logos at about 10.2B parameters) and Mistral Large 4. It also adds Chatterbox TTS and Nemotron 3.5 Transcribe on the audio side.

How do I update llama.cpp to 0.9.0?

The project README offers an install script from llama.app for macOS and Linux and a PowerShell command for Windows, plus Docker images, prebuilt binaries on the GitHub releases page, and building from source.

Does llama.cpp 0.9.0 change the API?

Yes, in small ways. llama.h adds LLAMA_FTYPE_MOSTLY_F8_E4M3, mtmd.h adds Chatterbox and predefined-voice functions, GET /props now reports a model’s predefined TTS voices, and /v1/responses now passes extra parameters such as seed and min_p through to the converted chat request.

Sources

Image: llama.cpp banner from the llama.cpp repository, MIT License, copyright the ggml authors (resized). FreeAnswers is not affiliated with or endorsed by the ggml-org project.

llama.cpp 0.9.0: Power-User Guide to FP8, Multi-Node Inference and New Audio Models