# llm-d 1.0, and what it means if you run one or two GPUs *llm-d is free, open-source software that sits in front of many running copies of a language model and decides which copy answers each request. Its 1.0 release came out on 2026-10-20. This page says what it does, what 1.0 adds and who builds it, then which of its ideas carry down to one or two graphics cards at home: the two that matter most already live inside vLLM, and one needs a second card. We read the project's pages and its founders' posts; we installed and ran nothing.* *Published 2026-10-22 (UTC) · Read from llm-d's release notes, documentation and blog, its founders' posts, and vLLM's and LMCache's documentation, each dated under Sources; nothing installed, nothing run · A small (human) team and a fleet of AI agents.* **the short version:** llm-d is a layer above model servers such as vLLM and SGLang (programs that run a language model on graphics cards) that turns many copies of a model into one service. It sends each request to a copy that already holds the start of that conversation in memory and is not too busy, moves that memory to CPU RAM or storage when the cards fill, and can split reading a prompt from writing the answer across machines. Version 1.0 adds what running it as a production service needs: priority-aware queueing, a standby router and a stable v1 configuration. It is Apache 2.0, was founded in 2025 by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA, and has been a CNCF Sandbox project since 2026-03-12. Every performance figure for it is self-reported by the project, its founders or a user writing with them; of those that give a card count, the smallest setup is four data-centre cards. On one card its router has nothing to choose between; what carries down is vLLM's own prefix caching and its offloading of that memory to CPU RAM, natively or through LMCache. On two cards, each holding a whole copy, keeping a conversation on one card is llm-d's job in miniature. We have measured none of this; the tests are queued. 2,741 words · about 12 minutes (at 220 words/min) · 1 table · data kit: no https://research.strata2signal.com/llm-d-and-one-or-two-gpus/ --- *If you're new here: [strata→signal](https://strata2signal.com) is a small workshop that builds things on its own machines and writes up what it measures. This page measures nothing; it is a reading. Every outside claim carries its source, a dated page named under Sources, and every performance number is its author's own, re-run by no one here.* ## The words this page leans on {#the-words-this-page-leans-on} - **Model server.** The program that runs a model on one or more graphics cards and answers requests: vLLM or SGLang. - **Replica.** One running copy of a model, on one card or a group of cards. - **KV cache.** The model's working notes on what it has read in a conversation, kept so each new word does not mean re-reading it all ([Reading is fast, writing is slow](https://research.strata2signal.com/reading-is-fast/) explains). - **Prefix caching.** Keeping those notes after a request ends, so a later request that begins the same way (a chat's history, an agent's instructions) skips re-reading it. - **Prefill and decode.** Reading the prompt, and writing the answer one token (a word or a piece of one) at a time. - **Kubernetes.** Software that runs programs across a fleet of servers and restarts them when they fail. A **pod** is its word for one running program; in the quotes below, one replica. - **Mixture-of-experts.** A model built from many blocks of weights, only a few of which work on each token. - **RDMA.** A network link that lets one machine read another's memory directly. ## What llm-d is {#what-llm-d-is} A service with many users runs many copies of a model, and something in front must decide which copy takes each request. The usual answer, a load balancer that deals requests out evenly, is expensive for a language model, because "Each vLLM pod manages its own cache in complete isolation," as one of llm-d's posts says, and "Standard load balancers naively spread traffic evenly using cache-blind metrics, scattering related requests across different pods and destroying cache locality." A conversation's tenth turn can land on a copy that never saw the first nine, and that copy re-reads all of it. llm-d is the thing in front, made aware of those notes. In its README's words: "Model servers like vLLM and SGLang handle efficiently running large language models on accelerators. llm-d provides state-of-the-art orchestration and optimizations above model servers". Its parts, by its own pages: - **A router** (its routing engine is the Endpoint Picker) that scores each copy by how much of a request's beginning it already holds and how busy it is. - **A tiered KV cache**: notes that no longer fit on the cards move to CPU RAM, then to SSD or shared storage (Google names its Managed Lustre service), instead of being thrown away. - **Prefill/decode disaggregation**: separate pools of cards read prompts and write answers, passing the notes between them over fast links. - **Wide expert parallelism**, spreading the largest mixture-of-experts models across many machines. - **Flow control**: router queues ordered by "a flow key that combines a fairness ID (the tenant or workload) and a priority band". - **Batch and async serving**: an OpenAI-style Batch API, and an async processor that pulls requests from queues such as Redis or Google Cloud Pub/Sub and runs them in spare capacity. The README calls the whole "a high-performance distributed inference serving stack optimized for production deployments on Kubernetes." ## What 1.0 adds {#what-1-0-adds} The v1.0.0 release was published on GitHub at 02:30 UTC on 2026-10-20 and announced that day at PyTorch Conference North America. IBM Research says the techniques are older: "These ideas are not new to version 1.0. What is new is the level of integration and operational maturity around them." The release notes name four themes; the quotations below are from the project's release post unless marked: - **Multi-tenant quality of service:** flow control by priority and fairness, with "controlled load shedding" (the release notes); "the defaults still mimic first-come-first-served, so existing deployments see no change until operators opt into multi-tenant policy". - **Zero-downtime operations:** "a hot standby that takes over scheduling if the active instance is replaced or fails", and DisaggregatedSet, which upgrades prefill and decode pools together. - **A stable foundation:** "The configuration API graduates to v1", and the Responses API can carry a conversation's state between turns. - **Agent-aware serving:** a session stays on the copy holding its notes, "sticky until saturated". Not yet, by the same post: the community "is already working toward active-active router configurations, inference request fail-over and migration". The release notes link the issue that tracked the release goal, "although not all of that was accomplished." ## Who builds it {#who-builds-it} The README names five founders: Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA. Red Hat's launch release (2025-05-20) listed six partners (AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI) and the university labs that began vLLM (Berkeley) and LMCache (Chicago). IBM Research names Red Hat, IBM and Google as lead maintainers; the project counts "more than 500 contributors from over 25 organizations". The code is under the Apache License 2.0. The CNCF (the Cloud Native Computing Foundation, part of the Linux Foundation) accepted llm-d "on March 12, 2026 at the Sandbox maturity level", and describes Sandbox projects as "Experimental projects not yet widely tested in production on the bleeding edge of technology". "1.0" is the project's own version number, not a CNCF level. The founders also build products around it: Red Hat ships it in Red Hat AI, Google runs its router as a managed service (GKE Inference Gateway), CoreWeave's customers "can run llm-d on CoreWeave Kubernetes Service alongside NVIDIA Dynamo", and NVIDIA contributed the NIXL transfer library "through NVIDIA Dynamo". That is no fault, but the numbers below come from the same companies' posts. ## The numbers, and whose they are {#the-numbers} Each row is the claim as its author states it; we re-ran none. | The claim | Whose | Setup, as stated | Source | |---|---|---|---| | Up to 3,000 concurrent coding agents; over 6.6 million output tokens a minute at peak | IBM Research and Red Hat | GLM-5.2 ("approximately 753-billion-parameter") on 544 NVIDIA H100 GPUs | IBM Research, 2026-09-08 | | "5 to 10 times less per token than equivalent commercial API pricing" | The same | "At current cloud rental rates", "with the largest savings on the input-heavy traffic patterns typical of agentic workloads" | The same | | 3x output throughput, 2x faster time to first token, against round-robin | Tesla and Red Hat | Llama 3.1 70B on 4× AMD MI300X | llm-d README | | 13.9x throughput with KV offloading, against GPU memory only | llm-d | 4× NVIDIA H100, 250 concurrent users | llm-d README | | Median time to first token cut by more than 85%, throughput 2.4x, against a plain Kubernetes Service | Google | An agentic coding workload on Ironwood TPU (TPU 7x), with prefix-aware routing and host-memory offload | Google Cloud, 2026-10-20 | | Prefix-cache hit rate doubled, 35% to 70% | Google | Its Gemini platform (formerly Vertex AI), in production | Google Cloud, 2026-10-20 | | With vLLM's prefix caching alone, time to first token from 4.3 seconds to 0.6, sending a ~10,000-token prompt a second time | llm-d blog | One vLLM copy of Qwen3-32B; hardware not stated | llm-d blog, 2025-09-24 | Of the rows that give a card count, the smallest is four data-centre cards. The posts do not always agree: IBM's and Red Hat's posts call GLM-5.2 "approximately 753-billion-parameter", the project's GLM-5.2 post "744B-parameter (39B active)"; for IBM Research's quantum-error-correction chip-design agents, IBM's press release reports "11.5 times lower inference cost than an all-proprietary configuration" and Red Hat's chief technology officer for AI "reduced blended serving costs five to ten times compared to proprietary options". The GLM-5.2 post, pricing a 64-GPU layout the same way, states its basis: "These figures cover GPU rental only, not the full cost of operating an API service," and "These figures assume that the cluster sustains the measured load 24/7." By its own breakdown, one layout's output cost "is about 81% of Z.ai's public GLM-5.2 rate" (the price Z.ai lists for its GLM-5.2 API); the savings sit on input, which "accounts for 98.6% of served tokens at this operating point." ## If you run one or two GPUs {#one-or-two-gpus} Red Hat's chief technology officer for AI gives llm-d's reason to exist: "Getting a single server to perform efficiently was table stakes. The next challenge was orchestrating model serving at cluster scale." Most of llm-d answers a question one home machine never asks. **One card: nothing to route, two ideas to keep.** One card runs one copy, so the router has no copy to choose, and splitting reading from writing means separate pools of cards. But two of llm-d's ideas work inside a single vLLM. The first is prefix caching. llm-d's own post: "In a single-instance environment, engines like vLLM leverage Automatic Prefix Caching to cut redundant work". vLLM has had it on by default since its V1 engine ("we now enable prefix caching by default in V1", 2025-01-27), and its documentation states the limit: it "only reduces the time of processing the queries (the prefilling phase) and does not reduce the time of generating new tokens (the decoding phase)." It shortens the wait before an answer starts; the answer is written no faster. The last row of the numbers table above is this feature alone, on one copy of a model. The second is offloading those notes to CPU RAM. On a 24 GB card the weights and the notes share one memory ([our vLLM history](https://research.strata2signal.com/a-short-history-of-vllm/) explains the paging), and when the notes fill their share, vLLM evicts the least recently used, so a conversation you come back to may be re-read in part or in full. vLLM can keep evicted notes in CPU RAM instead: `--kv-offloading-size` sets a buffer "in GiB" (off by default), and `--kv-offloading-backend` picks `native`, the default, or `lmcache`. LMCache, a separate Apache 2.0 project, offloads to "CPU RAM and Local Storage" on "the same machine that is running inference", and its walkthrough asks for "A Machine with at least one GPU." It reserves that RAM as "page-locked" memory, which the rest of the machine cannot use; its default is 5.0, which its own example labels "5GB of Pinned CPU memory" (llm-d's guide gives the unit as GiB). For scale: llm-d's guide puts Qwen3-32B's notes at "256 KiB per token", so one GiB of RAM holds 4,096 tokens of them (our arithmetic; other models differ), and gives each copy "a 300 GiB CPU tier" by default. At home, the RAM you can spare sets the size, and the trip back is not free: "CPU or remote hits still have to move KV back to the GPU." Whether restoring notes over a home PCIe slot beats re-reading them is a measurement we have not made. A smaller cousin of flow control carries down too: vLLM's `--scheduling-policy` offers `priority`, though its default is `fcfs`, first come, first served. **Two cards: llm-d's job in miniature.** Give each card a whole copy of a model and you have two replicas, so llm-d's question appears at its smallest: which card takes the next turn? vLLM can already serve both copies behind one address with `--data-parallel-size 2`, but its documentation says its built-in balancing "is based on the running and waiting queues in each of the engines" and "could be made more sophisticated in future by incorporating KV cache aware logic." A balancer that looks only at queues, or deals turns out in rotation, can send a conversation's second turn to the card that never saw its first. A rule that keeps each conversation on one card avoids that, and llm-d's version keeps a session in place until its card is full. That is our reasoning from the sources; we have not measured it. Nor does the router need Kubernetes now: a May 2026 post on llm-d's blog describes a mode that "lets the full routing stack run as a plain process or container with no Kubernetes anywhere in sight: on an HPC cluster, inside a Ray job, on a bare-metal rack, or on your laptop." It runs behind Envoy, a separate proxy; per-request priority set through Kubernetes falls "back to static configuration" or is "not yet available", and spotting a dead worker becomes "the surrounding orchestrator's job". On two cards that is two more programs to run for a choice between two. **Why the full stack stays in the data centre.** The project's tested guides assume a cluster: the quickstart's prerequisites are "kubectl, helm", and it deploys "2 replicas of Qwen/Qwen3-32B by default"; the tiered-cache guide's default is "4 GPUs in total", with an "H100 80 GB reference", and its Mooncake option needs RDMA networking or "the pods stay Pending". The release post's tested NVIDIA paths are "Hopper (H100 and H200), Blackwell (B200), and Grace Blackwell (GB200 NVL72 and GB300)"; it names no GeForce card. Even the post that freed the router calls Kubernetes "the recommended choice" for "production inference at scale", and says of research sites scheduled another way, "Adding a Kubernetes control plane buys operational overhead and little else." In our view one home box sits nearer that second case: a cluster manager adds parts to watch, and its gains come from having many machines. ## What we would measure {#what-we-would-measure} Three tests, queued and not yet run, on one EVGA GeForce RTX 3090 XC3 Ultra (24 GB), the card [our vLLM bench page](https://research.strata2signal.com/ollama-and-vllm-on-one-rtx-3090/) used, and then on two such cards, each holding a whole copy of one model: 1. **Prefix caching on and off**, replaying multi-turn conversations: the wait before each turn's answer starts. 2. **Notes overflowing the card**, with offload off, vLLM native and LMCache, at a few RAM sizes: whether a returning conversation starts sooner than by re-reading. 3. **Two cards**: vLLM's own `--data-parallel-size 2`, rotation, a sticky per-conversation rule, and llm-d's router as a plain process, scored on cache hits and the wait before answers start. Each test's pass rule will be published before its first run; until then, nothing here is a result of ours. ## How to check our work {#how-to-check-our-work} Every page under Sources was fetched with plain GETs on 2026-10-21 between 04:54 and 04:59 UTC (vLLM's "Data Parallel Deployment" page at 05:15 UTC), its text extracted, and every quotation read in that copy. Every page was fetched again for the check between 05:10 and 05:18 UTC, and each quotation was found in that copy too. llm-d's README and LICENSE were read at commit `3fe8687a` (2026-10-20 03:25 UTC); vLLM's flags and its data-parallel mode in its stable documentation, when its newest release was v0.32.0 (2026-10-18). Nothing was installed, built or run. ## Who ran this {#who-ran-this} A small (human) team asked the question and approves this page for publication. One AI agent, Claude Opus 5.5, read the sources and drafted the page with a table tracing each claim to its sentence; it was checked against those sources, and the findings fixed, before release. Thanks to the llm-d authors for release notes that say what they did not finish. ## Sources {#sources} Read 2026-10-21 (UTC); each source's own date is given. - llm-d, "llm-d v1.0: State-of-the-art inference, ready for Day 2", 2026-10-20. https://llm-d.ai/blog/llm-d-v1.0-state-of-the-art-inference-ready-for-day-2 - llm-d, release notes v1.0.0, 2026-10-20. https://github.com/llm-d/llm-d/releases/tag/v1.0.0 - llm-d, README and LICENSE at commit `3fe8687a`. https://github.com/llm-d/llm-d - llm-d documentation (v1.0): Architecture, Quickstart, Configure a Tiered Prefix Cache. https://llm-d.ai/docs/architecture · https://llm-d.ai/docs/getting-started/quickstart · https://llm-d.ai/docs/well-lit-paths/foundations/tiered-prefix-cache - llm-d blog: "KV-Cache Wins You Can See", 2025-09-24; "No Kubernetes? No Problem: llm-d Now Runs Anywhere", 2026-05-26; "Serving GLM-5.2 for Agentic Workloads on llm-d", 2026-07-22. https://llm-d.ai/blog/kvcache-wins-you-can-see · https://llm-d.ai/blog/running-llm-d-without-kubernetes · https://llm-d.ai/blog/serving-glm-5-2-agentic-workloads-on-llm-d - IBM Research, "llm-d 1.0 hardens open-source AI inference for production", 2026-10-20, and "How llm-d makes the most of the hardware you already have", 2026-09-08. https://research.ibm.com/blog/llmd-1-0-is-ready-for-production-inference-serving · https://research.ibm.com/blog/running-open-models-on-h100-gpus-with-llmd - IBM Newsroom, "llm-d 1.0 Advances Open, Cost-Efficient AI Inference for Production Scale", 2026-10-20. https://newsroom.ibm.com/2026-10-20-llm-d-1-0-advances-open,-cost-efficient-ai-inference-for-production-scale - Google Cloud, "Announcing llm-d 1.0", 2026-10-20, and "llm-d officially a CNCF Sandbox project", 2026-03-24. https://cloud.google.com/blog/topics/ai-infrastructure/llm-d-1-0-an-open-source-stack-for-distributed-inference · https://cloud.google.com/blog/products/containers-kubernetes/llm-d-officially-a-cncf-sandbox-project - Red Hat, Brian Stevens, "Crossing the threshold to dependable open source AI serving", 2026-10-20, and "Red Hat Launches the llm-d Community", 2025-05-20. https://www.redhat.com/en/blog/crossing-threshold-dependable-open-source-ai-serving · https://www.redhat.com/en/about/press-releases/red-hat-launches-llm-d-community-powering-distributed-gen-ai-inference-scale - CoreWeave, "CoreWeave Joins Red Hat Open Source Project llm-d as Founding Member", 2025-05-20. https://www.coreweave.com/blog/coreweave-joins-red-hat-open-source-ai-project - NVIDIA, "NVIDIA Dynamo Accelerates llm-d Community Initiatives", 2025-05-21. https://developer.nvidia.com/blog/nvidia-dynamo-accelerates-llm-d-community-initiatives-for-advancing-large-scale-distributed-inference - CNCF, llm-d project page, and "Welcome llm-d to the CNCF", 2026-03-24. https://www.cncf.io/projects/llm-d/ · https://www.cncf.io/blog/2026/03/24/welcome-llm-d-to-the-cncf-evolving-kubernetes-into-sota-ai-infrastructure/ - vLLM documentation: "Automatic Prefix Caching" (feature and design pages, latest), "Engine Arguments" and "Data Parallel Deployment" (stable); and "vLLM V1: A Major Upgrade to vLLM's Core Architecture", 2025-01-27. https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/ · https://docs.vllm.ai/en/latest/design/prefix_caching/ · https://docs.vllm.ai/en/stable/configuration/engine_args/ · https://docs.vllm.ai/en/stable/serving/data_parallel_deployment/ · https://blog.vllm.ai/2025/01/27/v1-alpha-release.html - LMCache, "CPU RAM" documentation and README. https://docs.lmcache.ai/kv_cache/cpu_ram.html · https://github.com/LMCache/LMCache *A reading, not a measurement. Every page above was read on 2026-10-21 (UTC). Nothing ran.* Corrections and later readings will be added below, each dated (UTC), saying in plain words what changed.