{
 "schema_version": "1.0",
 "page_kind": "article",
 "pack_kind": "generate",
 "slug": "llm-d-and-one-or-two-gpus",
 "title": "llm-d 1.0, and what it means if you run one or two GPUs",
 "dek": "llm-d is free, open-source software that decides which of many running copies of a language model answers each request; its 1.0 came out on 2026-10-20, and of its ideas that carry down to one or two graphics cards at home, the two that matter most already live inside vLLM.",
 "published": "2026-10-22",
 "series": [
  "guide"
 ],
 "licence": "CC BY 4.0",
 "status": "pending-judge",
 "pack_note": "no judge is seated, so every chip is held",
 "notice": "items with published:false are held and are not this page's published words",
 "source": {
  "url": "https://research.strata2signal.com/llm-d-and-one-or-two-gpus/",
  "md_url": "https://research.strata2signal.com/llm-d-and-one-or-two-gpus/index.md",
  "md_sha": "f8e9703ab47659a72196ad75d64504eff37fa82c0ecec47194c5ca0d73023092",
  "html_sha": "f18b11066a2cd38e89eb23452079b841e1ede4463ba00cfe4310ff3b773bd9db",
  "anchors_sha": "08221af20225edfd85c0bf8ece3b71ef90d36f8287644b3cef66ea3e1c38d037"
 },
 "short": {
  "paragraph": "llm-d is a layer above model servers such as vLLM and SGLang (programs that run a language model on graphics cards) that turns many copies of a model into one service. It sends each request to a copy that already holds the start of that conversation in memory and is not too busy, moves that memory to CPU RAM or storage when the cards fill, and can split reading a prompt from writing the answer across machines. Version 1.0 adds what running it as a production service needs: priority-aware queueing, a standby router and a stable v1 configuration. It is Apache 2.0, was founded in 2025 by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA, and has been a CNCF Sandbox project since 2026-03-12. Every performance figure for it is self-reported by the project, its founders or a user writing with them; of those that give a card count, the smallest setup is four data-centre cards. On one card its router has nothing to choose between; what carries down is vLLM's own prefix caching and its offloading of that memory to CPU RAM, natively or through LMCache. On two cards, each holding a whole copy, keeping a conversation on one card is llm-d's job in miniature. We have measured none of this; the tests are queued.",
  "from_dek": false,
  "counts": {
   "words": 2741,
   "minutes": 12,
   "tables": 1,
   "kit": false
  },
  "bullets": [
   {
    "text": "the v1.0.0 release was published on github at 02:30 UTC",
    "figure": "02:30 UTC",
    "cite": "what-1-0-adds",
    "quote": "What 1.0 adds The v1.0.0 release was published on GitHub at 02:30 UTC on 2026-10-20 and announced that day at PyTorch Conference North America.",
    "span": [
     5752,
     5762
    ]
   }
  ]
 },
 "sections": [
  {
   "id": "the-words-this-page-leans-on",
   "heading": "The words this page leans on",
   "level": 2,
   "span": [
    2541,
    3704
   ],
   "chunks": [
    [
     2541,
     3704
    ]
   ],
   "chars": 1163,
   "digest": "this section defines technical terms including model server, replica, kv cache, prefix caching, prefill and decode, kubernetes, mixture-of-experts, and rdma.",
   "digest_skipped": null
  },
  {
   "id": "what-llm-d-is",
   "heading": "What llm-d is",
   "level": 2,
   "span": [
    3704,
    5672
   ],
   "chunks": [
    [
     3704,
     5672
    ]
   ],
   "chars": 1968,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "what-1-0-adds",
   "heading": "What 1.0 adds",
   "level": 2,
   "span": [
    5672,
    7067
   ],
   "chunks": [
    [
     5672,
     7067
    ]
   ],
   "chars": 1395,
   "digest": "the v1.0.0 release introduces multi-tenant quality of service, zero-downtime operations, a stable foundation with a v1 configuration api, and agent-aware serving.",
   "digest_skipped": null
  },
  {
   "id": "who-builds-it",
   "heading": "Who builds it",
   "level": 2,
   "span": [
    7067,
    8291
   ],
   "chunks": [
    [
     7067,
     8291
    ]
   ],
   "chars": 1224,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "the-numbers",
   "heading": "The numbers, and whose they are",
   "level": 2,
   "span": [
    8291,
    10795
   ],
   "chunks": [
    [
     8291,
     10795
    ]
   ],
   "chars": 2504,
   "digest": "the section presents various performance claims from different authors, including throughput, time to first token, and cost savings for different workloads and hardware setups.",
   "digest_skipped": null
  },
  {
   "id": "one-or-two-gpus",
   "heading": "If you run one or two GPUs",
   "level": 2,
   "span": [
    10795,
    15818
   ],
   "chunks": [
    [
     10795,
     15818
    ]
   ],
   "chars": 5023,
   "digest": "this section discusses how llm-d concepts like prefix caching and kv offloading apply to single or dual gpu setups, noting that while some features work in vLLM, the full stack is designed for data centre clusters.",
   "digest_skipped": null
  },
  {
   "id": "what-we-would-measure",
   "heading": "What we would measure",
   "level": 2,
   "span": [
    15818,
    16709
   ],
   "chunks": [
    [
     15818,
     16709
    ]
   ],
   "chars": 891,
   "digest": "the authors propose three tests using an evga geforce rtx 3090 xc3 ultra to measure prefix caching, kv offloading, and two-card routing performance.",
   "digest_skipped": null
  },
  {
   "id": "how-to-check-our-work",
   "heading": "How to check our work",
   "level": 2,
   "span": [
    16709,
    17327
   ],
   "chunks": [
    [
     16709,
     17327
    ]
   ],
   "chars": 618,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "who-ran-this",
   "heading": "Who ran this",
   "level": 2,
   "span": [
    17327,
    17718
   ],
   "chunks": [
    [
     17327,
     17718
    ]
   ],
   "chars": 391,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "sources",
   "heading": "Sources",
   "level": 2,
   "span": [
    17718,
    21423
   ],
   "chunks": [
    [
     17718,
     21423
    ]
   ],
   "chars": 3705,
   "digest": null,
   "digest_skipped": null
  }
 ],
 "chips": [
  {
   "id": "c-b4702723",
   "q": "what technical terms define the model serving environment?",
   "a": "The environment involves model servers like vLLM or SGLang, replicas of models, KV caches for conversation notes, prefix caching, prefill and decode processes, Kubernetes orchestration, mixture-of-experts models, and RDMA network links.",
   "cites": [
    "the-words-this-page-leans-on"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-2f1f268b",
   "q": "how does llm-d manage model requests and cache locality?",
   "a": "llm-d acts as an orchestration layer that uses a router to score copies based on existing cache and busyness. It employs a tiered KV cache moving notes to CPU RAM or storage, utilizes prefill/decode disaggregation, and implements flow control via a router that queues requests by fairness and priority.",
   "cites": [
    "what-llm-d-is"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-326e6186",
   "q": "what features characterize the v1.0.0 release?",
   "a": "The release introduces multi-tenant quality of service through controlled load shedding, zero-downtime operations with hot standbys and DisaggregatedSet, a stable configuration API, and agent-aware serving that keeps sessions on specific copies until they are saturated.",
   "cites": [
    "what-1-0-adds"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-21b658f6",
   "q": "how does prefix caching function on a single card?",
   "a": "In a single-instance environment, engines like vLLM use automatic prefix caching to reduce the time of the prefilling phase. While it shortens the wait before an answer starts, it does not reduce the time of generating new tokens during the decoding phase.",
   "cites": [
    "one-or-two-gpus"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-e2cb99dc",
   "q": "what testing remains planned for the described setup?",
   "a": "Planned tests include measuring the wait before turns in multi-turn conversations with prefix caching, evaluating whether offloading notes to CPU RAM or LMCache beats re-reading, and comparing different routing and scaling methods on two cards.",
   "cites": [
    "what-we-would-measure"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  }
 ],
 "related": [
  {
   "slug": "how-the-long-table-works",
   "why": "shares ground with § The numbers, and whose they are · § Numbers, so it isn't just talk"
  },
  {
   "slug": "the-same-eighteen-pictures-card-by-card",
   "why": "shares ground with § What we would measure · § The EVGA 3090 Ti FTW3 Ultra's column, at 350 W and 300 W (added 2026-09-26)"
  },
  {
   "slug": "one-3090-against-one-3090-ti",
   "why": "shares ground with § What we would measure · § The question"
  }
 ],
 "thanks": "## Who ran this {#who-ran-this}\n\nA small (human) team asked the question and approves this page for publication. One AI agent, Claude Opus 5.5, read the sources and drafted the page with a table tracing each claim to its sentence; it was checked against those sources, and the findings fixed, before release. Thanks to the llm-d authors for release notes that say what they did not finish.",
 "kit": null,
 "seat_class": {
  "writer": "gemma-class",
  "judge": null,
  "writer_runtime": "vllm"
 },
 "bench": {
  "bullets_written": 3,
  "bullets_kept": 1,
  "digests_written": 7,
  "digests_kept": 5,
  "chips_written": 7,
  "chips_kept": 5,
  "chips_grounded": 0,
  "chips_published": 0,
  "dropped_by": {
   "new_noun": 0,
   "figure": 2,
   "length": 0,
   "cite": 0,
   "judge": 0,
   "quote": 0,
   "directive": 0,
   "redaction": 0,
   "profanity": 0,
   "identifier": 0,
   "bare_figure": 0,
   "house_voice": 0,
   "editorial": 0,
   "hand-edit": 4
  },
  "new_noun_tokens_checked": [
   "000",
   "02",
   "12",
   "2.0",
   "2.4x",
   "2026",
   "3",
   "30",
   "3090",
   "35%",
   "50",
   "500",
   "6.6",
   "70%",
   "85%",
   "API",
   "Apache",
   "CNCF",
   "CPU",
   "Cloud",
   "CoreWeave",
   "DisaggregatedSet",
   "Google",
   "Hat",
   "IBM",
   "Ironwood",
   "KV",
   "Kubernetes",
   "LMCache",
   "License",
   "March",
   "NVIDIA",
   "RAM",
   "RDMA",
   "Red",
   "Research",
   "SGLang",
   "Sandbox",
   "TPU",
   "UTC",
   "v1",
   "v1.0.0",
   "xc3"
  ],
  "new_noun_tokens_withheld": 0,
  "figure_definition": "v3",
  "figure_boundary": "guarded",
  "drops": [
   {
    "kind": "bullet",
    "reason": "figure",
    "item": "the project was accepted into the cncf sandbox on March 12, 2026",
    "detail": "terminal token not verbatim in the cited span"
   },
   {
    "kind": "chip",
    "reason": "figure",
    "item": "Founding members include Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. The project has more than 500 contributors from over 50 organizations and is accepted as a CNCF Sandbox project.",
    "detail": "figures absent from cited spans: ['50']"
   },
   {
    "kind": "bullet",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 49,
    "text_sha": "15b5d146f860",
    "figure": "70",
    "withheld": "one bullet removed from this pack by hand",
    "removed_utc": "2026-10-22",
    "why": "Google's own claim, for its Gemini platform (formerly Vertex AI) in production, loses its owner and its setup: the line reads as a measured fact about llm-d, and the page re-ran none of the numbers in its table",
    "detail": "removed from this pack by hand on 2026-10-22 (UTC), after the writer ran and after the gates had counted: Google's own claim, for its Gemini platform (formerly Vertex AI) in production, loses its owner and its setup: the line reads as a measured fact about llm-d, and the page re-ran none of the numbers in its table. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 285,
    "text_sha": "b4e030223c69",
    "section": "what-llm-d-is",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-22",
    "why": "'a high-performance distributed inference serving stack' is the README's description of itself, which the page quotes as the README's words ('The README calls the whole …'); the digest states it as a fact",
    "detail": "removed from this pack by hand on 2026-10-22 (UTC), after the writer ran and after the gates had counted: 'a high-performance distributed inference serving stack' is the README's description of itself, which the page quotes as the README's words ('The README calls the whole …'); the digest states it as a fact. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 212,
    "text_sha": "0456b2b74267",
    "section": "who-builds-it",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-22",
    "why": "'more than 500 contributors' is the project's own count ('the project counts …') stated as a fact, and the section's two conditions are lost: the CNCF calls Sandbox projects 'not yet widely tested in production', and the numbers below come from the same companies' posts",
    "detail": "removed from this pack by hand on 2026-10-22 (UTC), after the writer ran and after the gates had counted: 'more than 500 contributors' is the project's own count ('the project counts …') stated as a fact, and the section's two conditions are lost: the CNCF calls Sandbox projects 'not yet widely tested in production', and the numbers below come from the same companies' posts. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 315,
    "text_sha": "5629963f6a97",
    "chip_id": "c-e9ca7320",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-10-22",
    "why": "the claims lose their setups: IBM Research's and Red Hat's 3,000 agents and 6.6 million tokens a minute ran GLM-5.2 on 544 data-centre cards, and Google's 85% and 2.4x are against a plain Kubernetes Service; the page's point is that the smallest stated setup is four data-centre cards",
    "detail": "removed from this pack by hand on 2026-10-22 (UTC), after the writer ran and after the gates had counted: the claims lose their setups: IBM Research's and Red Hat's 3,000 agents and 6.6 million tokens a minute ran GLM-5.2 on 544 data-centre cards, and Google's 85% and 2.4x are against a plain Kubernetes Service; the page's point is that the smallest stated setup is four data-centre cards. Not a gate drop, and recorded here so the ledger adds up."
   }
  ]
 },
 "generated_utc": "2026-10-22T10:15:42Z"
}