{
 "schema_version": "1.0",
 "page_kind": "article",
 "pack_kind": "generate",
 "slug": "how-an-embedding-model-works",
 "title": "How an embedding model turns anything into a list of numbers",
 "dek": "An embedding model turns a sentence, a photograph or a few seconds of sound into a fixed-length list of numbers: what happens inside one, how one model holds them all in one space at several sizes, where keyword search still wins, and how to choose one.",
 "published": "2026-10-10",
 "series": [
  "guide"
 ],
 "licence": "CC BY 4.0",
 "status": "pending-judge",
 "pack_note": "no judge is seated, so every chip is held",
 "notice": "items with published:false are held and are not this page's published words",
 "source": {
  "url": "https://research.strata2signal.com/how-an-embedding-model-works/",
  "md_url": "https://research.strata2signal.com/how-an-embedding-model-works/index.md",
  "md_sha": "8f9437c7c5805c84140f9f5162fb1a9d8c6f3936ae56ce565c1b67a14718868a",
  "html_sha": "a7abf521c21a28cbfc3f7ebe8a761e4e7b0b8e0330f411590a23761695eaccbc",
  "anchors_sha": "5dc507eeb7552972d8038d909baf72e402ea9b17ca958b4807f3f56f848a4eff"
 },
 "short": {
  "paragraph": "An embedding model turns a piece of content into a vector, a fixed-length list of numbers, placed so that things that belong together land close together; search becomes \"which stored vectors point the way the question's does?\". The model learns where to put things from millions of pairs that belong together, a question and its answer, a photograph and its caption, which is also how one model can put a photograph beside the sentence that describes it. The rest of this page is the mechanism, with examples small enough to check with a calculator, and what we measured on EmbeddingGemma 2, the open embedding model Google released on 2026-10-06.",
  "from_dek": false,
  "counts": {
   "words": 7914,
   "minutes": 36,
   "tables": 2,
   "kit": false
  },
  "bullets": []
 },
 "sections": [
  {
   "id": "the-core-idea",
   "heading": "The core idea: a position on a map",
   "level": 2,
   "span": [
    6256,
    9872
   ],
   "chunks": [
    [
     6256,
     9872
    ]
   ],
   "chars": 3616,
   "digest": "an embedding model reads input and writes a vector, which is a fixed-length list of numbers representing a point in a space of dimensions. search becomes geometry by returning stored vectors nearest to a question. cosine similarity measures how close vectors are, ranging from 1 to -1.",
   "digest_skipped": null
  },
  {
   "id": "from-text-to-a-vector",
   "heading": "From text to a vector",
   "level": 2,
   "span": [
    9872,
    12850
   ],
   "chunks": [
    [
     9872,
     12850
    ]
   ],
   "chars": 2978,
   "digest": "a text embedder is a transformer that processes tokens through layers of bidirectional attention. the process involves looking up token vectors, applying a projection, and using mean pooling to collapse them into one fixed-length vector, which is then normalised to length 1.",
   "digest_skipped": null
  },
  {
   "id": "contrastive-training",
   "heading": "Contrastive training: learning from pairs",
   "level": 2,
   "span": [
    12850,
    16833
   ],
   "chunks": [
    [
     12850,
     16833
    ]
   ],
   "chars": 3983,
   "digest": "embedders learn from pairs that belong together using contrastive loss. training involves a batch of pairs where true pairs are scored against negatives. refinements include using hard negatives and multiple training stages, such as distillation, to improve the model.",
   "digest_skipped": null
  },
  {
   "id": "images-audio-video",
   "heading": "Images, audio and video, and the shared space",
   "level": 2,
   "span": [
    16833,
    21859
   ],
   "chunks": [
    [
     16833,
     21859
    ]
   ],
   "chars": 5026,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "the-modality-gap",
   "heading": "The modality gap",
   "level": 2,
   "span": [
    21859,
    24226
   ],
   "chunks": [
    [
     21859,
     24226
    ]
   ],
   "chars": 2367,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "matryoshka-sizes",
   "heading": "Matryoshka sizes: one vector, several lengths",
   "level": 2,
   "span": [
    24226,
    28314
   ],
   "chunks": [
    [
     24226,
     28314
    ]
   ],
   "chars": 4088,
   "digest": "Matryoshka Representation Learning trains a vector so that its prefixes also make good shorter vectors. when cutting a vector to a smaller size, one must re-normalise to maintain ranking quality, as skipping this step can degrade performance.",
   "digest_skipped": null
  },
  {
   "id": "storage-precision",
   "heading": "Storage precision: how many bytes per number",
   "level": 2,
   "span": [
    28314,
    32538
   ],
   "chunks": [
    [
     28314,
     32538
    ]
   ],
   "chars": 4224,
   "digest": "precision refers to the model's computation, the weights, and the stored vectors. storing vectors in formats like int8 or binary can save significant space. the article notes that computing in float16 is a hazard for EmbeddingGemma 2.",
   "digest_skipped": null
  },
  {
   "id": "task-prefixes",
   "heading": "Task prefixes: telling the model what a text is for",
   "level": 2,
   "span": [
    32538,
    35133
   ],
   "chunks": [
    [
     32538,
     35133
    ]
   ],
   "chars": 2595,
   "digest": "many embedders use short fixed strings called task prefixes to distinguish between different types of text, such as a query versus a passage. using the correct prefix is important, as a wrong prefix can hide the intended meaning.",
   "digest_skipped": null
  },
  {
   "id": "where-embedders-fail",
   "heading": "Where embedders fail, and why keyword search still wins on exact matches",
   "level": 2,
   "span": [
    35133,
    39843
   ],
   "chunks": [
    [
     35133,
     39843
    ]
   ],
   "chars": 4710,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "evaluation-and-contamination",
   "heading": "Evaluation, and the contamination problem",
   "level": 2,
   "span": [
    39843,
    43311
   ],
   "chunks": [
    [
     39843,
     43311
    ]
   ],
   "chars": 3468,
   "digest": "model-card scores often come from the MTEB benchmark. however, contamination can occur if a model is trained on benchmark data. using pre-registration with frozen, original questions is an alternative to avoid this problem.",
   "digest_skipped": null
  },
  {
   "id": "single-vector-or-late-interaction",
   "heading": "Single vector or late interaction",
   "level": 2,
   "span": [
    43311,
    45525
   ],
   "chunks": [
    [
     43311,
     45525
    ]
   ],
   "chars": 2214,
   "digest": "late interaction keeps a vector per token, which can improve retrieval by finding strong evidence for each word, but it increases the space footprint. a cross-encoder or re-ranker can re-order a shortlist to improve accuracy.",
   "digest_skipped": null
  },
  {
   "id": "why-vectors-never-mix",
   "heading": "Why vectors from different models never mix",
   "level": 2,
   "span": [
    45525,
    47731
   ],
   "chunks": [
    [
     45525,
     47731
    ]
   ],
   "chars": 2206,
   "digest": "vectors from different models cannot be mixed because each model uses its own coordinates. even different runtimes for the same model can produce different vectors. one should use one model, one prefix, one size, and one runtime per index.",
   "digest_skipped": null
  },
  {
   "id": "cost-and-hardware",
   "heading": "Cost and hardware",
   "level": 2,
   "span": [
    47731,
    50552
   ],
   "chunks": [
    [
     47731,
     50552
    ]
   ],
   "chars": 2821,
   "digest": "embedding costs include compute for indexing and searching. searching is relatively cheap, but re-embedding a large corpus when switching models is a significant expense. approximate nearest-neighbour indices like HNSW can help scale search.",
   "digest_skipped": null
  },
  {
   "id": "how-to-choose",
   "heading": "How to choose an embedder: a checklist",
   "level": 2,
   "span": [
    50552,
    52642
   ],
   "chunks": [
    [
     50552,
     52642
    ]
   ],
   "chars": 2090,
   "digest": "choosing an embedder requires writing a test before reading leaderboards, checking training data, running BM25, and selecting appropriate size and precision. it is also important to respect compute precision and record the provenance of every vector.",
   "digest_skipped": null
  },
  {
   "id": "what-to-take-with-you",
   "heading": "What to take with you",
   "level": 2,
   "span": [
    52642,
    53844
   ],
   "chunks": [
    [
     52642,
     53844
    ]
   ],
   "chars": 1202,
   "digest": "an embedder is a transformer, projection, and pooling step trained on pairs. a shared space maintains a gap between modalities, and Matryoshka vectors require re-normalisation when cut. keyword search remains effective for exact matches.",
   "digest_skipped": null
  },
  {
   "id": "how-to-check-our-work-and-see-it-live",
   "heading": "How to check our work — and see it live",
   "level": 2,
   "span": [
    53844,
    54350
   ],
   "chunks": [
    [
     53844,
     54350
    ]
   ],
   "chars": 506,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "the-rest-of-the-seminar",
   "heading": "The rest of the seminar",
   "level": 2,
   "span": [
    54350,
    55407
   ],
   "chunks": [
    [
     54350,
     55407
    ]
   ],
   "chars": 1057,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "who-ran-this-and-thanks",
   "heading": "Who ran this, and thanks",
   "level": 2,
   "span": [
    55407,
    55997
   ],
   "chunks": [
    [
     55407,
     55997
    ]
   ],
   "chars": 590,
   "digest": null,
   "digest_skipped": "credits"
  }
 ],
 "chips": [
  {
   "id": "c-4f5400eb",
   "q": "how do embedding models represent input data?",
   "a": "An embedding model reads an input and writes a vector, which is a fixed-length list of numbers. Each number acts as a coordinate in a space of dimensions, allowing the model to place similar inputs close together geometrically.",
   "cites": [
    "the-core-idea"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-319f299b",
   "q": "what steps transform text into a single vector?",
   "a": "The process involves cutting text into tokens, looking up vectors for those tokens, applying bidirectional attention through transformer layers, projecting the width, performing mean pooling to collapse vectors, and finally normalising to length 1.",
   "cites": [
    "from-text-to-a-vector"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-e490a7f5",
   "q": "how does contrastive training improve model performance?",
   "a": "The model learns from pairs that belong together by using a contrastive loss. It scores every question against every passage in a batch, rewarding true pairs for outscoring incorrect pairings, known as negatives.",
   "cites": [
    "contrastive-training"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  }
 ],
 "related": [
  {
   "slug": "embeddinggemma-2-on-release-day",
   "why": "shares ground with § Cost and hardware · § What an embedding model is"
  },
  {
   "slug": "the-compressed-photograph",
   "why": "shares ground with § Storage precision: how many bytes per number · § What the freed memory buys"
  },
  {
   "slug": "how-a-vision-model-sees",
   "why": "shares ground with § From text to a vector · § What an image becomes on the way in"
  }
 ],
 "thanks": "## Who ran this, and thanks {#who-ran-this-and-thanks}\n\nThis page stands on the authors of every paper and model card linked where it is cited, and on Google DeepMind and Nomic AI for open models whose cards say what they are; the cards and Google's announcement were read on 2026-10-09 (UTC). The licences this shelf has read first-hand are on its [licences](https://research.strata2signal.com/licences/) page. A small human team asked for this explainer and chose what it would cover; a fleet of AI agents read the sources, computed the examples and drafted it under that team's rulings.",
 "kit": null,
 "seat_class": {
  "writer": "gemma-class",
  "judge": null,
  "writer_runtime": "vllm"
 },
 "bench": {
  "bullets_written": 3,
  "bullets_kept": 0,
  "digests_written": 15,
  "digests_kept": 12,
  "chips_written": 8,
  "chips_kept": 3,
  "chips_grounded": 0,
  "chips_published": 0,
  "dropped_by": {
   "new_noun": 1,
   "figure": 1,
   "length": 0,
   "cite": 0,
   "judge": 0,
   "quote": 0,
   "directive": 0,
   "redaction": 0,
   "profanity": 0,
   "identifier": 0,
   "bare_figure": 0,
   "house_voice": 0,
   "editorial": 0,
   "hand-edit": 9
  },
  "new_noun_tokens_checked": [
   "0.47",
   "0.546",
   "0.56",
   "1",
   "128",
   "2",
   "2.53",
   "256",
   "32-bit",
   "BM25",
   "CPU-hours",
   "EmbeddingGemma",
   "Google's",
   "HNSW",
   "Learning",
   "MTEB",
   "Matryoshka",
   "Representation",
   "float16",
   "int8"
  ],
  "new_noun_tokens_withheld": 1,
  "figure_definition": "v3",
  "figure_boundary": "guarded",
  "drops": [
   {
    "kind": "bullet",
    "reason": "figure",
    "item": "the text model in 32-bit floats through Google's code cost 2.53 CPU-hours per million",
    "detail": "figures absent from span: ['32']"
   },
   {
    "kind": "chip",
    "reason": "new_noun",
    "item": null,
    "item_chars": 13,
    "withheld": "a string naming something the article does not",
    "detail": ""
   },
   {
    "kind": "bullet",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 70,
    "text_sha": "510b75f0f07b",
    "figure": "0.546",
    "withheld": "one bullet removed from this pack by hand",
    "removed_utc": "2026-10-10",
    "why": "loses what its figure is measured against and names a model the page does not: the page gives 0.546 as the average cosine at which EmbeddingGemma 2 puts a caption to a passage about something else, read on the public COCO captions, and the line drops the passage and calls the model the first EmbeddingGemma 2, where on this page the first EmbeddingGemma is the earlier model its 2025 report describes",
    "detail": "removed from this pack by hand on 2026-10-10 (UTC), after the writer ran and after the gates had counted: loses what its figure is measured against and names a model the page does not: the page gives 0.546 as the average cosine at which EmbeddingGemma 2 puts a caption to a passage about something else, read on the public COCO captions, and the line drops the passage and calls the model the first EmbeddingGemma 2, where on this page the first EmbeddingGemma is the earlier model its 2025 report describes. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "bullet",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 96,
    "text_sha": "9f80b40767f0",
    "figure": "0.56",
    "withheld": "one bullet removed from this pack by hand",
    "removed_utc": "2026-10-10",
    "why": "gives a figure measured elsewhere with no source: 0.47 and 0.56 are the average cosines Liang and colleagues report in their 2022 paper for pairs of inputs inside three trained models, and the page names the paper or card beside every figure it did not measure itself; a summary line with no source, on a page whose own figures come from its two bench pages on EmbeddingGemma 2, reads as a measurement made here",
    "detail": "removed from this pack by hand on 2026-10-10 (UTC), after the writer ran and after the gates had counted: gives a figure measured elsewhere with no source: 0.47 and 0.56 are the average cosines Liang and colleagues report in their 2022 paper for pairs of inputs inside three trained models, and the page names the paper or card beside every figure it did not measure itself; a summary line with no source, on a page whose own figures come from its two bench pages on EmbeddingGemma 2, reads as a measurement made here. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 247,
    "text_sha": "add5e33a616d",
    "section": "images-audio-video",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-10",
    "why": "the digest has every modality turned into a sequence for one shared transformer; the section names two designs, a separate model per modality, as in the towers of CLIP and the Nomic pair this page compares, and front ends feeding one transformer, and gives the second for EmbeddingGemma 2 only as the design its card points to",
    "detail": "removed from this pack by hand on 2026-10-10 (UTC), after the writer ran and after the gates had counted: the digest has every modality turned into a sequence for one shared transformer; the section names two designs, a separate model per modality, as in the towers of CLIP and the Nomic pair this page compares, and front ends feeding one transformer, and gives the second for EmbeddingGemma 2 only as the design its card points to. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 217,
    "text_sha": "b91eedc25527",
    "section": "the-modality-gap",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-10",
    "why": "the digest offers a per-modality z-score as a way to improve results; the section says it helped on the public COCO set and left EmbeddingGemma 2 with no right image ranked first on the 37 images of this site, all exploratory and no general fix, and gives the rule as filtering by modality or ranking each separately and merging",
    "detail": "removed from this pack by hand on 2026-10-10 (UTC), after the writer ran and after the gates had counted: the digest offers a per-modality z-score as a way to improve results; the section says it helped on the public COCO set and left EmbeddingGemma 2 with no right image ranked first on the 37 images of this site, all exploratory and no general fix, and gives the rule as filtering by modality or ranking each separately and merging. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 224,
    "text_sha": "f0642c0449dc",
    "section": "where-embedders-fail",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-10",
    "why": "the digest says reciprocal rank fusion can improve retrieval; the section shows fusion below the better single list on both sets this page measured, under BM25 on the 68-question text test and under EmbeddingGemma 2 alone on the code test, both exploratory, and says fusion is not free",
    "detail": "removed from this pack by hand on 2026-10-10 (UTC), after the writer ran and after the gates had counted: the digest says reciprocal rank fusion can improve retrieval; the section shows fusion below the better single list on both sets this page measured, under BM25 on the 68-question text test and under EmbeddingGemma 2 alone on the code test, both exploratory, and says fusion is not free. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 277,
    "text_sha": "adb831f5c764",
    "chip_id": "c-cedc96aa",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-10-10",
    "why": "the answer gives one design as the way modalities share an embedding, one shared transformer context ending in one projection and pooling step; the section names two designs, separate models per modality as in CLIP and the Nomic pair, and front ends feeding one transformer, and gives the second for EmbeddingGemma 2 only as the design its card points to, where the page found no training report for it as of 2026-10-09 (UTC)",
    "detail": "removed from this pack by hand on 2026-10-10 (UTC), after the writer ran and after the gates had counted: the answer gives one design as the way modalities share an embedding, one shared transformer context ending in one projection and pooling step; the section names two designs, separate models per modality as in CLIP and the Nomic pair, and front ends feeding one transformer, and gives the second for EmbeddingGemma 2 only as the design its card points to, where the page found no training report for it as of 2026-10-09 (UTC). Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 302,
    "text_sha": "012c9158bc82",
    "chip_id": "c-6b348034",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-10-10",
    "why": "the answer has unrelated text sit closer to an image than its own caption, with no model named; the section gives that reading for the Nomic pair on transformers 4.57.6 alone, where to a caption any unrelated passage looks closer than its own photograph, while EmbeddingGemma 2 puts a caption closer to its own photograph than to an unrelated passage, on average",
    "detail": "removed from this pack by hand on 2026-10-10 (UTC), after the writer ran and after the gates had counted: the answer has unrelated text sit closer to an image than its own caption, with no model named; the section gives that reading for the Nomic pair on transformers 4.57.6 alone, where to a caption any unrelated passage looks closer than its own photograph, while EmbeddingGemma 2 puts a caption closer to its own photograph than to an unrelated passage, on average. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 264,
    "text_sha": "78cfcaa55194",
    "chip_id": "c-b6433862",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-10-10",
    "why": "the answer makes 256 numbers close to lossless for Matryoshka vectors in general; the section gives close to lossless as the word of the EmbeddingGemma 2 card for its own 768-number vectors, and its own exploratory cuts on the 68-question test cost recall at 256, more from an 8-bit copy of the model",
    "detail": "removed from this pack by hand on 2026-10-10 (UTC), after the writer ran and after the gates had counted: the answer makes 256 numbers close to lossless for Matryoshka vectors in general; the section gives close to lossless as the word of the EmbeddingGemma 2 card for its own 768-number vectors, and its own exploratory cuts on the 68-question test cost recall at 256, more from an 8-bit copy of the model. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 261,
    "text_sha": "f6c06cbab7eb",
    "chip_id": "c-67035d2c",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-10-10",
    "why": "the answer says the right prefix can significantly improve recall; the section prints its prefix results as exploratory or as extra runs that decide nothing, with no test of significance, and its warning that a wrong prefix hides means the damage does not show in a score and is caught by re-embedding stored texts and comparing vectors",
    "detail": "removed from this pack by hand on 2026-10-10 (UTC), after the writer ran and after the gates had counted: the answer says the right prefix can significantly improve recall; the section prints its prefix results as exploratory or as extra runs that decide nothing, with no test of significance, and its warning that a wrong prefix hides means the damage does not show in a score and is caught by re-embedding stored texts and comparing vectors. Not a gate drop, and recorded here so the ledger adds up."
   }
  ]
 },
 "generated_utc": "2026-10-10T12:10:29Z"
}