# Arm 2's substrate, measured — benchbox, 2026-09-21 *What this file is, for a reader who scrolled straight here: the receipts behind arm 2 of the Jev bench (README §Arm 2) — the vLLM install, the two things benchbox turned out not to have, the model's revision, what the server picked for FP8 on Ampere, and the measured price of the host bridge between the two cards. Every line is output from the box, not a recollection. Times are UTC, 2026-09-21.* --- ## 1. The install, and what it took | | | |---|---| | started / finished | **18:15:38Z → 18:20:20Z (4 min 42 s)**, against a 45-minute refusal ceiling | | venv | `/workshop/bench-vllm-venv`, built by `uv 0.12.7` | | interpreter | **CPython 3.12.14** (uv-managed) — **not** benchbox's system 3.14.4 | | command | `uv pip install "vllm==0.29.0"` (the model card's own pin) | | log | `~/bench-arm2/logs/install-vllm.log` on benchbox | **Why 3.12 and not the box's Python.** benchbox's only system interpreter is **3.14.4**. The vLLM wheel itself is `cp38-abi3` and would import, but its dependency set is not built for 3.14. The uv-managed **3.12.14** was already on the box (it is what `~/lab/venv` runs on), so it cost no download. Declared in README §A2.3.6 as a deviation from "Python 3 there". ### Versions, as installed | package | version | |---|---| | vllm | **0.29.0** | | torch | **2.13.0+cu130** | | NCCL (in torch) | **2.29.7** | | triton | 3.7.1 | | transformers | 5.17.0 | | flashinfer-python | 0.6.18 | | huggingface-hub | 1.32.0 | | numpy | 2.3.5 | ### The driver check, done before the install was trusted ``` $ nvidia-smi --query-gpu=driver_version --format=csv,noheader 595.84 $ nvidia-smi | head -4 | NVIDIA-SMI 595.84 Driver Version: 595.84 CUDA Version: 13.2 | ``` The wheel is built against **CUDA 13.0**; the driver supports **13.2**. A CUDA runtime is forward-compatible within a major version with a driver at least as new, so **13.0 on a 13.2 driver is in bounds** — and it was checked rather than assumed, because the reverse (a 13.x wheel on a 12.x driver) is the ordinary way this fails. `torch.cuda.is_available()` → `True`, both boards enumerated as `sm86`, 24,126 MiB each. --- ## 2. Two things benchbox did not have, and neither needed sudo A lane has no sudo on this box. Both gaps were closed inside the venv. ### 2.1 — There is no C compiler on benchbox at all ``` $ for c in cc gcc g++ clang clang++ x86_64-linux-gnu-gcc gcc-15 gcc-14 gcc-13; do printf "%-24s %s\n" "$c" "$(command -v $c || echo -)"; done cc - gcc - g++ - clang - ... $ ls /usr/bin/*gcc* /usr/bin/*clang* /usr/lib/gcc/x86_64-linux-gnu/ 2>&1 (nothing) ``` vLLM's first engine start died on it, in exactly those words: ``` RuntimeError: Worker failed with error 'Failed to find C compiler. Please specify via CC environment variable or set triton.knobs.build.impl.' ``` Triton JITs a small C launcher stub per kernel and needs a host compiler to do it. **The fix, no sudo:** `uv pip install ziglang` (0.16.0) into the bench venv — zig ships a complete clang — plus a two-line shim at `~/bench-arm2/shim/cc` that makes zig answer to gcc's calling convention. Two translations were needed and both are in the shim's own comments: * `-fno-sanitize=all` — zig adds ubsan traps by default; triton expects a bare stub and a trap inside one aborts the worker. * `-l:libcuda.so.1` — gcc's "link this exact filename" form, which zig's `lld` would not resolve (`ld.lld: error: unable to find library -l:libcuda.so.1`). The shim resolves each one against the `-L` paths of that same command line, then the system library dir, and passes an absolute path. **A name it cannot resolve is passed through unchanged**, so a real link error stays the linker's sentence and never becomes the shim's guess. Proved before the big start, on a real triton kernel: ``` $ CC=~/bench-arm2/shim/cc python triton_smoke.py triton 3.7.1 JIT OK, max err 0.0 ``` ### 2.2 — There is no CUDA toolkit either, and flashinfer's sampler wanted `nvcc` The second engine start got past triton, loaded both cards, compiled the graph, and died in the sampler warmup: ``` File ".../flashinfer/jit/cpp_ext.py", line 61, in get_cuda_path RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist ``` flashinfer's `top_k_top_p_sampling_from_logits` is a JIT-compiled CUDA kernel. **The fix, no sudo and no home-brew:** vLLM's own switch, `VLLM_USE_FLASHINFER_SAMPLER=0`, which puts the PyTorch sampler in its place. **Why this cannot touch a number in this bench.** Every arm sends `temperature: 0`, so token selection is greedy either way; and the readout does not read the sampler at all — it reads **raw logprobs** at position 0 (vLLM's default `logprobs-mode`). The sampler decides which token comes back as *text*, which only the generating control even looks at, and there greedy is greedy. Nothing about the quantization, the tensor-parallel split, the attention backend or the scores changed. *(A third, harmless one: `deep_gemm` also wants `CUDA_HOME` and logs an `AssertionError` traceback at import. vLLM catches it and moves on — it is a warning, not a failure, and it is in the log above the kernel-selection lines.)* --- ## 3. The model | | | |---|---| | repository | `openjev/openjev-FP8` | | **revision** | **`4ec320f267401e67c9be04d5df1be4d2b6b64f10`** | | downloaded | 18:16:0xZ → **18:19:49Z**, `hf download`, exit 0 | | into | `~/hf-cache/hub/models--openjev--openjev-FP8/snapshots/4ec320f2…` | | size | **30.41 GB**, 12 safetensors shards | | architecture | `Qwen3_5ForConditionalGeneration` (`model_type: qwen3_5`) — hybrid: 48 linear-attention layers and 16 full-attention layers, hidden 5,120, 64 layers | | **licence** | **CC BY-NC 4.0 — research / non-commercial only. BENCH-ONLY; these weights never become a seat.** | Its own conversion receipt, shipped in the repo (`FP8_CONVERSION_RECEIPT.json`): ```json {"scheme": "fp8 e4m3, dynamic activations, 128x128 weight blocks, scale_inv = absmax/448 (bf16)", "quantized_tensors": 400, "tensors_total": 1584, "worst_relative_dequant_error": {"value": 0.037109375, "tensor": "model.language_model.layers.42.linear_attn.out_proj.weight"}} ``` **The thinking channel, checked in the template rather than assumed.** Arm 1 was bitten by a chat template whose first output position was a channel marker and not a letter (§R6). This template does the same thing and takes the same cure: line 165 of `chat_template.jinja` reads ```jinja {%- if enable_thinking is defined and enable_thinking is false %} {{- '\n\n\n\n' }} ``` — the closed pair is written *into the generation prompt*, so position 0 is the answer. Every arm-2 call sends `chat_template_kwargs={"enable_thinking": false}`, which is what the OpenJev helper sends, and every row records its position-0 token so a recurrence would be visible rather than inferred. --- ## 4. What the server picked for FP8 on Ampere The 3090 is `sm86` and has **no native FP8 arithmetic**, so the question the pre-registration asked (§A2.4) was which weight-only path vLLM would choose. Its own line, verbatim: ``` INFO [__init__.py:695] Selected MarlinFP8ScaledMMLinearKernel for Fp8LinearMethod INFO [qwen_gdn_linear_attn.py:167] Using Triton/FLA GDN prefill kernel (requested=triton, head_k_dim=128). INFO [qwen_gdn_linear_attn.py:519] GDN decode kernel: cuda INFO [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'] INFO [flash_attn.py:897] Using FlashAttention version 2 ``` **Marlin**: the weights stay FP8 in memory and are dequantised into the tensor-core path at matmul time. So arm 2 gets FP8's *memory* on a card that cannot do FP8 *arithmetic* — which is the whole reason a 30 GB checkpoint fits two 24 GB boards at all. --- ## 5. The host bridge, as a number (README §A2.8) Measured at **18:22:15Z** with nothing else on either board (the arm-1 ollama unit was stopped at 18:21:00Z), by `phb_allreduce.py` under `torchrun --nproc_per_node 2`, torch 2.13.0+cu130, NCCL 2.29.7. **NCCL's own verdict on the pair, from the run's log:** ``` NCCL INFO P2P is disabled between connected GPUs 0 and 1. NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 isAllCudaP2p 0 NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0) ``` GeForce cards do not do peer-to-peer over PCIe, so **every byte of every all-reduce is staged through host memory** — card → RAM → card. That, and not the lane count alone, is what the numbers below are measuring. | one all-reduce carries | bytes | median | bus bandwidth | |---|---|---|---| | one decode step (1 token × 5,120 × bf16) | 10.0 KiB | **50.2 µs** | 0.20 GB/s | | 4 tokens | 40.0 KiB | 82.9 µs | 0.49 GB/s | | 16 tokens | 160.0 KiB | 155.6 µs | 1.05 GB/s | | 64 tokens | 640.0 KiB | 423.9 µs | 1.55 GB/s | | 256 tokens | 2.5 MiB | 1.50 ms | 1.75 GB/s | | 1,024 tokens | 10.0 MiB | 5.82 ms | 1.80 GB/s | | 4,096 tokens | 40.0 MiB | 23.11 ms | 1.82 GB/s | | 8,192 tokens | 80.0 MiB | 46.10 ms | 1.82 GB/s | | a full 16,384-token prefill | 160.0 MiB | **92.04 ms** | **1.82 GB/s** | **The two ends say two different things.** * **The ceiling is 1.82 GB/s.** That is what this pair of cards can move between them, ever, at any message size. * **The floor under one written token is 6.4 ms** — 128 all-reduces (2 per layer × 64 layers) at 50.2 µs each, *before the model computes anything*. A tensor-parallel decode on this box pays that on every token. * **A full 16,384-token prefill costs 11.8 s of pure communication** (92.04 ms × 128), however fast the cards compute. ### A correction to the topology, measured rather than inherited Arm 1's §R7 and this arm's brief both record **card 1** as the one on four lanes. **Under load, it is the other way round.** `pcie.link.width.current` sampled once a second through the microbench (`receipts/phb-linkwidth.csv`): | board | trained under load | the card's own maximum | |---|---|---| | **card 0** `GPU-46890836…` (04:00.0) | **gen 3 × 4** | gen 3 × 16 | | **card 1** `GPU-7aa0be10…` (2B:00.0) | **gen 4 × 16** | gen 4 × 16 | So **card 0 — the one arm 1 ran its whole bench on — is the four-lane card**, and card 1 has the wide slot. At idle both downtrain (gen 1) and nvidia-smi's `width.max` reports the *card's* capability rather than the slot's, which is how the claim got recorded backwards in the first place. `lspci -vv`'s `LnkCap` would settle it in one line and is **root-only on this box**, so a lane cannot read it; the trained width under real traffic is the reading that was available, and it is the one that matters anyway. The measured 1.82 GB/s is consistent with a gen-3 ×4 link (≈3.9 GB/s theoretical) carrying host-staged traffic. **What this does NOT change:** arm 1 ran one model whole on one card with no collective traffic at all, so its numbers are untouched by which slot that card sits in. What it changes is the *explanation* — arm 1's bench ran on the narrow card, which is fine for a single-card arm and is exactly the constraint arm 2 inherits.