{ "model_id": "nemo-q4-k-m", "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "arm": "control-3090", "box_class": "gpu-ql3090-24g", "mode": "context-ladder", "arm_cards": [ "" ], "target_uuid": "", "started_utc": "2026-10-15T05:34:20Z", "prompt_sha256": "90eedd0c53f9554ae3837674504fcb7090013d9a432a653352183c0f25a7ce5c", "num_predict": 256, "ladder_note": "The frozen prompt is about 537 tokens. Climbing this ladder measures the cost of RESERVING a context window, NOT the cost of filling one: a 128k window with 537 tokens in it decodes at nearly the speed of an 8k window with 537 tokens in it. What changes is the VRAM the reservation costs, and therefore what fits. Any tok/s difference across levels is a second-order effect of the allocation and is not a long-prompt number. The FILLED-WINDOW arm in this same directory is the one that answers the long-prompt question, and the two must never be quoted as if they were the same measurement.", "fit_rule_note": "TWO RULES ON THIS BENCH, one per kind of arm, because the card is 12 GB and neither bank model fits it whole. On the `control-3090` arm the rule is the strict one, inherited unchanged: a context rung FITS when `/api/ps` reports `size_vram == size` -- every byte of weights and KV on the card, nothing in host RAM -- and the runner's own resident memory stays inside the allowance below; a rung that spills is printed as `does not fit` with its per-card memory and the ladder stops there. On the `card` arm that rule can be satisfied by NOTHING in the bank (the MoE's weights alone are 16.75 GiB, the dense model's 14.14 GiB, against a 12 GB board), so the rule is `any-vram`: the rung is scored whenever anything reached the card, EVERY figure carries its VRAM/RAM split fraction, and the ladder's stop condition changes from a VRAM stop to a HOST-MEMORY stop -- see LADDER_STOP_NOTE. A spilled figure is never printed in a column headed by a whole one.", "ladder_stop_note": "On a 24 GB card the ladder climbed until a rung spilled and stopped there, because the spill WAS the ceiling. On this 12 GB card every rung spills by construction, so that stop would fire at the first rung and measure nothing. The `card` arm's ladder therefore stops on the HOST side instead: a rung is refused when the model's RAM-resident portion plus the runner's own resident memory would exceed this box's available RAM allowance, read from the box rather than typed -- and this box has 30 GB where the three benches before it had 45 GB, which is itself a confound this bench declares rather than discovers. The HEADLINE for a spilled arm is therefore not 'the largest window that fits' but 'the largest window at which this box loaded the model and generated at all', and it is labelled in those words wherever it appears.", "fit_rule_by_arm": { "card": "any-vram", "cpu-box": "cpu-only", "control-3090": "whole" }, "cross_arm_ratio_note": "a `card` figure is a SPILLED figure (`any-vram`: part of the model is in host RAM) and a `control-3090` figure is a WHOLE figure (`whole`: every byte on the card). A ratio between them is not a card-to-card ratio -- it is what a 12 GB card costs against a 24 GB one for a model only the 24 GB card can hold, which is a more useful claim and a different one. Every such cell says which side spilled and by how much.", "pcie_note": "This card negotiated PCIe x16 of a x16-capable generation-3 link -- the same x16 slot the 3090 used, and the slot on which the pair bench measured a x16 + x4 asymmetry. With one card there is no per-token traffic BETWEEN cards, so the link is not in the decode path the way it was for the split pair; it still carries the model in and the tokens out. Every run records `pcie.link.width.current` UNDER LOAD, because an idle card drops its link to save power and a width read at rest would flatter the result.", "thermal_layout_note": "ONE GeForce RTX 3090 Ti 24 GB in a consumer desktop, alone on the board's x16 slot -- the same box, the same case and the same airflow in which one GeForce RTX 3090 24 GB was measured earlier the same day and two GeForce RTX 3080 10 GB cards the night before. A temperature here is a reading of THAT arrangement. Against the single 3090 the arrangement is the same one, which is what makes the two cards' thermal rows comparable; against the PAIR it is not, because with one card there is no second board warming the air or blocking a face. The card is traced on every run.", "card_labels": { "": "the x16 seat at withheld-2026-10-16 - vendor unread - " }, "card_identity": { "": { "index": null, "bus_id": "withheld-2026-10-16", "pci_sub_device_id": null, "vendor": null, "board_per_operator": null, "vbios": null, "serial": null, "pcie_link_width_max": null, "pcie_link_width_negotiated": null, "memory_total_mib": null, "memory_bus_width_bits": null, "clocks_max_sm_mhz": null, "clocks_max_memory_mhz": null, "power_default_limit_w": null, "power_min_limit_w": null, "power_max_limit_w": null, "power_limit_at_bench_start_w": null, "note": "the single GeForce RTX 3080 Ti 12 GB seated in this box's x16 slot at withheld-2026-10-16, in place of a GeForce RTX 3090 24 GB. THIS BOX'S BOOT-TIME GPU POWER-CAP UNIT WILL FAIL FOR IT: `bench-gpu-power-cap.service` is enabled and its config (/etc/bench/gpu-power-caps.conf, written 2026-08-28) names the two 3090s this box carried, both at 280 W. After the swap the removed card's row fails and NOTHING is applied to this card, so it comes up at its own DEFAULT -- the identical trap the one-3090-Ti bench recorded on the other box. The lesson is carried: a cap read once at the start of a night is not a cap, and every arm reads the limit back in the same call as its own data." } }, "memory_temp_support": { "checked_utc": "2026-10-15T05:34:20Z", "paths": { "nvidia-smi --query-gpu=temperature.memory": { "raw": "0, , N/A\n1, , N/A", "readable": false }, "nvidia-smi -q -d TEMPERATURE": { "lines": [ "GPU Shutdown Temp : 98 C", "GPU Slowdown Temp : 95 C", "GPU Target Temperature : 83 C", "Memory Current Temp : N/A", "Memory Max Operating Temp : N/A", "GPU Shutdown Temp : 98 C", "GPU Slowdown Temp : 95 C", "GPU Target Temperature : 83 C", "Memory Current Temp : N/A", "Memory Max Operating Temp : N/A" ], "readable": false }, "NVML NVML_FI_DEV_MEMORY_TEMP": { "available": true, "field_id": 82, "cards": { "0": { "call_rc": 0, "field_rc": 3, "supported": false, "not_supported": true, "value_c": null }, "1": { "call_rc": 0, "field_rc": 3, "supported": false, "not_supported": true, "value_c": null } } } }, "readable": false, "verdict": "the memory die's temperature is NOT readable on these cards through any of the three paths asked, so the memory-temperature stop condition could not arm and NO memory temperature is reported anywhere in this bench. The core temperature stop and the driver's own thermal-slowdown reasons are the thermal instrument instead." }, "runner_rss_allowance_gib": 4.0, "num_gpu_option": null, "num_gpu_policy": "layers left to ollama's own planner (`auto`)", "num_gpu_note": "ollama decides for itself how many of a model's layers to put on the card. On the pair of 3080s that decision was measurably conservative -- gemma4:26b loaded 75.2% into VRAM and spilled 4.25 GiB to host RAM while about 5 GiB of the two cards' 20 GiB sat unused -- so the pair bench reported `auto` and a forced layer count side by side. On ONE 24 GB card there is no placement decision to make: the expected reading is 100% on the card with no spill. This harness records `options.num_gpu` on every run as `num_gpu_option` and reads the planner's actual placement back from /api/ps, so a spill on a card with room is visible as a finding rather than absorbed into a tok/s figure. A value at or above the model's layer count means every layer. ON THIS CARD THE QUESTION INVERTS. The pair bench forced `num_gpu` UP to recover cards the planner had left half empty; on a 12 GB board the model cannot fit however high the count goes, so forcing it up is not a recovery -- it is either ignored or an out-of-memory refusal, and either is a result. What is worth measuring is whether the planner leaves VRAM on the table on a card it CANNOT fill: so the probe BRACKETS the planner's own choice rather than jumping to the layer count. The planner's num_gpu is read back from the server, the model's own block count is read from /api/show (never typed -- the pair bench knew gemma4:26b had 31 layers because it READ it, and this bench has no receipt for the dense model's), and the probe walks UPWARD from the planner's choice in the steps PREREG SS5.4 registers, stopping at the FIRST refusal and recording it. An out-of-memory answer is written into the result file as that rung's outcome, never as a discarded run.", "instance_env": { "posture": "speed", "unit": "ql-ollama", "started_utc": "2026-10-15T05:33:54Z", "base_url": "http://127.0.0.1:withheld-2026-10-16", "ollama_bin_sha256_prefix": "39c8dea520be", "ollama_version": "0.32.13", "OLLAMA_MODELS": "store/ (the bench's own)", "CUDA_VISIBLE_DEVICES": "the stage's card (state/card.env)", "OLLAMA_VULKAN": "0", "GGML_VK_VISIBLE_DEVICES": "-1", "HIP_VISIBLE_DEVICES": "", "ROCR_VISIBLE_DEVICES": "", "OLLAMA_FLASH_ATTENTION": "1", "OLLAMA_KV_CACHE_TYPE": "q8_0", "OLLAMA_CONTEXT_LENGTH": 32768, "OLLAMA_": 1, "OLLAMA_MAX_LOADED_MODELS": 1, "OLLAMA_KEEP_ALIVE": "0", "discovery_cuda_devices": 1, "discovery_vulkan_devices": 0 }, "ups_before_arm": { "ups": "pr1500@localhost", "read_utc": "2026-10-15T05:34:21Z", "available": true, "raw": "battery.charge: 100\nbattery.charge.low: 10\nbattery.charge.warning: 20\nbattery.mfr.date: CPS\nbattery.runtime: 3975\nbattery.runtime.low: 300\nbattery.status: 100%\nbattery.type: PbAcid\nbattery.voltage: 27.2\nbattery.voltage.nominal: 24\ndevice.mfr: CPS\ndevice.model: CP1500PFCLCDa\ndevice.serial: \ndevice.type: ups\ndriver.debug: 0\ndriver.flag.allow_killpower: 0\ndriver.name: usbhid-ups\ndriver.parameter.interrupt_pipe_no_events_tolerance: -1\ndriver.parameter.pollfreq: 12\ndriver.parameter.pollinterval: 2\ndriver.parameter.port: auto\ndriver.parameter.synchronous: auto\ndriver.state: quiet\ndriver.version: 2.8.4\ndriver.version.data: CyberPower HID 0.84\ndriver.version.internal: 0.67\ndriver.version.usb: libusb-1.0.29 (API: 0x0100010B)\ninput.sensitivity: normal\ninput.transfer.high: 139\ninput.transfer.low: 100\ninput.voltage: 123.0\ninput.voltage.nominal: 120\noutput.voltage: 123.0\nups.beeper.status: enabled\nups.delay.shutdown: 60\nups.delay.start: 120\nups.firmware: CR01802H1\nups.load: 12\nups.mfr: CPS\nups.model: CP1500PFCLCDa\nups.power: 116\nups.power.nominal: 1500\nups.productid: 0601\nups.realpower: 128\nups.realpower.nominal: 1000\nups.serial: \nups.status: OL\nups.test.result: No test initiated\nups.timer.shutdown: -60\nups.timer.start: -60\nups.vendorid: 0764", "battery.charge": 100.0, "input.voltage": 123.0, "output.voltage": 123.0, "ups.load": 12.0, "ups.realpower": 128.0, "ups.realpower.nominal": 1000.0, "ups.status": "OL", "load_pct": 12.0, "stop_threshold_pct": 80.0, "over_stop_threshold": false }, "levels": [ { "num_ctx": 4096, "started_utc": "2026-10-15T05:34:21Z", "loads": true, "size_total": 7570079087, "size_vram": 7570079087, "context_length_loaded": 4096, "fits_fully_in_vram": true, "verdict": "fits: fully resident on the card", "offload_split": "100.0% VRAM / 0.0% RAM", "vram_delta_mib": { "": 7535.0 }, "memory_used_mib_per_card": { "": 7536.0 }, "memory_used_mib_per_card_before": { "": 1.0 }, "runner_peak_rss_gib": 0.79, "runner_rss_within_allowance": true, "runner_procs": [ { "pid": 6590, "rss_kb": 831156, "threads": 23 } ], "free_m_at_level": "total used free shared buff/cache available\nMem: 30990 2135 19246 37 10043 28854\nSwap: 8191 0 8191", "kv_footprint_bytes": 0, "kv_footprint_gib": 0.0, "runs": [ { "ttft_ms": 537.52, "first_token_was_thinking": false, "wall_s": 3.439, "eval_count": 256, "eval_duration_ns": 2898598000, "decode_tok_s": 88.319, "prompt_eval_count": 519, "prompt_eval_duration_ns": 29738000, "prefill_tok_s": 17452.418, "load_duration_ns": 500194404, "total_duration_ns": 3435592183, "streamed_chunks": 256, "response_chars": 1013, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 87.885, "decode_tok_s_wall_gross": 74.439, "num_ctx_option": 4096, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 109.19, "median": 109.25, "max": 109.29, "mean": 109.25, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 7536.0, "median": 7536.0, "max": 7536.0, "mean": 7536.0, "n": 6 }, "power_w_per_card": { "": 109.25 }, "memory_used_mib_per_card": { "": 7536.0 }, "power_w_all_cards": 109.25, "ups_whole_box": { "load_pct": 19.0, "realpower_w": 193.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 259.01, "max_w": 299.73, "median_w": 298.57, "mean_w_whole_window": 234.05, "max_w_whole_window": 299.73, "limit_w": 300.0, "over_idle_w": 149.76, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.78, "median": 23.2, "max": 23.9, "mean": 23.33, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 23.0, "median": 23.0, "max": 23.0, "mean": 23.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 109.22, "median": 297.42, "max": 299.73, "mean": 234.05, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 7536.0, "median": 7536.0, "max": 7536.0, "mean": 7536.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 78.5, "n": 12 }, "temperature_c": { "min": 35.0, "median": 40.5, "max": 42.0, "mean": 39.6, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1537.5, "max": 1725.0, "mean": 1551.2, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 112.92, "median": 298.57, "max": 299.73, "mean": 259.01, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 7536.0, "median": 7536.0, "max": 7536.0, "mean": 7536.0, "n": 10 }, "utilization_pct": { "min": 60.0, "median": 98.0, "max": 98.0, "mean": 94.2, "n": 10 }, "temperature_c": { "min": 38.0, "median": 41.0, "max": 42.0, "mean": 40.5, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1530.0, "max": 1560.0, "mean": 1516.5, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 2932.68, "power_mean_w_all_cards": 259.01, "energy_j_per_1k_tokens_all_cards": 2932.68, "watts_per_tok_s": 2.9327, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 42.0, "temp_min_c": 38.0, "temp_rise_c": 4.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1560.0, "clock_median_mhz": 1530.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 259.01, "power_pct_of_limit": 86.3, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 86.3% of the cap and the card at 42 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 42.0, "temp_min_c": 38.0, "temp_rise_c": 4.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1560.0, "clock_median_mhz": 1530.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 259.01, "power_pct_of_limit": 86.3, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 86.3% of the cap and the card at 42 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 7536.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 19.0, "median": 19.0, "max": 19.0, "mean": 19.0, "n": 2 }, "whole_box_realpower_w": { "min": 193.0, "median": 193.0, "max": 193.0, "mean": 193.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 1 }, { "ttft_ms": 481.8, "first_token_was_thinking": false, "wall_s": 3.3891, "eval_count": 256, "eval_duration_ns": 2902380000, "decode_tok_s": 88.203, "prompt_eval_count": 519, "prompt_eval_duration_ns": 15274000, "prefill_tok_s": 33979.311, "load_duration_ns": 460650029, "total_duration_ns": 3383893861, "streamed_chunks": 256, "response_chars": 1013, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 87.709, "decode_tok_s_wall_gross": 75.535, "num_ctx_option": 4096, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 109.4, "median": 109.48, "max": 109.48, "mean": 109.45, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 7536.0, "median": 7536.0, "max": 7536.0, "mean": 7536.0, "n": 6 }, "power_w_per_card": { "": 109.45 }, "memory_used_mib_per_card": { "": 7536.0 }, "power_w_all_cards": 109.45, "ups_whole_box": { "load_pct": 19.0, "realpower_w": 197.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 262.1, "max_w": 300.88, "median_w": 298.21, "mean_w_whole_window": 236.66, "max_w_whole_window": 300.88, "limit_w": 300.0, "over_idle_w": 152.65, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.79, "median": 23.16, "max": 23.83, "mean": 23.24, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 23.0, "median": 23.0, "max": 23.0, "mean": 23.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 109.42, "median": 294.37, "max": 300.88, "mean": 236.66, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 7536.0, "median": 7536.0, "max": 7536.0, "mean": 7536.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 78.1, "n": 12 }, "temperature_c": { "min": 38.0, "median": 44.0, "max": 45.0, "mean": 42.9, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1455.0, "median": 1515.0, "max": 1725.0, "mean": 1551.2, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 126.84, "median": 298.21, "max": 300.88, "mean": 262.1, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 7536.0, "median": 7536.0, "max": 7536.0, "mean": 7536.0, "n": 10 }, "utilization_pct": { "min": 56.0, "median": 98.0, "max": 98.0, "mean": 93.7, "n": 10 }, "temperature_c": { "min": 42.0, "median": 44.0, "max": 45.0, "mean": 43.9, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1455.0, "median": 1515.0, "max": 1590.0, "mean": 1516.5, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 2971.54, "power_mean_w_all_cards": 262.1, "energy_j_per_1k_tokens_all_cards": 2971.54, "watts_per_tok_s": 2.9716, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 45.0, "temp_min_c": 42.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1455.0, "clock_max_mhz": 1590.0, "clock_median_mhz": 1515.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 262.1, "power_pct_of_limit": 87.4, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 87.4% of the cap and the card at 45 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 45.0, "temp_min_c": 42.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1455.0, "clock_max_mhz": 1590.0, "clock_median_mhz": 1515.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 262.1, "power_pct_of_limit": 87.4, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 87.4% of the cap and the card at 45 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 7536.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 19.0, "median": 19.0, "max": 19.0, "mean": 19.0, "n": 2 }, "whole_box_realpower_w": { "min": 197.0, "median": 197.0, "max": 197.0, "mean": 197.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 2 }, { "ttft_ms": 447.74, "first_token_was_thinking": false, "wall_s": 3.3598, "eval_count": 256, "eval_duration_ns": 2906866000, "decode_tok_s": 88.067, "prompt_eval_count": 519, "prompt_eval_duration_ns": 14856000, "prefill_tok_s": 34935.38, "load_duration_ns": 425826734, "total_duration_ns": 3354404191, "streamed_chunks": 256, "response_chars": 1013, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 87.567, "decode_tok_s_wall_gross": 76.195, "num_ctx_option": 4096, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 109.67, "median": 109.71, "max": 109.74, "mean": 109.71, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 7536.0, "median": 7536.0, "max": 7536.0, "mean": 7536.0, "n": 6 }, "power_w_per_card": { "": 109.71 }, "memory_used_mib_per_card": { "": 7536.0 }, "power_w_all_cards": 109.71, "ups_whole_box": { "load_pct": 19.0, "realpower_w": 196.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 9, "mean_w": 259.65, "max_w": 301.25, "median_w": 299.41, "mean_w_whole_window": 232.39, "max_w_whole_window": 301.25, "limit_w": 300.0, "over_idle_w": 149.94, "samples": 11, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.91, "median": 23.37, "max": 23.93, "mean": 23.39, "n": 11 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 11 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 11 }, "temperature_c": { "min": 24.0, "median": 24.0, "max": 24.0, "mean": 24.0, "n": 11 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 11 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 11 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 11 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 11 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 11, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 109.7, "median": 298.94, "max": 301.25, "mean": 232.39, "n": 11 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 7536.0, "median": 7536.0, "max": 7536.0, "mean": 7536.0, "n": 11 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 80.2, "n": 11 }, "temperature_c": { "min": 41.0, "median": 46.0, "max": 48.0, "mean": 45.5, "n": 11 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 11 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 11 }, "clocks_sm_mhz": { "min": 1455.0, "median": 1500.0, "max": 1725.0, "mean": 1539.5, "n": 11 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 11 }, "throttle_sw_power_cap_fraction": 0.8182, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 11, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 128.32, "median": 299.41, "max": 301.25, "mean": 259.65, "n": 9 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 7536.0, "median": 7536.0, "max": 7536.0, "mean": 7536.0, "n": 9 }, "utilization_pct": { "min": 98.0, "median": 98.0, "max": 98.0, "mean": 98.0, "n": 9 }, "temperature_c": { "min": 45.0, "median": 47.0, "max": 48.0, "mean": 46.6, "n": 9 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 9 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 9 }, "clocks_sm_mhz": { "min": 1455.0, "median": 1500.0, "max": 1515.0, "mean": 1498.3, "n": 9 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 9 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 9 }, "busy_samples": 9 } } }, "energy_j_per_1k_tokens": 2948.31, "power_mean_w_all_cards": 259.65, "energy_j_per_1k_tokens_all_cards": 2948.31, "watts_per_tok_s": 2.9483, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 9, "temp_max_c": 48.0, "temp_min_c": 45.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1455.0, "clock_max_mhz": 1515.0, "clock_median_mhz": 1500.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 259.65, "power_pct_of_limit": 86.5, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "no clock drop during decode (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 9, "temp_max_c": 48.0, "temp_min_c": 45.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1455.0, "clock_max_mhz": 1515.0, "clock_median_mhz": 1500.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 259.65, "power_pct_of_limit": 86.5, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "no clock drop during decode (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 7536.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 11 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 19.0, "median": 19.0, "max": 19.0, "mean": 19.0, "n": 2 }, "whole_box_realpower_w": { "min": 196.0, "median": 196.0, "max": 196.0, "mean": 196.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 3 } ], "derived": { "decode_tok_s": { "min": 88.067, "median": 88.203, "max": 88.319, "mean": 88.196, "n": 3 }, "ttft_ms": { "min": 447.74, "median": 481.8, "max": 537.52, "mean": 489.02, "n": 3 }, "power_mean_w": { "min": 259.01, "median": 259.65, "max": 262.1, "mean": 260.25, "n": 3 }, "power_mean_w_all_cards": { "min": 259.01, "median": 259.65, "max": 262.1, "mean": 260.25, "n": 3 }, "energy_j_per_1k_tokens_all_cards": { "min": 2932.68, "median": 2948.31, "max": 2971.54, "mean": 2950.84, "n": 3 }, "energy_j_per_1k_tokens": { "min": 2932.68, "median": 2948.31, "max": 2971.54, "mean": 2950.84, "n": 3 }, "mem_clock_median_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "mem_clock_floor_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "peak_mem_bandwidth_gbs_at_median_clock": { "min": 912.1, "median": 912.1, "max": 912.1, "mean": 912.1, "n": 3 }, "decode_gbs": null, "decode_bandwidth_fraction_of_peak_pct": null, "sw_power_cap_active_fraction": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 3 }, "temp_max_c": { "min": 42.0, "median": 45.0, "max": 48.0, "mean": 45.0, "n": 3 }, "fan_max_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 3 } }, "finished_utc": "2026-10-15T05:35:15Z" }, { "num_ctx": 8192, "started_utc": "2026-10-15T05:35:15Z", "loads": true, "size_total": 8056628837, "size_vram": 8056628837, "context_length_loaded": 8192, "fits_fully_in_vram": true, "verdict": "fits: fully resident on the card", "offload_split": "100.0% VRAM / 0.0% RAM", "vram_delta_mib": { "": 470.0 }, "memory_used_mib_per_card": { "": 8006.0 }, "memory_used_mib_per_card_before": { "": 7536.0 }, "runner_peak_rss_gib": 0.82, "runner_rss_within_allowance": true, "runner_procs": [ { "pid": 7182, "rss_kb": 859460, "threads": 23 } ], "free_m_at_level": "total used free shared buff/cache available\nMem: 30990 2151 19230 69 10075 28839\nSwap: 8191 0 8191", "kv_footprint_bytes": 486549750, "kv_footprint_gib": 0.45, "runs": [ { "ttft_ms": 437.48, "first_token_was_thinking": false, "wall_s": 3.3597, "eval_count": 256, "eval_duration_ns": 2916267000, "decode_tok_s": 87.783, "prompt_eval_count": 519, "prompt_eval_duration_ns": 29668000, "prefill_tok_s": 17493.596, "load_duration_ns": 402007033, "total_duration_ns": 3353833958, "streamed_chunks": 256, "response_chars": 1002, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 87.263, "decode_tok_s_wall_gross": 76.198, "num_ctx_option": 8192, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 110.07, "median": 110.24, "max": 110.28, "mean": 110.2, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 8006.0, "median": 8006.0, "max": 8006.0, "mean": 8006.0, "n": 6 }, "power_w_per_card": { "": 110.2 }, "memory_used_mib_per_card": { "": 8006.0 }, "power_w_all_cards": 110.2, "ups_whole_box": { "load_pct": 19.0, "realpower_w": 197.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 9, "mean_w": 259.78, "max_w": 299.85, "median_w": 299.06, "mean_w_whole_window": 232.59, "max_w_whole_window": 299.85, "limit_w": 300.0, "over_idle_w": 149.58, "samples": 11, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.94, "median": 23.29, "max": 23.83, "mean": 23.32, "n": 11 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 11 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 11 }, "temperature_c": { "min": 24.0, "median": 24.0, "max": 24.0, "mean": 24.0, "n": 11 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 11 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 11 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 11 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 11 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 11, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 110.26, "median": 298.6, "max": 299.85, "mean": 232.59, "n": 11 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 8006.0, "median": 8006.0, "max": 8006.0, "mean": 8006.0, "n": 11 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 80.2, "n": 11 }, "temperature_c": { "min": 44.0, "median": 49.0, "max": 51.0, "mean": 48.5, "n": 11 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 11 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 11 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1500.0, "max": 1725.0, "mean": 1528.6, "n": 11 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 11 }, "throttle_sw_power_cap_fraction": 0.8182, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 11, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 129.12, "median": 299.06, "max": 299.85, "mean": 259.78, "n": 9 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 8006.0, "median": 8006.0, "max": 8006.0, "mean": 8006.0, "n": 9 }, "utilization_pct": { "min": 98.0, "median": 98.0, "max": 98.0, "mean": 98.0, "n": 9 }, "temperature_c": { "min": 48.0, "median": 50.0, "max": 51.0, "mean": 49.6, "n": 9 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 9 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 9 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1500.0, "max": 1515.0, "mean": 1485.0, "n": 9 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 9 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 9 }, "busy_samples": 9 } } }, "energy_j_per_1k_tokens": 2959.33, "power_mean_w_all_cards": 259.78, "energy_j_per_1k_tokens_all_cards": 2959.33, "watts_per_tok_s": 2.9593, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 9, "temp_max_c": 51.0, "temp_min_c": 48.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1515.0, "clock_median_mhz": 1500.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 259.78, "power_pct_of_limit": 86.6, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 86.6% of the cap and the card at 51 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 9, "temp_max_c": 51.0, "temp_min_c": 48.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1515.0, "clock_median_mhz": 1500.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 259.78, "power_pct_of_limit": 86.6, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 86.6% of the cap and the card at 51 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 8006.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 11 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 19.0, "median": 19.0, "max": 19.0, "mean": 19.0, "n": 2 }, "whole_box_realpower_w": { "min": 197.0, "median": 197.0, "max": 197.0, "mean": 197.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 1 }, { "ttft_ms": 478.2, "first_token_was_thinking": false, "wall_s": 3.3882, "eval_count": 256, "eval_duration_ns": 2907141000, "decode_tok_s": 88.059, "prompt_eval_count": 519, "prompt_eval_duration_ns": 14798000, "prefill_tok_s": 35072.307, "load_duration_ns": 456557065, "total_duration_ns": 3384836966, "streamed_chunks": 256, "response_chars": 1002, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 87.63, "decode_tok_s_wall_gross": 75.557, "num_ctx_option": 8192, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 110.52, "median": 110.57, "max": 110.63, "mean": 110.57, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 8006.0, "median": 8006.0, "max": 8006.0, "mean": 8006.0, "n": 6 }, "power_w_per_card": { "": 110.57 }, "memory_used_mib_per_card": { "": 8006.0 }, "power_w_all_cards": 110.57, "ups_whole_box": { "load_pct": 19.0, "realpower_w": 199.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 264.08, "max_w": 300.68, "median_w": 298.44, "mean_w_whole_window": 238.5, "max_w_whole_window": 300.68, "limit_w": 300.0, "over_idle_w": 153.51, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.95, "median": 23.47, "max": 24.01, "mean": 23.47, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 24.0, "median": 24.0, "max": 24.0, "mean": 24.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 110.59, "median": 297.51, "max": 300.68, "mean": 238.5, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 8006.0, "median": 8006.0, "max": 8006.0, "mean": 8006.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.4, "n": 12 }, "temperature_c": { "min": 47.0, "median": 51.5, "max": 52.0, "mean": 50.7, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1515.0, "max": 1725.0, "mean": 1546.2, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 130.88, "median": 298.44, "max": 300.68, "mean": 264.08, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 8006.0, "median": 8006.0, "max": 8006.0, "mean": 8006.0, "n": 10 }, "utilization_pct": { "min": 95.0, "median": 98.0, "max": 98.0, "mean": 97.7, "n": 10 }, "temperature_c": { "min": 49.0, "median": 52.0, "max": 52.0, "mean": 51.4, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1515.0, "max": 1590.0, "mean": 1510.5, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 2998.9, "power_mean_w_all_cards": 264.08, "energy_j_per_1k_tokens_all_cards": 2998.9, "watts_per_tok_s": 2.9989, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 52.0, "temp_min_c": 49.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1590.0, "clock_median_mhz": 1515.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 264.08, "power_pct_of_limit": 88.0, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 88.0% of the cap and the card at 52 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 52.0, "temp_min_c": 49.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1590.0, "clock_median_mhz": 1515.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 264.08, "power_pct_of_limit": 88.0, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 88.0% of the cap and the card at 52 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 8006.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 19.0, "median": 19.0, "max": 19.0, "mean": 19.0, "n": 2 }, "whole_box_realpower_w": { "min": 199.0, "median": 199.0, "max": 199.0, "mean": 199.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 2 }, { "ttft_ms": 506.02, "first_token_was_thinking": false, "wall_s": 3.4197, "eval_count": 256, "eval_duration_ns": 2910935000, "decode_tok_s": 87.944, "prompt_eval_count": 519, "prompt_eval_duration_ns": 14911000, "prefill_tok_s": 34806.519, "load_duration_ns": 485061676, "total_duration_ns": 3416436847, "streamed_chunks": 256, "response_chars": 1002, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 87.517, "decode_tok_s_wall_gross": 74.859, "num_ctx_option": 8192, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 110.48, "median": 110.53, "max": 110.64, "mean": 110.55, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 8006.0, "median": 8006.0, "max": 8006.0, "mean": 8006.0, "n": 6 }, "power_w_per_card": { "": 110.55 }, "memory_used_mib_per_card": { "": 8006.0 }, "power_w_all_cards": 110.55, "ups_whole_box": { "load_pct": 20.0, "realpower_w": 200.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 262.11, "max_w": 300.45, "median_w": 298.4, "mean_w_whole_window": 236.87, "max_w_whole_window": 300.45, "limit_w": 300.0, "over_idle_w": 151.56, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 23.01, "median": 23.55, "max": 24.13, "mean": 23.54, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 24.0, "median": 24.0, "max": 24.0, "mean": 24.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 110.68, "median": 297.44, "max": 300.45, "mean": 236.87, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 8006.0, "median": 8006.0, "max": 8006.0, "mean": 8006.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.7, "n": 12 }, "temperature_c": { "min": 49.0, "median": 53.5, "max": 54.0, "mean": 52.7, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1440.0, "median": 1515.0, "max": 1725.0, "mean": 1533.8, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 126.22, "median": 298.4, "max": 300.45, "mean": 262.11, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 8006.0, "median": 8006.0, "max": 8006.0, "mean": 8006.0, "n": 10 }, "utilization_pct": { "min": 98.0, "median": 98.0, "max": 98.0, "mean": 98.0, "n": 10 }, "temperature_c": { "min": 51.0, "median": 54.0, "max": 54.0, "mean": 53.4, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1440.0, "median": 1507.5, "max": 1515.0, "mean": 1495.5, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 2980.41, "power_mean_w_all_cards": 262.11, "energy_j_per_1k_tokens_all_cards": 2980.41, "watts_per_tok_s": 2.9804, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 54.0, "temp_min_c": 51.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1440.0, "clock_max_mhz": 1515.0, "clock_median_mhz": 1507.5, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 262.11, "power_pct_of_limit": 87.4, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "no clock drop during decode (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 54.0, "temp_min_c": 51.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1440.0, "clock_max_mhz": 1515.0, "clock_median_mhz": 1507.5, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 262.11, "power_pct_of_limit": 87.4, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "no clock drop during decode (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 8006.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 20.0, "median": 20.0, "max": 20.0, "mean": 20.0, "n": 2 }, "whole_box_realpower_w": { "min": 200.0, "median": 200.0, "max": 200.0, "mean": 200.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 3 } ], "derived": { "decode_tok_s": { "min": 87.783, "median": 87.944, "max": 88.059, "mean": 87.929, "n": 3 }, "ttft_ms": { "min": 437.48, "median": 478.2, "max": 506.02, "mean": 473.9, "n": 3 }, "power_mean_w": { "min": 259.78, "median": 262.11, "max": 264.08, "mean": 261.99, "n": 3 }, "power_mean_w_all_cards": { "min": 259.78, "median": 262.11, "max": 264.08, "mean": 261.99, "n": 3 }, "energy_j_per_1k_tokens_all_cards": { "min": 2959.33, "median": 2980.41, "max": 2998.9, "mean": 2979.55, "n": 3 }, "energy_j_per_1k_tokens": { "min": 2959.33, "median": 2980.41, "max": 2998.9, "mean": 2979.55, "n": 3 }, "mem_clock_median_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "mem_clock_floor_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "peak_mem_bandwidth_gbs_at_median_clock": { "min": 912.1, "median": 912.1, "max": 912.1, "mean": 912.1, "n": 3 }, "decode_gbs": null, "decode_bandwidth_fraction_of_peak_pct": null, "sw_power_cap_active_fraction": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 3 }, "temp_max_c": { "min": 51.0, "median": 52.0, "max": 54.0, "mean": 52.3, "n": 3 }, "fan_max_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 3 } }, "finished_utc": "2026-10-15T05:36:09Z" }, { "num_ctx": 16384, "started_utc": "2026-10-15T05:36:09Z", "loads": true, "size_total": 8786437733, "size_vram": 8786437733, "context_length_loaded": 16384, "fits_fully_in_vram": true, "verdict": "fits: fully resident on the card", "offload_split": "100.0% VRAM / 0.0% RAM", "vram_delta_mib": { "": 696.0 }, "memory_used_mib_per_card": { "": 8702.0 }, "memory_used_mib_per_card_before": { "": 8006.0 }, "runner_peak_rss_gib": 0.84, "runner_rss_within_allowance": true, "runner_procs": [ { "pid": 7787, "rss_kb": 876280, "threads": 23 } ], "free_m_at_level": "total used free shared buff/cache available\nMem: 30990 2172 19235 85 10065 28817\nSwap: 8191 0 8191", "kv_footprint_bytes": 1216358646, "kv_footprint_gib": 1.13, "runs": [ { "ttft_ms": 530.11, "first_token_was_thinking": false, "wall_s": 3.4517, "eval_count": 256, "eval_duration_ns": 2918486000, "decode_tok_s": 87.717, "prompt_eval_count": 519, "prompt_eval_duration_ns": 32420000, "prefill_tok_s": 16008.637, "load_duration_ns": 490700586, "total_duration_ns": 3448333470, "streamed_chunks": 256, "response_chars": 1002, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 87.282, "decode_tok_s_wall_gross": 74.166, "num_ctx_option": 16384, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 111.37, "median": 111.44, "max": 111.49, "mean": 111.43, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 8702.0, "median": 8702.0, "max": 8702.0, "mean": 8702.0, "n": 6 }, "power_w_per_card": { "": 111.43 }, "memory_used_mib_per_card": { "": 8702.0 }, "power_w_all_cards": 111.43, "ups_whole_box": { "load_pct": 19.0, "realpower_w": 198.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 261.68, "max_w": 299.49, "median_w": 298.78, "mean_w_whole_window": 236.64, "max_w_whole_window": 299.49, "limit_w": 300.0, "over_idle_w": 150.25, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 23.05, "median": 23.31, "max": 23.92, "mean": 23.45, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 24.0, "median": 24.0, "max": 25.0, "mean": 24.4, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 111.44, "median": 297.71, "max": 299.49, "mean": 236.64, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 8702.0, "median": 8702.0, "max": 8702.0, "mean": 8702.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.5, "n": 12 }, "temperature_c": { "min": 51.0, "median": 55.5, "max": 56.0, "mean": 54.7, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1500.0, "max": 1725.0, "mean": 1528.8, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 125.24, "median": 298.78, "max": 299.49, "mean": 261.68, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 8702.0, "median": 8702.0, "max": 8702.0, "mean": 8702.0, "n": 10 }, "utilization_pct": { "min": 97.0, "median": 98.0, "max": 98.0, "mean": 97.8, "n": 10 }, "temperature_c": { "min": 53.0, "median": 56.0, "max": 56.0, "mean": 55.4, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1500.0, "max": 1515.0, "mean": 1489.5, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 2983.24, "power_mean_w_all_cards": 261.68, "energy_j_per_1k_tokens_all_cards": 2983.24, "watts_per_tok_s": 2.9832, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 56.0, "temp_min_c": 53.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1515.0, "clock_median_mhz": 1500.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 261.68, "power_pct_of_limit": 87.2, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 87.2% of the cap and the card at 56 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 56.0, "temp_min_c": 53.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1515.0, "clock_median_mhz": 1500.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 261.68, "power_pct_of_limit": 87.2, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 87.2% of the cap and the card at 56 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 8702.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 19.0, "median": 19.0, "max": 19.0, "mean": 19.0, "n": 2 }, "whole_box_realpower_w": { "min": 198.0, "median": 198.0, "max": 198.0, "mean": 198.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 1 }, { "ttft_ms": 479.46, "first_token_was_thinking": false, "wall_s": 3.4062, "eval_count": 256, "eval_duration_ns": 2921709000, "decode_tok_s": 87.62, "prompt_eval_count": 519, "prompt_eval_duration_ns": 15277000, "prefill_tok_s": 33972.639, "load_duration_ns": 456444966, "total_duration_ns": 3400701398, "streamed_chunks": 256, "response_chars": 1002, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 87.126, "decode_tok_s_wall_gross": 75.156, "num_ctx_option": 16384, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 112.16, "median": 112.16, "max": 112.21, "mean": 112.18, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 8702.0, "median": 8702.0, "max": 8702.0, "mean": 8702.0, "n": 6 }, "power_w_per_card": { "": 112.18 }, "memory_used_mib_per_card": { "": 8702.0 }, "power_w_all_cards": 112.18, "ups_whole_box": { "load_pct": 19.0, "realpower_w": 199.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 259.97, "max_w": 300.37, "median_w": 298.34, "mean_w_whole_window": 235.31, "max_w_whole_window": 300.37, "limit_w": 300.0, "over_idle_w": 147.79, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 23.06, "median": 23.41, "max": 23.98, "mean": 23.48, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 25.0, "median": 25.0, "max": 25.0, "mean": 25.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 112.04, "median": 297.03, "max": 300.37, "mean": 235.31, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 8702.0, "median": 8702.0, "max": 8702.0, "mean": 8702.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 80.7, "n": 12 }, "temperature_c": { "min": 53.0, "median": 57.0, "max": 58.0, "mean": 56.1, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1440.0, "median": 1500.0, "max": 1725.0, "mean": 1523.8, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 128.69, "median": 298.34, "max": 300.37, "mean": 259.97, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 8702.0, "median": 8702.0, "max": 8702.0, "mean": 8702.0, "n": 10 }, "utilization_pct": { "min": 86.0, "median": 98.0, "max": 98.0, "mean": 96.8, "n": 10 }, "temperature_c": { "min": 55.0, "median": 57.0, "max": 58.0, "mean": 56.7, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1440.0, "median": 1492.5, "max": 1515.0, "mean": 1483.5, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 2967.02, "power_mean_w_all_cards": 259.97, "energy_j_per_1k_tokens_all_cards": 2967.02, "watts_per_tok_s": 2.967, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 58.0, "temp_min_c": 55.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1440.0, "clock_max_mhz": 1515.0, "clock_median_mhz": 1492.5, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 259.97, "power_pct_of_limit": 86.7, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "no clock drop during decode (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 58.0, "temp_min_c": 55.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1440.0, "clock_max_mhz": 1515.0, "clock_median_mhz": 1492.5, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 259.97, "power_pct_of_limit": 86.7, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "no clock drop during decode (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 8702.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 19.0, "median": 19.0, "max": 19.0, "mean": 19.0, "n": 2 }, "whole_box_realpower_w": { "min": 199.0, "median": 199.0, "max": 199.0, "mean": 199.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 2 }, { "ttft_ms": 507.83, "first_token_was_thinking": false, "wall_s": 3.442, "eval_count": 256, "eval_duration_ns": 2928839000, "decode_tok_s": 87.407, "prompt_eval_count": 519, "prompt_eval_duration_ns": 14915000, "prefill_tok_s": 34797.184, "load_duration_ns": 486118077, "total_duration_ns": 3436714872, "streamed_chunks": 256, "response_chars": 1002, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.907, "decode_tok_s_wall_gross": 74.376, "num_ctx_option": 16384, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 113.18, "median": 113.18, "max": 113.18, "mean": 113.18, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 8702.0, "median": 8702.0, "max": 8702.0, "mean": 8702.0, "n": 6 }, "power_w_per_card": { "": 113.18 }, "memory_used_mib_per_card": { "": 8702.0 }, "power_w_all_cards": 113.18, "ups_whole_box": { "load_pct": 20.0, "realpower_w": 201.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 261.57, "max_w": 301.24, "median_w": 298.92, "mean_w_whole_window": 236.86, "max_w_whole_window": 301.24, "limit_w": 300.0, "over_idle_w": 148.39, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.9, "median": 23.33, "max": 23.87, "mean": 23.36, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 25.0, "median": 25.0, "max": 25.0, "mean": 25.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 113.3, "median": 297.64, "max": 301.24, "mean": 236.86, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 8702.0, "median": 8702.0, "max": 8702.0, "mean": 8702.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.0, "n": 12 }, "temperature_c": { "min": 54.0, "median": 58.0, "max": 59.0, "mean": 57.3, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1410.0, "median": 1485.0, "max": 1725.0, "mean": 1517.5, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 124.82, "median": 298.92, "max": 301.24, "mean": 261.57, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 8702.0, "median": 8702.0, "max": 8702.0, "mean": 8702.0, "n": 10 }, "utilization_pct": { "min": 93.0, "median": 98.0, "max": 98.0, "mean": 97.2, "n": 10 }, "temperature_c": { "min": 56.0, "median": 58.0, "max": 59.0, "mean": 58.0, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1410.0, "median": 1485.0, "max": 1515.0, "mean": 1476.0, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 2992.56, "power_mean_w_all_cards": 261.57, "energy_j_per_1k_tokens_all_cards": 2992.56, "watts_per_tok_s": 2.9926, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 59.0, "temp_min_c": 56.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1410.0, "clock_max_mhz": 1515.0, "clock_median_mhz": 1485.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 261.57, "power_pct_of_limit": 87.2, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 87.2% of the cap and the card at 59 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 59.0, "temp_min_c": 56.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1410.0, "clock_max_mhz": 1515.0, "clock_median_mhz": 1485.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 261.57, "power_pct_of_limit": 87.2, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 87.2% of the cap and the card at 59 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 8702.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 20.0, "median": 20.0, "max": 20.0, "mean": 20.0, "n": 2 }, "whole_box_realpower_w": { "min": 201.0, "median": 201.0, "max": 201.0, "mean": 201.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 3 } ], "derived": { "decode_tok_s": { "min": 87.407, "median": 87.62, "max": 87.717, "mean": 87.581, "n": 3 }, "ttft_ms": { "min": 479.46, "median": 507.83, "max": 530.11, "mean": 505.8, "n": 3 }, "power_mean_w": { "min": 259.97, "median": 261.57, "max": 261.68, "mean": 261.07, "n": 3 }, "power_mean_w_all_cards": { "min": 259.97, "median": 261.57, "max": 261.68, "mean": 261.07, "n": 3 }, "energy_j_per_1k_tokens_all_cards": { "min": 2967.02, "median": 2983.24, "max": 2992.56, "mean": 2980.94, "n": 3 }, "energy_j_per_1k_tokens": { "min": 2967.02, "median": 2983.24, "max": 2992.56, "mean": 2980.94, "n": 3 }, "mem_clock_median_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "mem_clock_floor_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "peak_mem_bandwidth_gbs_at_median_clock": { "min": 912.1, "median": 912.1, "max": 912.1, "mean": 912.1, "n": 3 }, "decode_gbs": null, "decode_bandwidth_fraction_of_peak_pct": null, "sw_power_cap_active_fraction": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 3 }, "temp_max_c": { "min": 56.0, "median": 58.0, "max": 59.0, "mean": 57.7, "n": 3 }, "fan_max_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 3 } }, "finished_utc": "2026-10-15T05:37:04Z" }, { "num_ctx": 32768, "started_utc": "2026-10-15T05:37:04Z", "loads": true, "size_total": 10246055525, "size_vram": 10246055525, "context_length_loaded": 32768, "fits_fully_in_vram": true, "verdict": "fits: fully resident on the card", "offload_split": "100.0% VRAM / 0.0% RAM", "vram_delta_mib": { "": 1392.0 }, "memory_used_mib_per_card": { "": 10094.0 }, "memory_used_mib_per_card_before": { "": 8702.0 }, "runner_peak_rss_gib": 0.87, "runner_rss_within_allowance": true, "runner_procs": [ { "pid": 8388, "rss_kb": 909992, "threads": 23 } ], "free_m_at_level": "total used free shared buff/cache available\nMem: 30990 2197 19211 117 10096 28792\nSwap: 8191 0 8191", "kv_footprint_bytes": 2675976438, "kv_footprint_gib": 2.49, "runs": [ { "ttft_ms": 458.02, "first_token_was_thinking": false, "wall_s": 3.3978, "eval_count": 256, "eval_duration_ns": 2934630000, "decode_tok_s": 87.234, "prompt_eval_count": 519, "prompt_eval_duration_ns": 30781000, "prefill_tok_s": 16861.051, "load_duration_ns": 419656158, "total_duration_ns": 3392326749, "streamed_chunks": 256, "response_chars": 1002, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.742, "decode_tok_s_wall_gross": 75.343, "num_ctx_option": 32768, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 114.46, "median": 114.48, "max": 114.53, "mean": 114.49, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 10094.0, "median": 10094.0, "max": 10094.0, "mean": 10094.0, "n": 6 }, "power_w_per_card": { "": 114.49 }, "memory_used_mib_per_card": { "": 10094.0 }, "power_w_all_cards": 114.49, "ups_whole_box": { "load_pct": 25.0, "realpower_w": 321.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 265.0, "max_w": 299.89, "median_w": 298.55, "mean_w_whole_window": 239.94, "max_w_whole_window": 299.89, "limit_w": 300.0, "over_idle_w": 150.51, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.93, "median": 23.34, "max": 23.94, "mean": 23.46, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 25.0, "median": 25.0, "max": 25.0, "mean": 25.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 114.62, "median": 297.14, "max": 299.89, "mean": 239.94, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 10094.0, "median": 10094.0, "max": 10094.0, "mean": 10094.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.6, "n": 12 }, "temperature_c": { "min": 56.0, "median": 59.5, "max": 60.0, "mean": 58.9, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1485.0, "max": 1725.0, "mean": 1517.5, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 134.91, "median": 298.55, "max": 299.89, "mean": 265.0, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 10094.0, "median": 10094.0, "max": 10094.0, "mean": 10094.0, "n": 10 }, "utilization_pct": { "min": 97.0, "median": 98.0, "max": 98.0, "mean": 97.9, "n": 10 }, "temperature_c": { "min": 58.0, "median": 60.0, "max": 60.0, "mean": 59.5, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1477.5, "max": 1500.0, "mean": 1476.0, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 3037.8, "power_mean_w_all_cards": 265.0, "energy_j_per_1k_tokens_all_cards": 3037.8, "watts_per_tok_s": 3.0378, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 60.0, "temp_min_c": 58.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1500.0, "clock_median_mhz": 1477.5, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 265.0, "power_pct_of_limit": 88.3, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "no clock drop during decode (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 60.0, "temp_min_c": 58.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1500.0, "clock_median_mhz": 1477.5, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 265.0, "power_pct_of_limit": 88.3, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "no clock drop during decode (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 10094.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 20.0, "median": 22.5, "max": 25.0, "mean": 22.5, "n": 2 }, "whole_box_realpower_w": { "min": 205.0, "median": 263.0, "max": 321.0, "mean": 263.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 1 }, { "ttft_ms": 513.41, "first_token_was_thinking": false, "wall_s": 3.4544, "eval_count": 256, "eval_duration_ns": 2935981000, "decode_tok_s": 87.194, "prompt_eval_count": 519, "prompt_eval_duration_ns": 15107000, "prefill_tok_s": 34354.935, "load_duration_ns": 490908594, "total_duration_ns": 3448898682, "streamed_chunks": 256, "response_chars": 1002, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.706, "decode_tok_s_wall_gross": 74.109, "num_ctx_option": 32768, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 115.98, "median": 116.16, "max": 116.18, "mean": 116.11, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 10094.0, "median": 10094.0, "max": 10094.0, "mean": 10094.0, "n": 6 }, "power_w_per_card": { "": 116.11 }, "memory_used_mib_per_card": { "": 10094.0 }, "power_w_all_cards": 116.11, "ups_whole_box": { "load_pct": 20.0, "realpower_w": 203.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 262.63, "max_w": 300.73, "median_w": 298.11, "mean_w_whole_window": 238.21, "max_w_whole_window": 300.73, "limit_w": 300.0, "over_idle_w": 146.52, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.92, "median": 23.17, "max": 23.86, "mean": 23.28, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 25.0, "median": 25.0, "max": 25.0, "mean": 25.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 116.1, "median": 297.09, "max": 300.73, "mean": 238.21, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 10094.0, "median": 10094.0, "max": 10094.0, "mean": 10094.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.3, "n": 12 }, "temperature_c": { "min": 57.0, "median": 61.0, "max": 62.0, "mean": 60.1, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1410.0, "median": 1470.0, "max": 1725.0, "mean": 1506.2, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 129.53, "median": 298.11, "max": 300.73, "mean": 262.63, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 10094.0, "median": 10094.0, "max": 10094.0, "mean": 10094.0, "n": 10 }, "utilization_pct": { "min": 95.0, "median": 98.0, "max": 98.0, "mean": 97.6, "n": 10 }, "temperature_c": { "min": 59.0, "median": 61.0, "max": 62.0, "mean": 60.7, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1410.0, "median": 1470.0, "max": 1500.0, "mean": 1462.5, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 3012.02, "power_mean_w_all_cards": 262.63, "energy_j_per_1k_tokens_all_cards": 3012.02, "watts_per_tok_s": 3.012, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 62.0, "temp_min_c": 59.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1410.0, "clock_max_mhz": 1500.0, "clock_median_mhz": 1470.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 262.63, "power_pct_of_limit": 87.5, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 87.5% of the cap and the card at 62 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 62.0, "temp_min_c": 59.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1410.0, "clock_max_mhz": 1500.0, "clock_median_mhz": 1470.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 262.63, "power_pct_of_limit": 87.5, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 87.5% of the cap and the card at 62 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 10094.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 20.0, "median": 20.0, "max": 20.0, "mean": 20.0, "n": 2 }, "whole_box_realpower_w": { "min": 203.0, "median": 203.0, "max": 203.0, "mean": 203.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 2 }, { "ttft_ms": 455.5, "first_token_was_thinking": false, "wall_s": 3.3915, "eval_count": 256, "eval_duration_ns": 2933100000, "decode_tok_s": 87.28, "prompt_eval_count": 519, "prompt_eval_duration_ns": 14823000, "prefill_tok_s": 35013.155, "load_duration_ns": 433668015, "total_duration_ns": 3388102928, "streamed_chunks": 256, "response_chars": 1002, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.852, "decode_tok_s_wall_gross": 75.482, "num_ctx_option": 32768, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 117.22, "median": 117.29, "max": 117.3, "mean": 117.27, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 10094.0, "median": 10094.0, "max": 10094.0, "mean": 10094.0, "n": 6 }, "power_w_per_card": { "": 117.27 }, "memory_used_mib_per_card": { "": 10094.0 }, "power_w_all_cards": 117.27, "ups_whole_box": { "load_pct": 20.0, "realpower_w": 204.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 261.32, "max_w": 301.7, "median_w": 298.54, "mean_w_whole_window": 237.32, "max_w_whole_window": 301.7, "limit_w": 300.0, "over_idle_w": 144.05, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.9, "median": 23.4, "max": 23.88, "mean": 23.42, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 25.0, "median": 25.0, "max": 25.0, "mean": 25.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 117.3, "median": 298.08, "max": 301.7, "mean": 237.32, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 10094.0, "median": 10094.0, "max": 10094.0, "mean": 10094.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.4, "n": 12 }, "temperature_c": { "min": 58.0, "median": 62.0, "max": 63.0, "mean": 61.1, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1470.0, "max": 1725.0, "mean": 1507.5, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 133.38, "median": 298.54, "max": 301.7, "mean": 261.32, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 10094.0, "median": 10094.0, "max": 10094.0, "mean": 10094.0, "n": 10 }, "utilization_pct": { "min": 95.0, "median": 98.0, "max": 98.0, "mean": 97.7, "n": 10 }, "temperature_c": { "min": 60.0, "median": 62.0, "max": 63.0, "mean": 61.7, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1470.0, "max": 1485.0, "mean": 1464.0, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 2994.05, "power_mean_w_all_cards": 261.32, "energy_j_per_1k_tokens_all_cards": 2994.05, "watts_per_tok_s": 2.994, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 63.0, "temp_min_c": 60.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1485.0, "clock_median_mhz": 1470.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 261.32, "power_pct_of_limit": 87.1, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "no clock drop during decode (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 63.0, "temp_min_c": 60.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1485.0, "clock_median_mhz": 1470.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 261.32, "power_pct_of_limit": 87.1, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "no clock drop during decode (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 10094.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 20.0, "median": 20.0, "max": 20.0, "mean": 20.0, "n": 2 }, "whole_box_realpower_w": { "min": 204.0, "median": 204.0, "max": 204.0, "mean": 204.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 3 } ], "derived": { "decode_tok_s": { "min": 87.194, "median": 87.234, "max": 87.28, "mean": 87.236, "n": 3 }, "ttft_ms": { "min": 455.5, "median": 458.02, "max": 513.41, "mean": 475.64, "n": 3 }, "power_mean_w": { "min": 261.32, "median": 262.63, "max": 265.0, "mean": 262.98, "n": 3 }, "power_mean_w_all_cards": { "min": 261.32, "median": 262.63, "max": 265.0, "mean": 262.98, "n": 3 }, "energy_j_per_1k_tokens_all_cards": { "min": 2994.05, "median": 3012.02, "max": 3037.8, "mean": 3014.62, "n": 3 }, "energy_j_per_1k_tokens": { "min": 2994.05, "median": 3012.02, "max": 3037.8, "mean": 3014.62, "n": 3 }, "mem_clock_median_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "mem_clock_floor_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "peak_mem_bandwidth_gbs_at_median_clock": { "min": 912.1, "median": 912.1, "max": 912.1, "mean": 912.1, "n": 3 }, "decode_gbs": null, "decode_bandwidth_fraction_of_peak_pct": null, "sw_power_cap_active_fraction": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 3 }, "temp_max_c": { "min": 60.0, "median": 62.0, "max": 63.0, "mean": 61.7, "n": 3 }, "fan_max_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 3 } }, "finished_utc": "2026-10-15T05:37:59Z" }, { "num_ctx": 49152, "started_utc": "2026-10-15T05:37:59Z", "loads": true, "size_total": 11747616357, "size_vram": 11747616357, "context_length_loaded": 49152, "fits_fully_in_vram": true, "verdict": "fits: fully resident on the card", "offload_split": "100.0% VRAM / 0.0% RAM", "vram_delta_mib": { "": 1432.0 }, "memory_used_mib_per_card": { "": 11526.0 }, "memory_used_mib_per_card_before": { "": 10094.0 }, "runner_peak_rss_gib": 0.9, "runner_rss_within_allowance": true, "runner_procs": [ { "pid": 8955, "rss_kb": 943428, "threads": 23 } ], "free_m_at_level": "total used free shared buff/cache available\nMem: 30990 2249 19159 149 10128 28741\nSwap: 8191 0 8191", "kv_footprint_bytes": 4177537270, "kv_footprint_gib": 3.89, "runs": [ { "ttft_ms": 452.49, "first_token_was_thinking": false, "wall_s": 3.3936, "eval_count": 256, "eval_duration_ns": 2937107000, "decode_tok_s": 87.161, "prompt_eval_count": 519, "prompt_eval_duration_ns": 30388000, "prefill_tok_s": 17079.11, "load_duration_ns": 414592343, "total_duration_ns": 3388870070, "streamed_chunks": 256, "response_chars": 1002, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.701, "decode_tok_s_wall_gross": 75.436, "num_ctx_option": 49152, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 118.22, "median": 118.37, "max": 118.54, "mean": 118.37, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 11526.0, "median": 11526.0, "max": 11526.0, "mean": 11526.0, "n": 6 }, "power_w_per_card": { "": 118.37 }, "memory_used_mib_per_card": { "": 11526.0 }, "power_w_all_cards": 118.37, "ups_whole_box": { "load_pct": 22.0, "realpower_w": 228.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 264.12, "max_w": 302.0, "median_w": 298.58, "mean_w_whole_window": 239.85, "max_w_whole_window": 302.0, "limit_w": 300.0, "over_idle_w": 145.75, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.89, "median": 23.19, "max": 23.95, "mean": 23.39, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 26.0, "median": 26.0, "max": 26.0, "mean": 26.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 118.54, "median": 297.69, "max": 302.0, "mean": 239.85, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 11526.0, "median": 11526.0, "max": 11526.0, "mean": 11526.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.4, "n": 12 }, "temperature_c": { "min": 59.0, "median": 63.0, "max": 64.0, "mean": 62.3, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1470.0, "max": 1725.0, "mean": 1505.0, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 132.13, "median": 298.58, "max": 302.0, "mean": 264.12, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 11526.0, "median": 11526.0, "max": 11526.0, "mean": 11526.0, "n": 10 }, "utilization_pct": { "min": 95.0, "median": 98.0, "max": 98.0, "mean": 97.7, "n": 10 }, "temperature_c": { "min": 61.0, "median": 63.0, "max": 64.0, "mean": 63.0, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1470.0, "max": 1485.0, "mean": 1461.0, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 3030.27, "power_mean_w_all_cards": 264.12, "energy_j_per_1k_tokens_all_cards": 3030.27, "watts_per_tok_s": 3.0303, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 61.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1485.0, "clock_median_mhz": 1470.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 264.12, "power_pct_of_limit": 88.0, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "no clock drop during decode (the fans never spun up -- zero-RPM idle mode)" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 61.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 0.0, "fan_max_pct": 0.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1485.0, "clock_median_mhz": 1470.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 264.12, "power_pct_of_limit": 88.0, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "no clock drop during decode (the fans never spun up -- zero-RPM idle mode)" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 11526.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 22.0, "median": 22.0, "max": 22.0, "mean": 22.0, "n": 2 }, "whole_box_realpower_w": { "min": 228.0, "median": 228.0, "max": 228.0, "mean": 228.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 1 }, { "ttft_ms": 511.34, "first_token_was_thinking": false, "wall_s": 3.459, "eval_count": 256, "eval_duration_ns": 2942419000, "decode_tok_s": 87.003, "prompt_eval_count": 519, "prompt_eval_duration_ns": 14770000, "prefill_tok_s": 35138.795, "load_duration_ns": 489629376, "total_duration_ns": 3453184789, "streamed_chunks": 256, "response_chars": 1002, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.508, "decode_tok_s_wall_gross": 74.009, "num_ctx_option": 49152, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 119.23, "median": 119.77, "max": 119.94, "mean": 119.65, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 11526.0, "median": 11526.0, "max": 11526.0, "mean": 11526.0, "n": 6 }, "power_w_per_card": { "": 119.65 }, "memory_used_mib_per_card": { "": 11526.0 }, "power_w_all_cards": 119.65, "ups_whole_box": { "load_pct": 40.0, "realpower_w": 408.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 264.81, "max_w": 301.64, "median_w": 298.53, "mean_w_whole_window": 240.58, "max_w_whole_window": 301.64, "limit_w": 300.0, "over_idle_w": 145.16, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.92, "median": 23.26, "max": 23.8, "mean": 23.31, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 26.0, "median": 26.0, "max": 26.0, "mean": 26.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 119.44, "median": 297.28, "max": 301.64, "mean": 240.58, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 11526.0, "median": 11526.0, "max": 11526.0, "mean": 11526.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.7, "n": 12 }, "temperature_c": { "min": 60.0, "median": 63.5, "max": 64.0, "mean": 62.9, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 38.0, "median": 38.0, "max": 38.0, "mean": 38.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1395.0, "median": 1470.0, "max": 1725.0, "mean": 1500.0, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 136.26, "median": 298.53, "max": 301.64, "mean": 264.81, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 11526.0, "median": 11526.0, "max": 11526.0, "mean": 11526.0, "n": 10 }, "utilization_pct": { "min": 98.0, "median": 98.0, "max": 98.0, "mean": 98.0, "n": 10 }, "temperature_c": { "min": 62.0, "median": 64.0, "max": 64.0, "mean": 63.5, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 38.0, "median": 38.0, "max": 38.0, "mean": 38.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1395.0, "median": 1462.5, "max": 1485.0, "mean": 1455.0, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 3043.68, "power_mean_w_all_cards": 264.81, "energy_j_per_1k_tokens_all_cards": 3043.68, "watts_per_tok_s": 3.0437, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 62.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 38.0, "fan_max_pct": 38.0, "clock_floor_mhz": 1395.0, "clock_max_mhz": 1485.0, "clock_median_mhz": 1462.5, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 264.81, "power_pct_of_limit": 88.3, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "NEITHER cap: the SM clock varied with draw at 88.3% of the cap and the card at 64 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 62.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 38.0, "fan_max_pct": 38.0, "clock_floor_mhz": 1395.0, "clock_max_mhz": 1485.0, "clock_median_mhz": 1462.5, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 264.81, "power_pct_of_limit": 88.3, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "NEITHER cap: the SM clock varied with draw at 88.3% of the cap and the card at 64 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 11526.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 20.0, "median": 30.0, "max": 40.0, "mean": 30.0, "n": 2 }, "whole_box_realpower_w": { "min": 207.0, "median": 307.5, "max": 408.0, "mean": 307.5, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 2 }, { "ttft_ms": 460.24, "first_token_was_thinking": false, "wall_s": 3.41, "eval_count": 256, "eval_duration_ns": 2944852000, "decode_tok_s": 86.931, "prompt_eval_count": 519, "prompt_eval_duration_ns": 14771000, "prefill_tok_s": 35136.416, "load_duration_ns": 438404025, "total_duration_ns": 3404646824, "streamed_chunks": 256, "response_chars": 1002, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.448, "decode_tok_s_wall_gross": 75.073, "num_ctx_option": 49152, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 119.91, "median": 119.95, "max": 119.95, "mean": 119.94, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 11526.0, "median": 11526.0, "max": 11526.0, "mean": 11526.0, "n": 6 }, "power_w_per_card": { "": 119.94 }, "memory_used_mib_per_card": { "": 11526.0 }, "power_w_all_cards": 119.94, "ups_whole_box": { "load_pct": 20.0, "realpower_w": 206.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 262.74, "max_w": 300.92, "median_w": 298.86, "mean_w_whole_window": 238.95, "max_w_whole_window": 300.92, "limit_w": 300.0, "over_idle_w": 142.8, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.96, "median": 23.37, "max": 23.85, "mean": 23.39, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 26.0, "median": 26.0, "max": 26.0, "mean": 26.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 120.01, "median": 297.97, "max": 300.92, "mean": 238.95, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 11526.0, "median": 11526.0, "max": 11526.0, "mean": 11526.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 80.7, "n": 12 }, "temperature_c": { "min": 60.0, "median": 63.0, "max": 64.0, "mean": 62.8, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 49.0, "median": 49.0, "max": 49.0, "mean": 49.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1455.0, "max": 1725.0, "mean": 1512.5, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 138.8, "median": 298.86, "max": 300.92, "mean": 262.74, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 11526.0, "median": 11526.0, "max": 11526.0, "mean": 11526.0, "n": 10 }, "utilization_pct": { "min": 86.0, "median": 98.0, "max": 98.0, "mean": 96.8, "n": 10 }, "temperature_c": { "min": 62.0, "median": 63.0, "max": 64.0, "mean": 63.3, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 49.0, "median": 49.0, "max": 49.0, "mean": 49.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1425.0, "median": 1455.0, "max": 1620.0, "mean": 1470.0, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 3022.38, "power_mean_w_all_cards": 262.74, "energy_j_per_1k_tokens_all_cards": 3022.38, "watts_per_tok_s": 3.0224, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 62.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 49.0, "fan_max_pct": 49.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1620.0, "clock_median_mhz": 1455.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 262.74, "power_pct_of_limit": 87.6, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "NEITHER cap: the SM clock varied with draw at 87.6% of the cap and the card at 64 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 62.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 49.0, "fan_max_pct": 49.0, "clock_floor_mhz": 1425.0, "clock_max_mhz": 1620.0, "clock_median_mhz": 1455.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 262.74, "power_pct_of_limit": 87.6, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "NEITHER cap: the SM clock varied with draw at 87.6% of the cap and the card at 64 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 11526.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 20.0, "median": 20.0, "max": 20.0, "mean": 20.0, "n": 2 }, "whole_box_realpower_w": { "min": 206.0, "median": 206.0, "max": 206.0, "mean": 206.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 3 } ], "derived": { "decode_tok_s": { "min": 86.931, "median": 87.003, "max": 87.161, "mean": 87.032, "n": 3 }, "ttft_ms": { "min": 452.49, "median": 460.24, "max": 511.34, "mean": 474.69, "n": 3 }, "power_mean_w": { "min": 262.74, "median": 264.12, "max": 264.81, "mean": 263.89, "n": 3 }, "power_mean_w_all_cards": { "min": 262.74, "median": 264.12, "max": 264.81, "mean": 263.89, "n": 3 }, "energy_j_per_1k_tokens_all_cards": { "min": 3022.38, "median": 3030.27, "max": 3043.68, "mean": 3032.11, "n": 3 }, "energy_j_per_1k_tokens": { "min": 3022.38, "median": 3030.27, "max": 3043.68, "mean": 3032.11, "n": 3 }, "mem_clock_median_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "mem_clock_floor_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "peak_mem_bandwidth_gbs_at_median_clock": { "min": 912.1, "median": 912.1, "max": 912.1, "mean": 912.1, "n": 3 }, "decode_gbs": null, "decode_bandwidth_fraction_of_peak_pct": null, "sw_power_cap_active_fraction": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 3 }, "temp_max_c": { "min": 64.0, "median": 64.0, "max": 64.0, "mean": 64.0, "n": 3 }, "fan_max_pct": { "min": 0.0, "median": 38.0, "max": 49.0, "mean": 29.0, "n": 3 } }, "finished_utc": "2026-10-15T05:38:53Z" }, { "num_ctx": 65536, "started_utc": "2026-10-15T05:38:53Z", "loads": true, "size_total": 13169474927, "size_vram": 13169474927, "context_length_loaded": 65536, "fits_fully_in_vram": true, "verdict": "fits: fully resident on the card", "offload_split": "100.0% VRAM / 0.0% RAM", "vram_delta_mib": { "": 1350.0 }, "memory_used_mib_per_card": { "": 12876.0 }, "memory_used_mib_per_card_before": { "": 11526.0 }, "runner_peak_rss_gib": 0.85, "runner_rss_within_allowance": true, "runner_procs": [ { "pid": 9523, "rss_kb": 895432, "threads": 23 } ], "free_m_at_level": "total used free shared buff/cache available\nMem: 30990 2158 19250 97 10076 28831\nSwap: 8191 0 8191", "kv_footprint_bytes": 5599395840, "kv_footprint_gib": 5.21, "runs": [ { "ttft_ms": 490.6, "first_token_was_thinking": false, "wall_s": 3.4395, "eval_count": 256, "eval_duration_ns": 2943774000, "decode_tok_s": 86.963, "prompt_eval_count": 519, "prompt_eval_duration_ns": 29944000, "prefill_tok_s": 17332.354, "load_duration_ns": 453835069, "total_duration_ns": 3434062313, "streamed_chunks": 256, "response_chars": 1013, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.472, "decode_tok_s_wall_gross": 74.429, "num_ctx_option": 65536, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 119.21, "median": 119.35, "max": 119.35, "mean": 119.3, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 12876.0, "median": 12876.0, "max": 12876.0, "mean": 12876.0, "n": 6 }, "power_w_per_card": { "": 119.3 }, "memory_used_mib_per_card": { "": 12876.0 }, "power_w_all_cards": 119.3, "ups_whole_box": { "load_pct": 21.0, "realpower_w": 215.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 263.52, "max_w": 300.41, "median_w": 298.67, "mean_w_whole_window": 239.45, "max_w_whole_window": 300.41, "limit_w": 300.0, "over_idle_w": 144.22, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.86, "median": 23.24, "max": 23.81, "mean": 23.31, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 26.0, "median": 26.0, "max": 26.0, "mean": 26.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 119.09, "median": 297.69, "max": 300.41, "mean": 239.45, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 12876.0, "median": 12876.0, "max": 12876.0, "mean": 12876.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.2, "n": 12 }, "temperature_c": { "min": 59.0, "median": 62.5, "max": 63.0, "mean": 61.9, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 49.0, "median": 49.5, "max": 50.0, "mean": 49.5, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1395.0, "median": 1470.0, "max": 1725.0, "mean": 1497.5, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 132.9, "median": 298.67, "max": 300.41, "mean": 263.52, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 12876.0, "median": 12876.0, "max": 12876.0, "mean": 12876.0, "n": 10 }, "utilization_pct": { "min": 95.0, "median": 98.0, "max": 98.0, "mean": 97.5, "n": 10 }, "temperature_c": { "min": 61.0, "median": 63.0, "max": 63.0, "mean": 62.5, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 49.0, "median": 49.0, "max": 50.0, "mean": 49.4, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1395.0, "median": 1462.5, "max": 1470.0, "mean": 1452.0, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 3030.25, "power_mean_w_all_cards": 263.52, "energy_j_per_1k_tokens_all_cards": 3030.25, "watts_per_tok_s": 3.0303, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 63.0, "temp_min_c": 61.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 49.4, "fan_max_pct": 50.0, "clock_floor_mhz": 1395.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1462.5, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 263.52, "power_pct_of_limit": 87.8, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "NEITHER cap: the SM clock varied with draw at 87.8% of the cap and the card at 63 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 63.0, "temp_min_c": 61.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 49.4, "fan_max_pct": 50.0, "clock_floor_mhz": 1395.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1462.5, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 263.52, "power_pct_of_limit": 87.8, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "NEITHER cap: the SM clock varied with draw at 87.8% of the cap and the card at 63 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 12876.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 21.0, "median": 21.0, "max": 21.0, "mean": 21.0, "n": 2 }, "whole_box_realpower_w": { "min": 215.0, "median": 215.0, "max": 215.0, "mean": 215.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 1 }, { "ttft_ms": 512.36, "first_token_was_thinking": false, "wall_s": 3.4587, "eval_count": 256, "eval_duration_ns": 2943072000, "decode_tok_s": 86.984, "prompt_eval_count": 519, "prompt_eval_duration_ns": 14837000, "prefill_tok_s": 34980.117, "load_duration_ns": 490092552, "total_duration_ns": 3455365459, "streamed_chunks": 256, "response_chars": 1013, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.547, "decode_tok_s_wall_gross": 74.015, "num_ctx_option": 65536, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 119.01, "median": 119.02, "max": 119.04, "mean": 119.02, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 12876.0, "median": 12876.0, "max": 12876.0, "mean": 12876.0, "n": 6 }, "power_w_per_card": { "": 119.02 }, "memory_used_mib_per_card": { "": 12876.0 }, "power_w_all_cards": 119.02, "ups_whole_box": { "load_pct": 41.0, "realpower_w": 312.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 259.14, "max_w": 301.16, "median_w": 298.44, "mean_w_whole_window": 235.78, "max_w_whole_window": 301.16, "limit_w": 300.0, "over_idle_w": 140.12, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.96, "median": 23.27, "max": 23.85, "mean": 23.34, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 26.0, "median": 26.0, "max": 26.0, "mean": 26.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 118.96, "median": 297.23, "max": 301.16, "mean": 235.78, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 12876.0, "median": 12876.0, "max": 12876.0, "mean": 12876.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.0, "n": 12 }, "temperature_c": { "min": 59.0, "median": 62.0, "max": 63.0, "mean": 61.8, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 48.0, "median": 48.0, "max": 49.0, "mean": 48.1, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1395.0, "median": 1470.0, "max": 1725.0, "mean": 1503.8, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 127.77, "median": 298.44, "max": 301.16, "mean": 259.14, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 12876.0, "median": 12876.0, "max": 12876.0, "mean": 12876.0, "n": 10 }, "utilization_pct": { "min": 90.0, "median": 98.0, "max": 98.0, "mean": 97.2, "n": 10 }, "temperature_c": { "min": 61.0, "median": 62.5, "max": 63.0, "mean": 62.4, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 48.0, "median": 48.0, "max": 48.0, "mean": 48.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1395.0, "median": 1470.0, "max": 1485.0, "mean": 1459.5, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 2979.17, "power_mean_w_all_cards": 259.14, "energy_j_per_1k_tokens_all_cards": 2979.17, "watts_per_tok_s": 2.9792, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 63.0, "temp_min_c": 61.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 48.0, "fan_max_pct": 48.0, "clock_floor_mhz": 1395.0, "clock_max_mhz": 1485.0, "clock_median_mhz": 1470.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 259.14, "power_pct_of_limit": 86.4, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "NEITHER cap: the SM clock varied with draw at 86.4% of the cap and the card at 63 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 63.0, "temp_min_c": 61.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 48.0, "fan_max_pct": 48.0, "clock_floor_mhz": 1395.0, "clock_max_mhz": 1485.0, "clock_median_mhz": 1470.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 259.14, "power_pct_of_limit": 86.4, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "NEITHER cap: the SM clock varied with draw at 86.4% of the cap and the card at 63 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 12876.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 39.0, "median": 40.0, "max": 41.0, "mean": 40.0, "n": 2 }, "whole_box_realpower_w": { "min": 312.0, "median": 353.0, "max": 394.0, "mean": 353.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 2 }, { "ttft_ms": 525.02, "first_token_was_thinking": false, "wall_s": 3.4706, "eval_count": 256, "eval_duration_ns": 2942213000, "decode_tok_s": 87.009, "prompt_eval_count": 519, "prompt_eval_duration_ns": 14770000, "prefill_tok_s": 35138.795, "load_duration_ns": 503405015, "total_duration_ns": 3467184745, "streamed_chunks": 256, "response_chars": 1013, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.57, "decode_tok_s_wall_gross": 73.762, "num_ctx_option": 65536, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 119.41, "median": 119.47, "max": 119.49, "mean": 119.46, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 12876.0, "median": 12876.0, "max": 12876.0, "mean": 12876.0, "n": 6 }, "power_w_per_card": { "": 119.46 }, "memory_used_mib_per_card": { "": 12876.0 }, "power_w_all_cards": 119.46, "ups_whole_box": { "load_pct": 20.0, "realpower_w": 206.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 263.46, "max_w": 300.88, "median_w": 299.11, "mean_w_whole_window": 239.44, "max_w_whole_window": 300.88, "limit_w": 300.0, "over_idle_w": 144.0, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.91, "median": 23.26, "max": 23.85, "mean": 23.33, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 26.0, "median": 26.0, "max": 26.0, "mean": 26.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 119.33, "median": 297.96, "max": 300.88, "mean": 239.44, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 12876.0, "median": 12876.0, "max": 12876.0, "mean": 12876.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.6, "n": 12 }, "temperature_c": { "min": 59.0, "median": 62.5, "max": 63.0, "mean": 61.8, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 46.0, "median": 47.0, "max": 47.0, "mean": 46.8, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1395.0, "median": 1455.0, "max": 1725.0, "mean": 1495.0, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 132.58, "median": 299.11, "max": 300.88, "mean": 263.46, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 12876.0, "median": 12876.0, "max": 12876.0, "mean": 12876.0, "n": 10 }, "utilization_pct": { "min": 97.0, "median": 98.0, "max": 98.0, "mean": 97.9, "n": 10 }, "temperature_c": { "min": 60.0, "median": 63.0, "max": 63.0, "mean": 62.4, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 46.0, "median": 47.0, "max": 47.0, "mean": 46.8, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1395.0, "median": 1455.0, "max": 1470.0, "mean": 1449.0, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 3027.95, "power_mean_w_all_cards": 263.46, "energy_j_per_1k_tokens_all_cards": 3027.95, "watts_per_tok_s": 3.028, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 63.0, "temp_min_c": 60.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 46.8, "fan_max_pct": 47.0, "clock_floor_mhz": 1395.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1455.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 263.46, "power_pct_of_limit": 87.8, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 87.8% of the cap and the card at 63 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 63.0, "temp_min_c": 60.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 46.8, "fan_max_pct": 47.0, "clock_floor_mhz": 1395.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1455.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 263.46, "power_pct_of_limit": 87.8, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 87.8% of the cap and the card at 63 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 12876.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 20.0, "median": 20.0, "max": 20.0, "mean": 20.0, "n": 2 }, "whole_box_realpower_w": { "min": 206.0, "median": 206.0, "max": 206.0, "mean": 206.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 3 } ], "derived": { "decode_tok_s": { "min": 86.963, "median": 86.984, "max": 87.009, "mean": 86.985, "n": 3 }, "ttft_ms": { "min": 490.6, "median": 512.36, "max": 525.02, "mean": 509.33, "n": 3 }, "power_mean_w": { "min": 259.14, "median": 263.46, "max": 263.52, "mean": 262.04, "n": 3 }, "power_mean_w_all_cards": { "min": 259.14, "median": 263.46, "max": 263.52, "mean": 262.04, "n": 3 }, "energy_j_per_1k_tokens_all_cards": { "min": 2979.17, "median": 3027.95, "max": 3030.25, "mean": 3012.46, "n": 3 }, "energy_j_per_1k_tokens": { "min": 2979.17, "median": 3027.95, "max": 3030.25, "mean": 3012.46, "n": 3 }, "mem_clock_median_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "mem_clock_floor_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "peak_mem_bandwidth_gbs_at_median_clock": { "min": 912.1, "median": 912.1, "max": 912.1, "mean": 912.1, "n": 3 }, "decode_gbs": null, "decode_bandwidth_fraction_of_peak_pct": null, "sw_power_cap_active_fraction": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 3 }, "temp_max_c": { "min": 63.0, "median": 63.0, "max": 63.0, "mean": 63.0, "n": 3 }, "fan_max_pct": { "min": 47.0, "median": 48.0, "max": 50.0, "mean": 48.3, "n": 3 } }, "finished_utc": "2026-10-15T05:39:48Z" }, { "num_ctx": 98304, "started_utc": "2026-10-15T05:39:48Z", "loads": true, "size_total": 16189373807, "size_vram": 16189373807, "context_length_loaded": 98304, "fits_fully_in_vram": true, "verdict": "fits: fully resident on the card", "offload_split": "100.0% VRAM / 0.0% RAM", "vram_delta_mib": { "": 2880.0 }, "memory_used_mib_per_card": { "": 15756.0 }, "memory_used_mib_per_card_before": { "": 12876.0 }, "runner_peak_rss_gib": 0.89, "runner_rss_within_allowance": true, "runner_procs": [ { "pid": 10103, "rss_kb": 929840, "threads": 23 } ], "free_m_at_level": "total used free shared buff/cache available\nMem: 30990 2234 19204 129 10078 28755\nSwap: 8191 0 8191", "kv_footprint_bytes": 8619294720, "kv_footprint_gib": 8.03, "runs": [ { "ttft_ms": 481.49, "first_token_was_thinking": false, "wall_s": 3.4287, "eval_count": 256, "eval_duration_ns": 2941948000, "decode_tok_s": 87.017, "prompt_eval_count": 519, "prompt_eval_duration_ns": 30281000, "prefill_tok_s": 17139.46, "load_duration_ns": 444125964, "total_duration_ns": 3423336059, "streamed_chunks": 256, "response_chars": 1013, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.521, "decode_tok_s_wall_gross": 74.663, "num_ctx_option": 98304, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 119.15, "median": 119.19, "max": 119.3, "mean": 119.21, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 15756.0, "median": 15756.0, "max": 15756.0, "mean": 15756.0, "n": 6 }, "power_w_per_card": { "": 119.21 }, "memory_used_mib_per_card": { "": 15756.0 }, "power_w_all_cards": 119.21, "ups_whole_box": { "load_pct": 20.0, "realpower_w": 205.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 264.36, "max_w": 299.65, "median_w": 298.53, "mean_w_whole_window": 240.17, "max_w_whole_window": 299.65, "limit_w": 300.0, "over_idle_w": 145.15, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 23.02, "median": 23.35, "max": 23.93, "mean": 23.46, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 26.0, "median": 26.0, "max": 26.0, "mean": 26.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 119.22, "median": 297.37, "max": 299.65, "mean": 240.17, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 15756.0, "median": 15756.0, "max": 15756.0, "mean": 15756.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.4, "n": 12 }, "temperature_c": { "min": 59.0, "median": 63.0, "max": 63.0, "mean": 62.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 45.0, "median": 45.0, "max": 46.0, "mean": 45.2, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1410.0, "median": 1470.0, "max": 1725.0, "mean": 1498.8, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 135.44, "median": 298.53, "max": 299.65, "mean": 264.36, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 15756.0, "median": 15756.0, "max": 15756.0, "mean": 15756.0, "n": 10 }, "utilization_pct": { "min": 95.0, "median": 98.0, "max": 98.0, "mean": 97.7, "n": 10 }, "temperature_c": { "min": 61.0, "median": 63.0, "max": 63.0, "mean": 62.6, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 45.0, "median": 45.0, "max": 46.0, "mean": 45.1, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1410.0, "median": 1462.5, "max": 1470.0, "mean": 1453.5, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 3038.02, "power_mean_w_all_cards": 264.36, "energy_j_per_1k_tokens_all_cards": 3038.02, "watts_per_tok_s": 3.038, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 63.0, "temp_min_c": 61.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 45.1, "fan_max_pct": 46.0, "clock_floor_mhz": 1410.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1462.5, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 264.36, "power_pct_of_limit": 88.1, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "no clock drop during decode" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 63.0, "temp_min_c": 61.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 45.1, "fan_max_pct": 46.0, "clock_floor_mhz": 1410.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1462.5, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 264.36, "power_pct_of_limit": 88.1, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "no clock drop during decode" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 15756.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 20.0, "median": 20.0, "max": 20.0, "mean": 20.0, "n": 2 }, "whole_box_realpower_w": { "min": 205.0, "median": 205.0, "max": 205.0, "mean": 205.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 1 }, { "ttft_ms": 531.31, "first_token_was_thinking": false, "wall_s": 3.4808, "eval_count": 256, "eval_duration_ns": 2944771000, "decode_tok_s": 86.934, "prompt_eval_count": 519, "prompt_eval_duration_ns": 14537000, "prefill_tok_s": 35702.002, "load_duration_ns": 509234634, "total_duration_ns": 3475579400, "streamed_chunks": 256, "response_chars": 1013, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.456, "decode_tok_s_wall_gross": 73.547, "num_ctx_option": 98304, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 119.36, "median": 119.47, "max": 119.61, "mean": 119.48, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 15756.0, "median": 15756.0, "max": 15756.0, "mean": 15756.0, "n": 6 }, "power_w_per_card": { "": 119.48 }, "memory_used_mib_per_card": { "": 15756.0 }, "power_w_all_cards": 119.48, "ups_whole_box": { "load_pct": 22.0, "realpower_w": 221.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 252.12, "max_w": 299.78, "median_w": 297.56, "mean_w_whole_window": 230.04, "max_w_whole_window": 299.78, "limit_w": 300.0, "over_idle_w": 132.64, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.99, "median": 23.3, "max": 23.97, "mean": 23.4, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 27.0, "median": 27.0, "max": 27.0, "mean": 27.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 119.65, "median": 263.34, "max": 299.78, "mean": 230.04, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 15756.0, "median": 15756.0, "max": 15756.0, "mean": 15756.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.2, "n": 12 }, "temperature_c": { "min": 59.0, "median": 63.0, "max": 64.0, "mean": 62.2, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 45.0, "median": 45.0, "max": 45.0, "mean": 45.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1380.0, "median": 1455.0, "max": 1725.0, "mean": 1488.8, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 127.77, "median": 297.56, "max": 299.78, "mean": 252.12, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 15756.0, "median": 15756.0, "max": 15756.0, "mean": 15756.0, "n": 10 }, "utilization_pct": { "min": 92.0, "median": 98.0, "max": 98.0, "mean": 97.4, "n": 10 }, "temperature_c": { "min": 61.0, "median": 63.0, "max": 64.0, "mean": 62.9, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 45.0, "median": 45.0, "max": 45.0, "mean": 45.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1380.0, "median": 1455.0, "max": 1470.0, "mean": 1441.5, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 2900.14, "power_mean_w_all_cards": 252.12, "energy_j_per_1k_tokens_all_cards": 2900.14, "watts_per_tok_s": 2.9001, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 61.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 45.0, "fan_max_pct": 45.0, "clock_floor_mhz": 1380.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1455.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 252.12, "power_pct_of_limit": 84.0, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 84.0% of the cap and the card at 64 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 61.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 45.0, "fan_max_pct": 45.0, "clock_floor_mhz": 1380.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1455.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 252.12, "power_pct_of_limit": 84.0, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 84.0% of the cap and the card at 64 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 15756.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 22.0, "median": 22.0, "max": 22.0, "mean": 22.0, "n": 2 }, "whole_box_realpower_w": { "min": 221.0, "median": 221.0, "max": 221.0, "mean": 221.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 2 }, { "ttft_ms": 465.94, "first_token_was_thinking": false, "wall_s": 3.4184, "eval_count": 256, "eval_duration_ns": 2947606000, "decode_tok_s": 86.85, "prompt_eval_count": 519, "prompt_eval_duration_ns": 14995000, "prefill_tok_s": 34611.537, "load_duration_ns": 443774287, "total_duration_ns": 3413063159, "streamed_chunks": 256, "response_chars": 1013, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.368, "decode_tok_s_wall_gross": 74.888, "num_ctx_option": 98304, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 119.67, "median": 119.78, "max": 119.96, "mean": 119.8, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 15756.0, "median": 15756.0, "max": 15756.0, "mean": 15756.0, "n": 6 }, "power_w_per_card": { "": 119.8 }, "memory_used_mib_per_card": { "": 15756.0 }, "power_w_all_cards": 119.8, "ups_whole_box": { "load_pct": 40.0, "realpower_w": 405.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 265.31, "max_w": 301.71, "median_w": 298.16, "mean_w_whole_window": 241.06, "max_w_whole_window": 301.71, "limit_w": 300.0, "over_idle_w": 145.51, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.92, "median": 23.27, "max": 23.76, "mean": 23.32, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 27.0, "median": 27.0, "max": 27.0, "mean": 27.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 119.82, "median": 297.14, "max": 301.71, "mean": 241.06, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 15756.0, "median": 15756.0, "max": 15756.0, "mean": 15756.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 80.8, "n": 12 }, "temperature_c": { "min": 60.0, "median": 63.0, "max": 64.0, "mean": 62.8, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 45.0, "median": 45.0, "max": 45.0, "mean": 45.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1410.0, "median": 1462.5, "max": 1725.0, "mean": 1497.5, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 137.68, "median": 298.16, "max": 301.71, "mean": 265.31, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 15756.0, "median": 15756.0, "max": 15756.0, "mean": 15756.0, "n": 10 }, "utilization_pct": { "min": 88.0, "median": 98.0, "max": 98.0, "mean": 96.9, "n": 10 }, "temperature_c": { "min": 62.0, "median": 63.0, "max": 64.0, "mean": 63.3, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 45.0, "median": 45.0, "max": 45.0, "mean": 45.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1410.0, "median": 1455.0, "max": 1470.0, "mean": 1452.0, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 3054.8, "power_mean_w_all_cards": 265.31, "energy_j_per_1k_tokens_all_cards": 3054.8, "watts_per_tok_s": 3.0548, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 62.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 45.0, "fan_max_pct": 45.0, "clock_floor_mhz": 1410.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1455.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 265.31, "power_pct_of_limit": 88.4, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "no clock drop during decode" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 62.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 45.0, "fan_max_pct": 45.0, "clock_floor_mhz": 1410.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1455.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": false, "power_limit_w": 300.0, "power_mean_w": 265.31, "power_pct_of_limit": 88.4, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "no clock drop during decode" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 15756.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 20.0, "median": 30.0, "max": 40.0, "mean": 30.0, "n": 2 }, "whole_box_realpower_w": { "min": 205.0, "median": 305.0, "max": 405.0, "mean": 305.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 3 } ], "derived": { "decode_tok_s": { "min": 86.85, "median": 86.934, "max": 87.017, "mean": 86.934, "n": 3 }, "ttft_ms": { "min": 465.94, "median": 481.49, "max": 531.31, "mean": 492.91, "n": 3 }, "power_mean_w": { "min": 252.12, "median": 264.36, "max": 265.31, "mean": 260.6, "n": 3 }, "power_mean_w_all_cards": { "min": 252.12, "median": 264.36, "max": 265.31, "mean": 260.6, "n": 3 }, "energy_j_per_1k_tokens_all_cards": { "min": 2900.14, "median": 3038.02, "max": 3054.8, "mean": 2997.65, "n": 3 }, "energy_j_per_1k_tokens": { "min": 2900.14, "median": 3038.02, "max": 3054.8, "mean": 2997.65, "n": 3 }, "mem_clock_median_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "mem_clock_floor_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "peak_mem_bandwidth_gbs_at_median_clock": { "min": 912.1, "median": 912.1, "max": 912.1, "mean": 912.1, "n": 3 }, "decode_gbs": null, "decode_bandwidth_fraction_of_peak_pct": null, "sw_power_cap_active_fraction": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 3 }, "temp_max_c": { "min": 63.0, "median": 64.0, "max": 64.0, "mean": 63.7, "n": 3 }, "fan_max_pct": { "min": 45.0, "median": 45.0, "max": 46.0, "mean": 45.3, "n": 3 } }, "finished_utc": "2026-10-15T05:40:43Z" }, { "num_ctx": 131072, "started_utc": "2026-10-15T05:40:43Z", "loads": true, "size_total": 19209272687, "size_vram": 19209272687, "context_length_loaded": 131072, "fits_fully_in_vram": true, "verdict": "fits: fully resident on the card", "offload_split": "100.0% VRAM / 0.0% RAM", "vram_delta_mib": { "": 2880.0 }, "memory_used_mib_per_card": { "": 18636.0 }, "memory_used_mib_per_card_before": { "": 15756.0 }, "runner_peak_rss_gib": 0.92, "runner_rss_within_allowance": true, "runner_procs": [ { "pid": 10677, "rss_kb": 964140, "threads": 23 } ], "free_m_at_level": "total used free shared buff/cache available\nMem: 30990 2323 19115 161 10110 28667\nSwap: 8191 0 8191", "kv_footprint_bytes": 11639193600, "kv_footprint_gib": 10.84, "runs": [ { "ttft_ms": 540.6, "first_token_was_thinking": false, "wall_s": 3.4918, "eval_count": 256, "eval_duration_ns": 2945668000, "decode_tok_s": 86.907, "prompt_eval_count": 519, "prompt_eval_duration_ns": 29840000, "prefill_tok_s": 17392.761, "load_duration_ns": 503315016, "total_duration_ns": 3486000557, "streamed_chunks": 256, "response_chars": 1013, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.406, "decode_tok_s_wall_gross": 73.315, "num_ctx_option": 131072, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 119.79, "median": 119.88, "max": 119.94, "mean": 119.87, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 18636.0, "median": 18636.0, "max": 18636.0, "mean": 18636.0, "n": 6 }, "power_w_per_card": { "": 119.87 }, "memory_used_mib_per_card": { "": 18636.0 }, "power_w_all_cards": 119.87, "ups_whole_box": { "load_pct": 20.0, "realpower_w": 204.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 261.21, "max_w": 300.1, "median_w": 298.84, "mean_w_whole_window": 237.65, "max_w_whole_window": 300.1, "limit_w": 300.0, "over_idle_w": 141.34, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.91, "median": 23.35, "max": 24.01, "mean": 23.38, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 27.0, "median": 27.0, "max": 27.0, "mean": 27.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 119.83, "median": 297.62, "max": 300.1, "mean": 237.65, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 18636.0, "median": 18636.0, "max": 18636.0, "mean": 18636.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 79.2, "n": 12 }, "temperature_c": { "min": 60.0, "median": 63.0, "max": 64.0, "mean": 62.7, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 47.0, "median": 47.0, "max": 47.0, "mean": 47.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1380.0, "median": 1462.5, "max": 1725.0, "mean": 1493.8, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 125.4, "median": 298.84, "max": 300.1, "mean": 261.21, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 18636.0, "median": 18636.0, "max": 18636.0, "mean": 18636.0, "n": 10 }, "utilization_pct": { "min": 68.0, "median": 98.0, "max": 98.0, "mean": 95.0, "n": 10 }, "temperature_c": { "min": 61.0, "median": 63.0, "max": 64.0, "mean": 63.2, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 47.0, "median": 47.0, "max": 47.0, "mean": 47.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1380.0, "median": 1455.0, "max": 1485.0, "mean": 1447.5, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 3005.62, "power_mean_w_all_cards": 261.21, "energy_j_per_1k_tokens_all_cards": 3005.62, "watts_per_tok_s": 3.0056, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 61.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 47.0, "fan_max_pct": 47.0, "clock_floor_mhz": 1380.0, "clock_max_mhz": 1485.0, "clock_median_mhz": 1455.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 261.21, "power_pct_of_limit": 87.1, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 87.1% of the cap and the card at 64 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 61.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 47.0, "fan_max_pct": 47.0, "clock_floor_mhz": 1380.0, "clock_max_mhz": 1485.0, "clock_median_mhz": 1455.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 261.21, "power_pct_of_limit": 87.1, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 87.1% of the cap and the card at 64 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 18636.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 20.0, "median": 20.0, "max": 20.0, "mean": 20.0, "n": 2 }, "whole_box_realpower_w": { "min": 204.0, "median": 204.0, "max": 204.0, "mean": 204.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 1 }, { "ttft_ms": 482.35, "first_token_was_thinking": false, "wall_s": 3.4336, "eval_count": 256, "eval_duration_ns": 2946491000, "decode_tok_s": 86.883, "prompt_eval_count": 519, "prompt_eval_duration_ns": 15051000, "prefill_tok_s": 34482.759, "load_duration_ns": 461046025, "total_duration_ns": 3428330103, "streamed_chunks": 256, "response_chars": 1013, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.405, "decode_tok_s_wall_gross": 74.558, "num_ctx_option": 131072, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 119.96, "median": 120.0, "max": 120.02, "mean": 119.99, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 18636.0, "median": 18636.0, "max": 18636.0, "mean": 18636.0, "n": 6 }, "power_w_per_card": { "": 119.99 }, "memory_used_mib_per_card": { "": 18636.0 }, "power_w_all_cards": 119.99, "ups_whole_box": { "load_pct": 21.0, "realpower_w": 216.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 261.54, "max_w": 301.45, "median_w": 298.49, "mean_w_whole_window": 237.93, "max_w_whole_window": 301.45, "limit_w": 300.0, "over_idle_w": 141.55, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 23.09, "median": 23.38, "max": 23.86, "mean": 23.41, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 27.0, "median": 27.0, "max": 27.0, "mean": 27.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 119.92, "median": 297.11, "max": 301.45, "mean": 237.93, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 18636.0, "median": 18636.0, "max": 18636.0, "mean": 18636.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.4, "n": 12 }, "temperature_c": { "min": 60.0, "median": 63.0, "max": 64.0, "mean": 62.8, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 48.0, "median": 48.0, "max": 48.0, "mean": 48.0, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1395.0, "median": 1462.5, "max": 1725.0, "mean": 1497.5, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 135.41, "median": 298.49, "max": 301.45, "mean": 261.54, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 18636.0, "median": 18636.0, "max": 18636.0, "mean": 18636.0, "n": 10 }, "utilization_pct": { "min": 95.0, "median": 98.0, "max": 98.0, "mean": 97.7, "n": 10 }, "temperature_c": { "min": 62.0, "median": 63.0, "max": 64.0, "mean": 63.3, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 48.0, "median": 48.0, "max": 48.0, "mean": 48.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1395.0, "median": 1455.0, "max": 1470.0, "mean": 1452.0, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 3010.25, "power_mean_w_all_cards": 261.54, "energy_j_per_1k_tokens_all_cards": 3010.25, "watts_per_tok_s": 3.0103, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 62.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 48.0, "fan_max_pct": 48.0, "clock_floor_mhz": 1395.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1455.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 261.54, "power_pct_of_limit": 87.2, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "NEITHER cap: the SM clock varied with draw at 87.2% of the cap and the card at 64 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 62.0, "temp_rise_c": 2.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 48.0, "fan_max_pct": 48.0, "clock_floor_mhz": 1395.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1455.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 261.54, "power_pct_of_limit": 87.2, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": false, "verdict": "NEITHER cap: the SM clock varied with draw at 87.2% of the cap and the card at 64 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 18636.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 21.0, "median": 21.0, "max": 21.0, "mean": 21.0, "n": 2 }, "whole_box_realpower_w": { "min": 216.0, "median": 216.0, "max": 216.0, "mean": 216.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 2 }, { "ttft_ms": 493.09, "first_token_was_thinking": false, "wall_s": 3.4461, "eval_count": 256, "eval_duration_ns": 2947770000, "decode_tok_s": 86.845, "prompt_eval_count": 519, "prompt_eval_duration_ns": 14458000, "prefill_tok_s": 35897.081, "load_duration_ns": 472752853, "total_duration_ns": 3440462759, "streamed_chunks": 256, "response_chars": 1013, "thinking_chars": 0, "done_reason": "length", "decode_tok_s_wall": 86.351, "decode_tok_s_wall_gross": 74.286, "num_ctx_option": 131072, "power": { "idle_baseline": { "seconds": 3.0, "samples": 6, "power_w": { "min": 119.94, "median": 119.99, "max": 120.0, "mean": 119.98, "n": 6 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 6 }, "memory_used_mib": { "min": 18636.0, "median": 18636.0, "max": 18636.0, "mean": 18636.0, "n": 6 }, "power_w_per_card": { "": 119.98 }, "memory_used_mib_per_card": { "": 18636.0 }, "power_w_all_cards": 119.98, "ups_whole_box": { "load_pct": 41.0, "realpower_w": 413.0, "realpower_nominal_w": 1000.0, "status": "OL", "note": "the UPS's reading of EVERYTHING it feeds -- the whole box, not the cards. The card figures beside it are board power from nvidia-smi. Neither is the other." } }, "measured_over": "samples where the GPU was busy", "busy_samples": 10, "mean_w": 253.52, "max_w": 299.98, "median_w": 297.31, "mean_w_whole_window": 231.24, "max_w_whole_window": 299.98, "limit_w": 300.0, "over_idle_w": 133.54, "samples": 12, "per_gpu": { "": { "index": "0", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 22.93, "median": 23.32, "max": 23.84, "mean": 23.32, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "temperature_c": { "min": 27.0, "median": 27.0, "max": 27.0, "mean": 27.0, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 0.0, "median": 0.0, "max": 0.0, "mean": 0.0, "n": 12 }, "pcie_width_current": { "min": 4.0, "median": 4.0, "max": 4.0, "mean": 4.0, "n": 12 }, "clocks_sm_mhz": { "min": 210.0, "median": 210.0, "max": 210.0, "mean": 210.0, "n": 12 }, "clocks_mem_mhz": { "min": 405.0, "median": 405.0, "max": 405.0, "mean": 405.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": null, "busy_samples": 0 }, "": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 119.84, "median": 265.25, "max": 299.98, "mean": 231.24, "n": 12 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 18636.0, "median": 18636.0, "max": 18636.0, "mean": 18636.0, "n": 12 }, "utilization_pct": { "min": 0.0, "median": 98.0, "max": 98.0, "mean": 81.1, "n": 12 }, "temperature_c": { "min": 59.0, "median": 63.0, "max": 64.0, "mean": 62.2, "n": 12 }, "memory_temperature_c": null, "fan_pct": { "min": 48.0, "median": 48.0, "max": 49.0, "mean": 48.1, "n": 12 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 }, "clocks_sm_mhz": { "min": 1395.0, "median": 1470.0, "max": 1725.0, "mean": 1502.5, "n": 12 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 12 }, "throttle_sw_power_cap_fraction": 0.8333, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 12, "busy": { "index": "1", "name": "NVIDIA GeForce RTX 3090", "power_w": { "min": 131.3, "median": 297.31, "max": 299.98, "mean": 253.52, "n": 10 }, "power_limit_w": 300.0, "memory_used_mib": { "min": 18636.0, "median": 18636.0, "max": 18636.0, "mean": 18636.0, "n": 10 }, "utilization_pct": { "min": 92.0, "median": 98.0, "max": 98.0, "mean": 97.3, "n": 10 }, "temperature_c": { "min": 61.0, "median": 63.0, "max": 64.0, "mean": 62.9, "n": 10 }, "memory_temperature_c": null, "fan_pct": { "min": 48.0, "median": 48.0, "max": 48.0, "mean": 48.0, "n": 10 }, "pcie_width_current": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 10 }, "clocks_sm_mhz": { "min": 1395.0, "median": 1470.0, "max": 1470.0, "mean": 1458.0, "n": 10 }, "clocks_mem_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 10 }, "throttle_sw_power_cap_fraction": 1.0, "throttle_hw_slowdown_fraction": 0.0, "throttle_sw_thermal_fraction": 0.0, "throttle_hw_thermal_fraction": 0.0, "n": 10 }, "busy_samples": 10 } } }, "energy_j_per_1k_tokens": 2919.21, "power_mean_w_all_cards": 253.52, "energy_j_per_1k_tokens_all_cards": 2919.21, "watts_per_tok_s": 2.9192, "thermal": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 61.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 48.0, "fan_max_pct": 48.0, "clock_floor_mhz": 1395.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1470.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 253.52, "power_pct_of_limit": 84.5, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 84.5% of the cap and the card at 64 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" }, "thermal_per_card": { "": { "rule_version": 2, "measured_over": "samples where the GPU was busy (utilisation > 0)", "busy_samples": 10, "temp_max_c": 64.0, "temp_min_c": 61.0, "temp_rise_c": 3.0, "memory_temp_max_c": null, "memory_temp_median_c": null, "memory_temp_readable": false, "pcie_width_min": 16.0, "pcie_width_max": 16.0, "pcie_width_median": 16.0, "fan_mean_pct": 48.0, "fan_max_pct": 48.0, "clock_floor_mhz": 1395.0, "clock_max_mhz": 1470.0, "clock_median_mhz": 1470.0, "mem_clock_floor_mhz": 9501.0, "mem_clock_median_mhz": 9501.0, "mem_clock_max_mhz": 9501.0, "mem_clock_held": true, "memory_bus_width_bits": 384, "peak_mem_bandwidth_gbs_at_median_clock": 912.1, "peak_mem_bandwidth_gbs_at_floor_clock": 912.1, "peak_mem_bandwidth_gbs_at_max_clock": 912.1, "peak_mem_bandwidth_formula": "GB/s = clocks.mem MHz x 1e6 x 2 bits-per-clock x (bus bits / 8) / 1e9; a CEILING at the clock that was observed, never an achieved rate", "clock_dropped": true, "power_limit_w": 300.0, "power_mean_w": 253.52, "power_pct_of_limit": 84.5, "power_pinned_at_limit": false, "sw_power_cap_active_fraction": 1.0, "hw_slowdown_active_fraction": 0.0, "sw_thermal_active_fraction": 0.0, "hw_thermal_active_fraction": 0.0, "at_throttling_temperature": false, "stop_core_temp_c": 83.0, "stop_memory_temp_c": 100.0, "hit_core_temp_stop": false, "hit_memory_temp_stop": false, "throttle_temperature_c": 80.0, "temp_rising_within_run": true, "verdict": "NEITHER cap: the SM clock varied with draw at 84.5% of the cap and the card at 64 C. On a memory-bound decode the clock follows the work, and nothing here was limiting it" } }, "decode_bandwidth": { "model_tag": "mistral-nemo:12b-instruct-2407-q4_K_M", "model_density": "unknown", "weight_bytes": 7477217229, "peak_bandwidth_gbs_at_observed_clock": 912.1, "decode_gbs": null, "fraction_of_peak_pct": null, "basis": null, "refused_reason": "mistral-nemo:12b-instruct-2407-q4_K_M is not on the pre-registered dense/MoE list, so no bytes-per-second figure is computed for it." }, "memory_used_mib_per_card": { "": 18636.0 }, "pcie_width_under_load": { "": { "min": 16.0, "median": 16.0, "max": 16.0, "mean": 16.0, "n": 12 } }, "ups_during_run": { "ups": "pr1500@localhost", "samples": 2, "load_pct": { "min": 33.0, "median": 37.0, "max": 41.0, "mean": 37.0, "n": 2 }, "whole_box_realpower_w": { "min": 337.0, "median": 375.0, "max": 413.0, "mean": 375.0, "n": 2 }, "whole_box_note": "the UPS's reading of everything it feeds, not the cards; nvidia-smi's per-card watts are a different measurement of a smaller thing", "stop_threshold_pct": 80.0, "breached": false }, "stop_conditions_hit": [], "run": 3 } ], "derived": { "decode_tok_s": { "min": 86.845, "median": 86.883, "max": 86.907, "mean": 86.878, "n": 3 }, "ttft_ms": { "min": 482.35, "median": 493.09, "max": 540.6, "mean": 505.35, "n": 3 }, "power_mean_w": { "min": 253.52, "median": 261.21, "max": 261.54, "mean": 258.76, "n": 3 }, "power_mean_w_all_cards": { "min": 253.52, "median": 261.21, "max": 261.54, "mean": 258.76, "n": 3 }, "energy_j_per_1k_tokens_all_cards": { "min": 2919.21, "median": 3005.62, "max": 3010.25, "mean": 2978.36, "n": 3 }, "energy_j_per_1k_tokens": { "min": 2919.21, "median": 3005.62, "max": 3010.25, "mean": 2978.36, "n": 3 }, "mem_clock_median_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "mem_clock_floor_mhz": { "min": 9501.0, "median": 9501.0, "max": 9501.0, "mean": 9501.0, "n": 3 }, "peak_mem_bandwidth_gbs_at_median_clock": { "min": 912.1, "median": 912.1, "max": 912.1, "mean": 912.1, "n": 3 }, "decode_gbs": null, "decode_bandwidth_fraction_of_peak_pct": null, "sw_power_cap_active_fraction": { "min": 1.0, "median": 1.0, "max": 1.0, "mean": 1.0, "n": 3 }, "temp_max_c": { "min": 64.0, "median": 64.0, "max": 64.0, "mean": 64.0, "n": 3 }, "fan_max_pct": { "min": 47.0, "median": 48.0, "max": 48.0, "mean": 47.7, "n": 3 } }, "finished_utc": "2026-10-15T05:41:38Z" } ], "errors": [], "context_length_max": 1024000, "quantization": "Q4_K_M", "model_store_size_bytes": 7477217229, "model_store_size_gib": 6.96, "model_density": "unknown", "ups_after_arm": { "ups": "pr1500@localhost", "read_utc": "2026-10-15T05:41:38Z", "available": true, "raw": "battery.charge: 100\nbattery.charge.low: 10\nbattery.charge.warning: 20\nbattery.mfr.date: CPS\nbattery.runtime: 1325\nbattery.runtime.low: 300\nbattery.status: 100%\nbattery.type: PbAcid\nbattery.voltage: 27.2\nbattery.voltage.nominal: 24\ndevice.mfr: CPS\ndevice.model: CP1500PFCLCDa\ndevice.serial: \ndevice.type: ups\ndriver.debug: 0\ndriver.flag.allow_killpower: 0\ndriver.name: usbhid-ups\ndriver.parameter.interrupt_pipe_no_events_tolerance: -1\ndriver.parameter.pollfreq: 12\ndriver.parameter.pollinterval: 2\ndriver.parameter.port: auto\ndriver.parameter.synchronous: auto\ndriver.state: quiet\ndriver.version: 2.8.4\ndriver.version.data: CyberPower HID 0.84\ndriver.version.internal: 0.67\ndriver.version.usb: libusb-1.0.29 (API: 0x0100010B)\ninput.sensitivity: normal\ninput.transfer.high: 139\ninput.transfer.low: 100\ninput.voltage: 121.0\ninput.voltage.nominal: 120\noutput.voltage: 121.0\nups.beeper.status: enabled\nups.delay.shutdown: 60\nups.delay.start: 120\nups.firmware: CR01802H1\nups.load: 33\nups.mfr: CPS\nups.model: CP1500PFCLCDa\nups.power: 418\nups.power.nominal: 1500\nups.productid: 0601\nups.realpower: 337\nups.realpower.nominal: 1000\nups.serial: \nups.status: OL\nups.test.result: No test initiated\nups.timer.shutdown: -60\nups.timer.start: -60\nups.vendorid: 0764", "battery.charge": 100.0, "input.voltage": 121.0, "output.voltage": 121.0, "ups.load": 33.0, "ups.realpower": 337.0, "ups.realpower.nominal": 1000.0, "ups.status": "OL", "load_pct": 33.0, "stop_threshold_pct": 80.0, "over_stop_threshold": false }, "largest_context_that_fits": 131072, "levels_requested": [ 4096, 8192, 16384, 32768, 49152, 65536, 98304, 131072 ], "levels_not_reached": [], "ladder_topped_out": "every rung tested fits, up to and including 131,072 -- the ladder ran out of rungs before the card ran out of memory, so the figure reported is the largest window TESTED, not a measured ceiling", "headline": "the biggest context this arm holds for mistral-nemo:12b-instruct-2407-q4_K_M is 131,072 tokens", "finished_utc": "2026-10-15T05:41:38Z" }