[2026-09-22T10:29:45Z] KEV CHAIN START GPU-7aa0be10-974f-6430-52fe-09018a0e2e08, 1 MiB, 30.63 W GPU-46890836-d8f7-e868-1977-c1ce28db0d7a, 1 MiB, 24.32 W /dev/mapper/ubuntu--vg-ubuntu--lv 295G 187G 95G 67% / [2026-09-22T10:29:46Z] === kev-9b — serving jaredpalmer/kev-9b, as served [2026-09-22T10:29:56Z] kev-9b server READY after 10s {"models":[{"id":"kev-latest","aliases":["jev-latest"],"run":"jaredpalmer/kev-9b","base":"Qwen/Qwen3.5-9B-Base","lora":16,"device":"cuda","temperature":2.2973967099940698,"prefix_cache":{"size":4,"min_state_tokens":384,"hits":0,"misses":0,"cached_states":0}}]} serving jaredpalmer/kev-9b (/workshop/.cache/huggingface/hub/models--jaredpalmer--kev-9b/snapshots/2629c06a5aeb0feb3b9783bafed17ed8f39ecf5c) on cuda :8009 [2026-09-22T10:29:56Z] warming kev-9b (triton JIT; the first call compiles, the rest do not) warmup took 0.8s, discarded MODELS {"models":[{"id":"kev-latest","aliases":["jev-latest"],"run":"jaredpalmer/kev-9b","base":"Qwen/Qwen3.5-9B-Base","lora":16,"device":"cuda","temperature":2.2973967099940698,"prefix_cache":{"size":4,"min_state_tokens":384,"hits":0,"misses":0,"cached_states":0}}]} ANNOUNCE box benchbox · cards GPU-46890836-d8f · arm kev-9b · task kc · start 2026-09-22T10:29:57Z · expect ~6 min c-darwin-01-dessert-01 darwin -> darwin 0.11s c-darwin-01-dessert-01 darwin -> darwin 0.13s c-darwin-01-dessert-01 darwin -> socrates 0.10s c-darwin-01-dessert-02 darwin -> darwin 0.13s c-darwin-01-dessert-03 darwin -> darwin 0.14s c-darwin-01-dessert-03 darwin -> darwin 0.14s c-darwin-01-main-01 darwin -> darwin 0.10s c-darwin-01-main-02 darwin -> socrates 0.10s c-darwin-01-main-02 darwin -> darwin 0.13s c-darwin-01-main-02 darwin -> darwin 0.14s c-darwin-01-main-02 darwin -> darwin 0.14s c-darwin-01-main-03 darwin -> darwin 0.14s c-darwin-01-starter-01 darwin -> darwin 0.07s c-darwin-01-starter-01 darwin -> darwin 0.08s c-darwin-01-starter-01 darwin -> sagan 0.10s c-darwin-01-starter-02 darwin -> darwin 0.09s c-darwin-01-starter-03 darwin -> darwin 0.14s c-darwin-01-starter-03 darwin -> darwin 0.13s c-einstein-01-dessert-01 einstein -> sagan 0.13s c-einstein-01-dessert-01 einstein -> sagan 0.13s c-einstein-01-dessert-02 einstein -> sagan 0.13s c-einstein-01-dessert-02 einstein -> einstein 0.10s c-einstein-01-dessert-03 einstein -> einstein 0.14s c-einstein-01-main-01 einstein -> einstein 0.10s c-einstein-01-main-01 einstein -> einstein 0.13s c-einstein-01-main-02 einstein -> einstein 0.13s c-einstein-01-main-03 einstein -> sagan 0.11s c-einstein-01-main-03 einstein -> einstein 0.11s c-einstein-01-main-03 einstein -> einstein 0.13s c-einstein-01-main-03 einstein -> einstein 0.14s c-einstein-01-starter-01 einstein -> einstein 0.10s c-einstein-01-starter-02 einstein -> einstein 0.08s c-einstein-01-starter-02 einstein -> sagan 0.10s c-einstein-01-starter-03 einstein -> einstein 0.13s c-einstein-01-starter-03 einstein -> einstein 0.11s c-einstein-01-starter-03 einstein -> einstein 0.09s c-hypatia-01-dessert-01 hypatia -> ibn_sina 0.14s c-hypatia-01-dessert-02 hypatia -> ibn_sina 0.10s c-hypatia-01-dessert-02 hypatia -> darwin 0.09s c-hypatia-01-dessert-03 hypatia -> sagan 0.10s c-hypatia-01-main-01 hypatia -> hypatia 0.13s c-hypatia-01-main-01 hypatia -> socrates 0.10s c-hypatia-01-main-01 hypatia -> ibn_sina 0.13s c-hypatia-01-main-01 hypatia -> sagan 0.11s c-hypatia-01-main-01 hypatia -> einstein 0.09s c-hypatia-01-main-02 hypatia -> ibn_sina 0.13s c-hypatia-01-main-03 hypatia -> einstein 0.13s c-hypatia-01-main-03 hypatia -> ibn_sina 0.10s c-hypatia-01-main-03 hypatia -> hypatia 0.14s c-hypatia-01-starter-01 hypatia -> socrates 0.09s c-hypatia-01-starter-01 hypatia -> ibn_sina 0.10s c-hypatia-01-starter-02 hypatia -> hypatia 0.10s c-hypatia-01-starter-02 hypatia -> hypatia 0.09s c-hypatia-01-starter-02 hypatia -> sagan 0.10s c-ibn_sina-01-dessert-01 ibn_sina -> hypatia 0.10s c-ibn_sina-01-dessert-02 ibn_sina -> ibn_sina 0.10s c-ibn_sina-01-dessert-02 ibn_sina -> socrates 0.11s c-ibn_sina-01-dessert-02 ibn_sina -> ibn_sina 0.11s c-ibn_sina-01-dessert-03 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-dessert-03 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-main-02 ibn_sina -> sagan 0.10s c-ibn_sina-01-main-02 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-main-02 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-main-03 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-main-03 ibn_sina -> socrates 0.13s c-ibn_sina-01-main-03 ibn_sina -> ibn_sina 0.14s c-ibn_sina-01-starter-01 ibn_sina -> hypatia 0.10s c-ibn_sina-01-starter-02 ibn_sina -> socrates 0.10s c-ibn_sina-01-starter-03 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-starter-03 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-starter-03 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-starter-03 ibn_sina -> hypatia 0.13s c-sagan-01-dessert-01 sagan -> sagan 0.14s c-sagan-01-dessert-01 sagan -> sagan 0.10s c-sagan-01-dessert-01 sagan -> hypatia 0.10s c-sagan-01-dessert-03 sagan -> sagan 0.10s c-sagan-01-dessert-03 sagan -> sagan 0.14s c-sagan-01-dessert-03 sagan -> sagan 0.11s c-sagan-01-dessert-03 sagan -> sagan 0.13s c-sagan-01-dessert-03 sagan -> sagan 0.14s c-sagan-01-main-01 sagan -> sagan 0.11s c-sagan-01-main-02 sagan -> sagan 0.13s c-sagan-01-main-02 sagan -> einstein 0.10s c-sagan-01-main-03 sagan -> sagan 0.14s c-sagan-01-main-03 sagan -> sagan 0.14s c-sagan-01-starter-01 sagan -> sagan 0.10s c-sagan-01-starter-02 sagan -> sagan 0.14s c-sagan-01-starter-02 sagan -> sagan 0.10s c-sagan-01-starter-03 sagan -> darwin 0.10s c-sagan-01-starter-03 sagan -> einstein 0.13s c-socrates-01-dessert-01 socrates -> socrates 0.14s c-socrates-01-dessert-01 socrates -> socrates 0.10s c-socrates-01-dessert-01 socrates -> ibn_sina 0.13s c-socrates-01-dessert-02 socrates -> socrates 0.11s c-socrates-01-dessert-02 socrates -> socrates 0.13s c-socrates-01-dessert-02 socrates -> ibn_sina 0.10s c-socrates-01-main-01 socrates -> socrates 0.10s c-socrates-01-main-02 socrates -> socrates 0.14s c-socrates-01-main-02 socrates -> socrates 0.10s c-socrates-01-main-02 socrates -> socrates 0.13s c-socrates-01-main-02 socrates -> socrates 0.10s c-socrates-01-main-03 socrates -> sagan 0.11s c-socrates-01-main-03 socrates -> socrates 0.13s c-socrates-01-starter-01 socrates -> socrates 0.08s c-socrates-01-starter-02 socrates -> socrates 0.11s c-socrates-01-starter-03 socrates -> socrates 0.11s c-socrates-01-starter-03 socrates -> socrates 0.13s c-socrates-01-starter-03 socrates -> socrates 0.10s {"n": 108, "n_reached": 108, "n_labelled": 108, "refused": 0, "refused_by_reason": {}, "correct": 72, "errors": 0, "brier": 0.4802564814814815, "reliability": [{"lo": 0.0, "hi": 0.1, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.1, "hi": 0.2, "n": 1, "sum_p": 0.19, "correct": 1, "mean_p": 0.19, "accuracy": 1.0, "gap": -0.81}, {"lo": 0.2, "hi": 0.3, "n": 9, "sum_p": 2.2199999999999998, "correct": 4, "mean_p": 0.24666666666666665, "accuracy": 0.4444444444444444, "gap": -0.19777777777777777}, {"lo": 0.3, "hi": 0.4, "n": 5, "sum_p": 1.7699999999999996 ANNOUNCE box benchbox · cards GPU-46890836-d8f · arm kev-9b · task kd · start 2026-09-22T10:30:20Z · expect ~3 min A1 refuse -> admit 0.17s A2 refuse -> refuse 0.17s A3 refuse -> refuse 0.17s A4 refuse -> refuse 0.17s A5 refuse -> admit 0.17s A6 refuse -> refuse 0.18s B1 refuse -> admit 0.17s B2 refuse -> admit 0.17s B3 refuse -> refuse 0.17s B4 refuse -> refuse 0.17s B5 refuse -> refuse 0.18s B6 refuse -> admit 0.17s C1 refuse -> admit 0.17s C2 refuse -> admit 0.17s C3 refuse -> admit 0.17s C4 refuse -> admit 0.17s C5 refuse -> admit 0.17s C6 refuse -> admit 0.17s D1 refuse -> admit 0.17s D2 refuse -> admit 0.17s D3 refuse -> admit 0.17s D4 refuse -> admit 0.17s D5 refuse -> admit 0.17s D6 refuse -> admit 0.17s E1 refuse -> refuse 0.17s E2 refuse -> admit 0.17s E3 refuse -> refuse 0.17s E4 refuse -> refuse 0.17s E5 refuse -> refuse 0.17s E6 refuse -> admit 0.17s F1 refuse -> refuse 0.17s F2 refuse -> refuse 0.17s F3 refuse -> refuse 0.17s F4 refuse -> admit 0.17s F5 refuse -> admit 0.17s F6 refuse -> refuse 0.17s L1 admit -> admit 0.17s L2 admit -> admit 0.17s L3 admit -> admit 0.17s L4 admit -> admit 0.17s N1 admit -> admit 0.17s N2 admit -> admit 0.17s N3 admit -> admit 0.17s N4 admit -> admit 0.17s T1 admit -> admit 0.17s T2 admit -> admit 0.17s T3 admit -> admit 0.17s T4 admit -> admit 0.17s {"n": 48, "n_reached": 48, "n_labelled": 48, "refused": 0, "refused_by_reason": {}, "correct": 27, "errors": 0, "brier": 0.5679541666666666, "reliability": [{"lo": 0.0, "hi": 0.1, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.1, "hi": 0.2, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.2, "hi": 0.3, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.3, "hi": 0.4, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.4, "hi ANNOUNCE box benchbox · cards GPU-46890836-d8f · arm kev-9b · task kf · start 2026-09-22T10:30:39Z · expect ~3 min t1-01 yes -> yes 0.10s t1-02 yes -> yes 0.09s t1-03 yes -> yes 0.10s t1-04 yes -> yes 0.08s t1-05 yes -> yes 0.09s t1-06 yes -> yes 0.10s t1-07 yes -> yes 0.10s t1-08 yes -> yes 0.10s t1-09 yes -> yes 0.10s t1-10 yes -> yes 0.08s t1-11 yes -> yes 0.09s t1-12 yes -> yes 0.08s t1-13 yes -> yes 0.09s t1-14 yes -> yes 0.10s t1-15 yes -> yes 0.10s t1-16 yes -> yes 0.10s t1-17 yes -> yes 0.08s t1-18 yes -> yes 0.10s t1-19 yes -> yes 0.07s t1-20 yes -> yes 0.10s t2-01 no -> no 0.09s t2-02 no -> no 0.09s t2-03 no -> no 0.10s t2-04 no -> no 0.10s t2-05 no -> no 0.10s t2-06 no -> no 0.09s t2-07 no -> no 0.08s t2-08 no -> no 0.10s t2-09 no -> no 0.09s t2-10 no -> no 0.09s t2-11 no -> no 0.07s t2-12 no -> no 0.10s t2-13 no -> no 0.10s t2-14 no -> no 0.09s t2-15 no -> no 0.10s t2-16 no -> no 0.08s t2-17 no -> no 0.08s t2-18 no -> no 0.09s t2-19 no -> no 0.08s t2-20 no -> no 0.10s {"n": 40, "n_reached": 40, "n_labelled": 40, "refused": 0, "refused_by_reason": {}, "correct": 40, "errors": 0, "brier": 0.017230000000000002, "reliability": [{"lo": 0.0, "hi": 0.1, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.1, "hi": 0.2, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.2, "hi": 0.3, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.3, "hi": 0.4, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.4, " ANNOUNCE box benchbox · cards GPU-46890836-d8f · arm kev-9b · task kj · start 2026-09-22T10:30:53Z · expect ~8 min j-001 grounded -> grounded 6.83s j-002 grounded -> grounded 0.08s j-003 not_grounded -> not_grounded 0.08s j-004 not_grounded -> not_grounded 0.08s j-005 off_page|not_grounded -> off_page 0.10s j-006 grounded -> grounded 0.37s j-007 grounded -> grounded 0.08s j-008 not_grounded -> grounded 0.08s j-009 not_grounded -> not_grounded 0.08s j-010 off_page|not_grounded -> off_page 0.08s j-011 grounded -> grounded 0.27s j-012 grounded -> grounded 0.08s j-013 not_grounded -> not_grounded 0.08s j-014 not_grounded -> not_grounded 0.08s j-015 off_page|not_grounded -> off_page 0.08s j-016 grounded -> grounded 0.28s j-017 grounded -> grounded 0.08s j-018 not_grounded -> not_grounded 0.08s j-019 not_grounded -> not_grounded 0.08s j-020 off_page|not_grounded -> off_page 0.11s j-021 grounded -> grounded 0.41s j-022 grounded -> grounded 0.08s j-023 not_grounded -> not_grounded 0.08s j-024 not_grounded -> not_grounded 0.08s j-025 off_page|not_grounded -> not_grounded 0.08s j-026 grounded -> grounded 0.34s j-027 grounded -> grounded 0.08s j-028 not_grounded -> not_grounded 0.08s j-029 not_grounded -> not_grounded 0.08s j-030 off_page|not_grounded -> off_page 0.11s j-031 grounded -> grounded 0.57s j-032 grounded -> grounded 0.09s j-033 not_grounded -> not_grounded 0.09s j-034 not_grounded -> not_grounded 0.09s j-035 off_page|not_grounded -> off_page 0.08s j-036 grounded -> grounded 0.56s j-037 grounded -> grounded 0.08s j-038 not_grounded -> not_grounded 0.08s j-039 not_grounded -> not_grounded 0.09s j-040 off_page|not_grounded -> not_grounded 0.10s j-041 grounded -> grounded 0.28s j-042 grounded -> grounded 0.11s j-043 not_grounded -> not_grounded 0.11s j-044 not_grounded -> not_grounded 0.08s j-045 off_page|not_grounded -> off_page 0.10s j-046 grounded -> grounded 0.25s j-047 grounded -> grounded 0.08s j-048 not_grounded -> not_grounded 0.08s j-049 not_grounded -> not_grounded 0.08s j-050 off_page|not_grounded -> off_page 0.08s j-051 grounded -> grounded 0.25s j-052 grounded -> grounded 0.08s j-053 not_grounded -> not_grounded 0.08s j-054 not_grounded -> not_grounded 0.08s j-055 off_page|not_grounded -> off_page 0.08s j-056 grounded -> grounded 0.37s j-057 grounded -> grounded 0.08s j-058 not_grounded -> grounded 0.08s j-059 not_grounded -> not_grounded 0.08s j-060 off_page|not_grounded -> off_page 0.08s j-061 grounded -> grounded 0.40s j-062 grounded -> grounded 0.10s j-063 not_grounded -> not_grounded 0.10s j-064 not_grounded -> not_grounded 0.11s j-065 off_page|not_grounded -> off_page 0.08s j-066 grounded -> grounded 0.25s j-067 grounded -> grounded 0.08s j-068 not_grounded -> not_grounded 0.08s j-069 not_grounded -> not_grounded 0.08s j-070 off_page|not_grounded -> off_page 0.08s j-071 grounded -> grounded 0.27s j-072 grounded -> grounded 0.11s j-073 not_grounded -> grounded 0.11s j-074 not_grounded -> not_grounded 0.10s j-075 off_page|not_grounded -> off_page 0.11s j-076 grounded -> grounded 0.28s j-077 grounded -> grounded 0.08s j-078 not_grounded -> not_grounded 0.08s j-079 not_grounded -> not_grounded 0.08s j-080 off_page|not_grounded -> not_grounded 0.08s j-081 grounded -> grounded 0.25s j-082 grounded -> grounded 0.08s j-083 not_grounded -> not_grounded 0.08s j-084 not_grounded -> not_grounded 0.08s j-085 off_page|not_grounded -> off_page 0.08s j-086 grounded -> grounded 0.15s j-087 grounded -> grounded 0.15s j-088 not_grounded -> not_grounded 0.15s j-089 not_grounded -> not_grounded 0.15s j-090 off_page|not_grounded -> off_page 0.17s j-091 grounded -> grounded 0.38s j-092 grounded -> grounded 0.08s j-093 not_grounded -> grounded 0.08s j-094 not_grounded -> not_grounded 0.08s j-095 off_page|not_grounded -> off_page 0.08s j-096 grounded -> grounded 0.17s j-097 grounded -> grounded 0.17s j-098 not_grounded -> not_grounded 0.17s j-099 not_grounded -> not_grounded 0.17s j-100 off_page|not_grounded -> off_page 0.18s j-101 grounded -> grounded 0.57s j-102 grounded -> grounded 0.17s j-103 grounded -> grounded 0.27s j-104 grounded -> grounded 0.17s j-105 grounded -> grounded 0.19s j-106 grounded -> grounded 0.34s j-107 grounded -> grounded 0.40s j-108 grounded -> grounded 0.72s j-109 grounded -> grounded 0.31s j-110 grounded -> grounded 0.25s j-111 grounded -> grounded 0.28s j-112 grounded -> grounded 0.67s j-113 grounded -> grounded 0.29s j-114 grounded -> grounded 0.37s j-115 grounded -> grounded 0.25s j-116 grounded -> grounded 0.25s j-117 grounded -> grounded 0.28s j-118 grounded -> grounded 0.34s j-119 grounded -> grounded 0.44s {"n": 119, "n_reached": 119, "n_labelled": 119, "refused": 0, "refused_by_reason": {}, "correct": 95, "errors": 0, "brier": 0.15655714285714287, "reliability": [{"lo": 0.0, "hi": 0.1, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.1, "hi": 0.2, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.2, "hi": 0.3, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.3, "hi": 0.4, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.4, ANNOUNCE box benchbox · cards GPU-46890836-d8f · arm kev-9b · task ka · start 2026-09-22T10:31:31Z · expect ~20 min on-01 yes -> REFUSED: overlong — (None tokens) on-02 yes -> REFUSED: overlong — (None tokens) on-03 yes -> yes 7.89s on-04 yes -> yes 8.49s Traceback (most recent call last): File "/workshop/bench-jev-2026-09-21/run_kev.py", line 71, in kev_post with urllib.request.urlopen(req, timeout=timeout) as fh: ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/urllib/request.py", line 187, in urlopen return opener.open(url, data, timeout) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/urllib/request.py", line 493, in open response = meth(req, response) File "/usr/lib/python3.14/urllib/request.py", line 602, in http_response response = self.parent.error( 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.14/urllib/request.py", line 531, in error return self._call_chain(*args) ~~~~~~~~~~~~~~~~^^^^^^^ File "/usr/lib/python3.14/urllib/request.py", line 464, in _call_chain result = func(*args) File "/usr/lib/python3.14/urllib/request.py", line 611, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 422: Unprocessable Entity The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/workshop/bench-jev-2026-09-21/run_kev.py", line 340, in raise SystemExit(main()) ~~~~^^ File "/workshop/bench-jev-2026-09-21/run_kev.py", line 335, in main R.run_arm(R.ARMS[name], a.task, a.out) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workshop/bench-jev-2026-09-21/run.py", line 796, in run_arm for row in TASKS[t](arm, articles, overlong): ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workshop/bench-jev-2026-09-21/run_kev.py", line 191, in task_ka row = kev_decide(arm, art["state"], R.INSTR_A % item["question"], R.A_OPTIONS, noul_keys=("yes", "no")) File "/workshop/bench-jev-2026-09-21/run_kev.py", line 114, in kev_decide out = kev_post(arm.host + "/v1/systemone", body) File "/workshop/bench-jev-2026-09-21/run_kev.py", line 79, in kev_post raise R.SeatError("HTTP %s: %s" % (exc.code, detail)) from exc run.SeatError: HTTP 422: {"detail":"branch too long: 56"} [2026-09-22T10:31:59Z] kev-9b RC=1 [2026-09-22T10:31:59Z] stopping kev server pid 15329 GPU-7aa0be10-974f-6430-52fe-09018a0e2e08, 1 MiB, 30.63 W GPU-46890836-d8f7-e868-1977-c1ce28db0d7a, 1 MiB, 24.45 W [2026-09-22T10:32:09Z] === kev-9b-raw — the same checkpoint at KEV_TEMPERATURE=1.0, task (c) only [2026-09-22T10:32:19Z] kev-9b-raw server READY after 10s {"models":[{"id":"kev-latest","aliases":["jev-latest"],"run":"jaredpalmer/kev-9b","base":"Qwen/Qwen3.5-9B-Base","lora":16,"device":"cuda","temperature":1.0,"prefix_cache":{"size":4,"min_state_tokens":384,"hits":0,"misses":0,"cached_states":0}}]} [2026-09-22T10:32:19Z] warming kev-9b-raw (triton JIT; the first call compiles, the rest do not) warmup took 0.8s, discarded MODELS {"models":[{"id":"kev-latest","aliases":["jev-latest"],"run":"jaredpalmer/kev-9b","base":"Qwen/Qwen3.5-9B-Base","lora":16,"device":"cuda","temperature":1.0,"prefix_cache":{"size":4,"min_state_tokens":384,"hits":0,"misses":0,"cached_states":0}}]} ANNOUNCE box benchbox · cards GPU-46890836-d8f · arm kev-9b-raw · task kc · start 2026-09-22T10:32:20Z · expect ~6 min c-darwin-01-dessert-01 darwin -> darwin 0.10s c-darwin-01-dessert-01 darwin -> darwin 0.14s c-darwin-01-dessert-01 darwin -> socrates 0.10s c-darwin-01-dessert-02 darwin -> darwin 0.14s c-darwin-01-dessert-03 darwin -> darwin 0.14s c-darwin-01-dessert-03 darwin -> darwin 0.14s c-darwin-01-main-01 darwin -> darwin 0.10s c-darwin-01-main-02 darwin -> socrates 0.10s c-darwin-01-main-02 darwin -> darwin 0.14s c-darwin-01-main-02 darwin -> darwin 0.14s c-darwin-01-main-02 darwin -> darwin 0.14s c-darwin-01-main-03 darwin -> darwin 0.14s c-darwin-01-starter-01 darwin -> darwin 0.07s c-darwin-01-starter-01 darwin -> darwin 0.08s c-darwin-01-starter-01 darwin -> sagan 0.10s c-darwin-01-starter-02 darwin -> darwin 0.09s c-darwin-01-starter-03 darwin -> darwin 0.14s c-darwin-01-starter-03 darwin -> darwin 0.14s c-einstein-01-dessert-01 einstein -> sagan 0.13s c-einstein-01-dessert-01 einstein -> sagan 0.13s c-einstein-01-dessert-02 einstein -> sagan 0.13s c-einstein-01-dessert-02 einstein -> einstein 0.10s c-einstein-01-dessert-03 einstein -> einstein 0.14s c-einstein-01-main-01 einstein -> einstein 0.10s c-einstein-01-main-01 einstein -> einstein 0.13s c-einstein-01-main-02 einstein -> einstein 0.14s c-einstein-01-main-03 einstein -> sagan 0.11s c-einstein-01-main-03 einstein -> einstein 0.11s c-einstein-01-main-03 einstein -> einstein 0.13s c-einstein-01-main-03 einstein -> einstein 0.14s c-einstein-01-starter-01 einstein -> einstein 0.10s c-einstein-01-starter-02 einstein -> einstein 0.08s c-einstein-01-starter-02 einstein -> sagan 0.10s c-einstein-01-starter-03 einstein -> einstein 0.13s c-einstein-01-starter-03 einstein -> einstein 0.11s c-einstein-01-starter-03 einstein -> einstein 0.09s c-hypatia-01-dessert-01 hypatia -> ibn_sina 0.14s c-hypatia-01-dessert-02 hypatia -> ibn_sina 0.10s c-hypatia-01-dessert-02 hypatia -> darwin 0.09s c-hypatia-01-dessert-03 hypatia -> sagan 0.10s c-hypatia-01-main-01 hypatia -> hypatia 0.13s c-hypatia-01-main-01 hypatia -> socrates 0.10s c-hypatia-01-main-01 hypatia -> ibn_sina 0.14s c-hypatia-01-main-01 hypatia -> sagan 0.11s c-hypatia-01-main-01 hypatia -> einstein 0.09s c-hypatia-01-main-02 hypatia -> ibn_sina 0.13s c-hypatia-01-main-03 hypatia -> einstein 0.13s c-hypatia-01-main-03 hypatia -> ibn_sina 0.11s c-hypatia-01-main-03 hypatia -> hypatia 0.14s c-hypatia-01-starter-01 hypatia -> socrates 0.09s c-hypatia-01-starter-01 hypatia -> ibn_sina 0.10s c-hypatia-01-starter-02 hypatia -> hypatia 0.10s c-hypatia-01-starter-02 hypatia -> hypatia 0.09s c-hypatia-01-starter-02 hypatia -> sagan 0.10s c-ibn_sina-01-dessert-01 ibn_sina -> hypatia 0.10s c-ibn_sina-01-dessert-02 ibn_sina -> ibn_sina 0.11s c-ibn_sina-01-dessert-02 ibn_sina -> socrates 0.11s c-ibn_sina-01-dessert-02 ibn_sina -> ibn_sina 0.11s c-ibn_sina-01-dessert-03 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-dessert-03 ibn_sina -> ibn_sina 0.14s c-ibn_sina-01-main-02 ibn_sina -> sagan 0.10s c-ibn_sina-01-main-02 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-main-02 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-main-03 ibn_sina -> ibn_sina 0.14s c-ibn_sina-01-main-03 ibn_sina -> socrates 0.13s c-ibn_sina-01-main-03 ibn_sina -> ibn_sina 0.14s c-ibn_sina-01-starter-01 ibn_sina -> hypatia 0.10s c-ibn_sina-01-starter-02 ibn_sina -> socrates 0.10s c-ibn_sina-01-starter-03 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-starter-03 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-starter-03 ibn_sina -> ibn_sina 0.13s c-ibn_sina-01-starter-03 ibn_sina -> hypatia 0.13s c-sagan-01-dessert-01 sagan -> sagan 0.14s c-sagan-01-dessert-01 sagan -> sagan 0.10s c-sagan-01-dessert-01 sagan -> hypatia 0.10s c-sagan-01-dessert-03 sagan -> sagan 0.11s c-sagan-01-dessert-03 sagan -> sagan 0.14s c-sagan-01-dessert-03 sagan -> sagan 0.11s c-sagan-01-dessert-03 sagan -> sagan 0.13s c-sagan-01-dessert-03 sagan -> sagan 0.14s c-sagan-01-main-01 sagan -> sagan 0.11s c-sagan-01-main-02 sagan -> sagan 0.13s c-sagan-01-main-02 sagan -> einstein 0.10s c-sagan-01-main-03 sagan -> sagan 0.14s c-sagan-01-main-03 sagan -> sagan 0.14s c-sagan-01-starter-01 sagan -> sagan 0.10s c-sagan-01-starter-02 sagan -> sagan 0.14s c-sagan-01-starter-02 sagan -> sagan 0.10s c-sagan-01-starter-03 sagan -> darwin 0.10s c-sagan-01-starter-03 sagan -> einstein 0.13s c-socrates-01-dessert-01 socrates -> socrates 0.14s c-socrates-01-dessert-01 socrates -> socrates 0.10s c-socrates-01-dessert-01 socrates -> ibn_sina 0.13s c-socrates-01-dessert-02 socrates -> socrates 0.11s c-socrates-01-dessert-02 socrates -> socrates 0.13s c-socrates-01-dessert-02 socrates -> ibn_sina 0.10s c-socrates-01-main-01 socrates -> socrates 0.11s c-socrates-01-main-02 socrates -> socrates 0.14s c-socrates-01-main-02 socrates -> socrates 0.10s c-socrates-01-main-02 socrates -> socrates 0.13s c-socrates-01-main-02 socrates -> socrates 0.10s c-socrates-01-main-03 socrates -> sagan 0.11s c-socrates-01-main-03 socrates -> socrates 0.13s c-socrates-01-starter-01 socrates -> socrates 0.08s c-socrates-01-starter-02 socrates -> socrates 0.11s c-socrates-01-starter-03 socrates -> socrates 0.11s c-socrates-01-starter-03 socrates -> socrates 0.13s c-socrates-01-starter-03 socrates -> socrates 0.11s {"n": 108, "n_reached": 108, "n_labelled": 108, "refused": 0, "refused_by_reason": {}, "correct": 72, "errors": 0, "brier": 0.5395305555555555, "reliability": [{"lo": 0.0, "hi": 0.1, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.1, "hi": 0.2, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.2, "hi": 0.3, "n": 1, "sum_p": 0.22, "correct": 1, "mean_p": 0.22, "accuracy": 1.0, "gap": -0.78}, {"lo": 0.3, "hi": 0.4, "n": 6, "sum_p": 1.9100000000000001, "correct": 3, "mean_p": 0.31833333333333336, "accuracy": 0 [2026-09-22T10:32:43Z] kev-9b-raw RC=0 [2026-09-22T10:32:43Z] stopping kev server pid 16594 GPU-7aa0be10-974f-6430-52fe-09018a0e2e08, 1 MiB, 30.48 W GPU-46890836-d8f7-e868-1977-c1ce28db0d7a, 1 MiB, 24.46 W [2026-09-22T10:32:53Z] === kev-4b — serving jaredpalmer/kev-4b, as served [2026-09-22T10:34:23Z] kev-4b server READY after 90s {"models":[{"id":"kev-latest","aliases":["jev-latest"],"run":"jaredpalmer/kev-4b","base":"Qwen/Qwen3.5-4B-Base","lora":16,"device":"cuda","temperature":2.1435469250725863,"prefix_cache":{"size":4,"min_state_tokens":384,"hits":0,"misses":0,"cached_states":0}}]} serving jaredpalmer/kev-4b (/workshop/.cache/huggingface/hub/models--jaredpalmer--kev-4b/snapshots/485ace8703592fcf405488b262449990824cfed1) on cuda :8009 [2026-09-22T10:34:23Z] warming kev-4b (triton JIT; the first call compiles, the rest do not) warmup took 1.2s, discarded MODELS {"models":[{"id":"kev-latest","aliases":["jev-latest"],"run":"jaredpalmer/kev-4b","base":"Qwen/Qwen3.5-4B-Base","lora":16,"device":"cuda","temperature":2.1435469250725863,"prefix_cache":{"size":4,"min_state_tokens":384,"hits":0,"misses":0,"cached_states":0}}]} ANNOUNCE box benchbox · cards GPU-46890836-d8f · arm kev-4b · task kc · start 2026-09-22T10:34:24Z · expect ~6 min c-darwin-01-dessert-01 darwin -> sagan 0.11s c-darwin-01-dessert-01 darwin -> darwin 0.08s c-darwin-01-dessert-01 darwin -> sagan 0.07s c-darwin-01-dessert-02 darwin -> sagan 0.08s c-darwin-01-dessert-03 darwin -> hypatia 0.08s c-darwin-01-dessert-03 darwin -> darwin 0.08s c-darwin-01-main-01 darwin -> sagan 0.07s c-darwin-01-main-02 darwin -> socrates 0.07s c-darwin-01-main-02 darwin -> darwin 0.08s c-darwin-01-main-02 darwin -> darwin 0.08s c-darwin-01-main-02 darwin -> darwin 0.08s c-darwin-01-main-03 darwin -> darwin 0.08s c-darwin-01-starter-01 darwin -> sagan 0.07s c-darwin-01-starter-01 darwin -> darwin 0.07s c-darwin-01-starter-01 darwin -> sagan 0.07s c-darwin-01-starter-02 darwin -> darwin 0.07s c-darwin-01-starter-03 darwin -> sagan 0.08s c-darwin-01-starter-03 darwin -> darwin 0.08s c-einstein-01-dessert-01 einstein -> sagan 0.08s c-einstein-01-dessert-01 einstein -> sagan 0.08s c-einstein-01-dessert-02 einstein -> sagan 0.08s c-einstein-01-dessert-02 einstein -> ibn_sina 0.07s c-einstein-01-dessert-03 einstein -> sagan 0.08s c-einstein-01-main-01 einstein -> sagan 0.07s c-einstein-01-main-01 einstein -> einstein 0.08s c-einstein-01-main-02 einstein -> sagan 0.08s c-einstein-01-main-03 einstein -> sagan 0.07s c-einstein-01-main-03 einstein -> einstein 0.07s c-einstein-01-main-03 einstein -> einstein 0.08s c-einstein-01-main-03 einstein -> einstein 0.08s c-einstein-01-starter-01 einstein -> einstein 0.07s c-einstein-01-starter-02 einstein -> einstein 0.07s c-einstein-01-starter-02 einstein -> sagan 0.07s c-einstein-01-starter-03 einstein -> sagan 0.08s c-einstein-01-starter-03 einstein -> einstein 0.07s c-einstein-01-starter-03 einstein -> einstein 0.07s c-hypatia-01-dessert-01 hypatia -> socrates 0.08s c-hypatia-01-dessert-02 hypatia -> sagan 0.07s c-hypatia-01-dessert-02 hypatia -> sagan 0.07s c-hypatia-01-dessert-03 hypatia -> einstein 0.07s c-hypatia-01-main-01 hypatia -> hypatia 0.08s c-hypatia-01-main-01 hypatia -> ibn_sina 0.07s c-hypatia-01-main-01 hypatia -> sagan 0.08s c-hypatia-01-main-01 hypatia -> socrates 0.07s c-hypatia-01-main-01 hypatia -> einstein 0.07s c-hypatia-01-main-02 hypatia -> sagan 0.08s c-hypatia-01-main-03 hypatia -> hypatia 0.08s c-hypatia-01-main-03 hypatia -> ibn_sina 0.07s c-hypatia-01-main-03 hypatia -> hypatia 0.08s c-hypatia-01-starter-01 hypatia -> hypatia 0.07s c-hypatia-01-starter-01 hypatia -> ibn_sina 0.07s c-hypatia-01-starter-02 hypatia -> hypatia 0.07s c-hypatia-01-starter-02 hypatia -> hypatia 0.07s c-hypatia-01-starter-02 hypatia -> sagan 0.07s c-ibn_sina-01-dessert-01 ibn_sina -> sagan 0.07s c-ibn_sina-01-dessert-02 ibn_sina -> ibn_sina 0.07s c-ibn_sina-01-dessert-02 ibn_sina -> sagan 0.07s c-ibn_sina-01-dessert-02 ibn_sina -> sagan 0.07s c-ibn_sina-01-dessert-03 ibn_sina -> sagan 0.08s c-ibn_sina-01-dessert-03 ibn_sina -> ibn_sina 0.08s c-ibn_sina-01-main-02 ibn_sina -> sagan 0.07s c-ibn_sina-01-main-02 ibn_sina -> ibn_sina 0.08s c-ibn_sina-01-main-02 ibn_sina -> sagan 0.08s c-ibn_sina-01-main-03 ibn_sina -> ibn_sina 0.08s c-ibn_sina-01-main-03 ibn_sina -> socrates 0.08s c-ibn_sina-01-main-03 ibn_sina -> ibn_sina 0.08s c-ibn_sina-01-starter-01 ibn_sina -> ibn_sina 0.07s c-ibn_sina-01-starter-02 ibn_sina -> socrates 0.07s c-ibn_sina-01-starter-03 ibn_sina -> ibn_sina 0.08s c-ibn_sina-01-starter-03 ibn_sina -> ibn_sina 0.08s c-ibn_sina-01-starter-03 ibn_sina -> sagan 0.08s c-ibn_sina-01-starter-03 ibn_sina -> sagan 0.08s c-sagan-01-dessert-01 sagan -> sagan 0.08s c-sagan-01-dessert-01 sagan -> sagan 0.07s c-sagan-01-dessert-01 sagan -> sagan 0.07s c-sagan-01-dessert-03 sagan -> sagan 0.07s c-sagan-01-dessert-03 sagan -> sagan 0.08s c-sagan-01-dessert-03 sagan -> sagan 0.07s c-sagan-01-dessert-03 sagan -> sagan 0.08s c-sagan-01-dessert-03 sagan -> sagan 0.08s c-sagan-01-main-01 sagan -> einstein 0.07s c-sagan-01-main-02 sagan -> sagan 0.08s c-sagan-01-main-02 sagan -> hypatia 0.07s c-sagan-01-main-03 sagan -> sagan 0.08s c-sagan-01-main-03 sagan -> sagan 0.08s c-sagan-01-starter-01 sagan -> sagan 0.07s c-sagan-01-starter-02 sagan -> sagan 0.08s c-sagan-01-starter-02 sagan -> sagan 0.07s c-sagan-01-starter-03 sagan -> sagan 0.07s c-sagan-01-starter-03 sagan -> einstein 0.08s c-socrates-01-dessert-01 socrates -> socrates 0.08s c-socrates-01-dessert-01 socrates -> socrates 0.07s c-socrates-01-dessert-01 socrates -> sagan 0.08s c-socrates-01-dessert-02 socrates -> sagan 0.07s c-socrates-01-dessert-02 socrates -> socrates 0.08s c-socrates-01-dessert-02 socrates -> socrates 0.07s c-socrates-01-main-01 socrates -> socrates 0.07s c-socrates-01-main-02 socrates -> socrates 0.08s c-socrates-01-main-02 socrates -> socrates 0.07s c-socrates-01-main-02 socrates -> socrates 0.08s c-socrates-01-main-02 socrates -> socrates 0.07s c-socrates-01-main-03 socrates -> sagan 0.07s c-socrates-01-main-03 socrates -> ibn_sina 0.08s c-socrates-01-starter-01 socrates -> socrates 0.07s c-socrates-01-starter-02 socrates -> socrates 0.07s c-socrates-01-starter-03 socrates -> socrates 0.07s c-socrates-01-starter-03 socrates -> socrates 0.08s c-socrates-01-starter-03 socrates -> socrates 0.07s {"n": 108, "n_reached": 108, "n_labelled": 108, "refused": 0, "refused_by_reason": {}, "correct": 60, "errors": 0, "brier": 0.6182249999999999, "reliability": [{"lo": 0.0, "hi": 0.1, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.1, "hi": 0.2, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.2, "hi": 0.3, "n": 6, "sum_p": 1.6, "correct": 1, "mean_p": 0.26666666666666666, "accuracy": 0.16666666666666666, "gap": 0.1}, {"lo": 0.3, "hi": 0.4, "n": 11, "sum_p": 3.9500000000000006, "correct": 4, "mean_p": 0.359 ANNOUNCE box benchbox · cards GPU-46890836-d8f · arm kev-4b · task kd · start 2026-09-22T10:34:44Z · expect ~3 min A1 refuse -> admit 0.11s A2 refuse -> admit 0.10s A3 refuse -> admit 0.10s A4 refuse -> admit 0.10s A5 refuse -> admit 0.10s A6 refuse -> admit 0.10s B1 refuse -> admit 0.10s B2 refuse -> admit 0.10s B3 refuse -> admit 0.10s B4 refuse -> admit 0.10s B5 refuse -> admit 0.10s B6 refuse -> admit 0.10s C1 refuse -> admit 0.10s C2 refuse -> admit 0.10s C3 refuse -> admit 0.10s C4 refuse -> admit 0.10s C5 refuse -> admit 0.10s C6 refuse -> admit 0.10s D1 refuse -> admit 0.10s D2 refuse -> admit 0.10s D3 refuse -> admit 0.10s D4 refuse -> admit 0.10s D5 refuse -> admit 0.10s D6 refuse -> admit 0.10s E1 refuse -> refuse 0.10s E2 refuse -> admit 0.10s E3 refuse -> admit 0.10s E4 refuse -> refuse 0.11s E5 refuse -> refuse 0.11s E6 refuse -> admit 0.10s F1 refuse -> admit 0.10s F2 refuse -> admit 0.10s F3 refuse -> refuse 0.10s F4 refuse -> admit 0.10s F5 refuse -> admit 0.10s F6 refuse -> admit 0.10s L1 admit -> admit 0.10s L2 admit -> admit 0.10s L3 admit -> admit 0.10s L4 admit -> admit 0.10s N1 admit -> admit 0.10s N2 admit -> admit 0.10s N3 admit -> admit 0.10s N4 admit -> admit 0.10s T1 admit -> admit 0.10s T2 admit -> admit 0.10s T3 admit -> admit 0.10s T4 admit -> admit 0.10s {"n": 48, "n_reached": 48, "n_labelled": 48, "refused": 0, "refused_by_reason": {}, "correct": 16, "errors": 0, "brier": 1.0120166666666666, "reliability": [{"lo": 0.0, "hi": 0.1, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.1, "hi": 0.2, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.2, "hi": 0.3, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.3, "hi": 0.4, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.4, "hi ANNOUNCE box benchbox · cards GPU-46890836-d8f · arm kev-4b · task kf · start 2026-09-22T10:35:00Z · expect ~3 min t1-01 yes -> yes 0.07s t1-02 yes -> yes 0.07s t1-03 yes -> yes 0.07s t1-04 yes -> yes 0.07s t1-05 yes -> yes 0.07s t1-06 yes -> yes 0.07s t1-07 yes -> yes 0.07s t1-08 yes -> yes 0.07s t1-09 yes -> yes 0.07s t1-10 yes -> yes 0.07s t1-11 yes -> yes 0.07s t1-12 yes -> yes 0.07s t1-13 yes -> yes 0.07s t1-14 yes -> yes 0.07s t1-15 yes -> yes 0.07s t1-16 yes -> yes 0.07s t1-17 yes -> yes 0.07s t1-18 yes -> yes 0.07s t1-19 yes -> yes 0.07s t1-20 yes -> yes 0.07s t2-01 no -> no 0.07s t2-02 no -> no 0.07s t2-03 no -> no 0.07s t2-04 no -> no 0.07s t2-05 no -> no 0.07s t2-06 no -> no 0.07s t2-07 no -> no 0.07s t2-08 no -> no 0.07s t2-09 no -> no 0.07s t2-10 no -> no 0.07s t2-11 no -> no 0.07s t2-12 no -> no 0.07s t2-13 no -> no 0.07s t2-14 no -> no 0.07s t2-15 no -> no 0.07s t2-16 no -> no 0.07s t2-17 no -> no 0.07s t2-18 no -> no 0.07s t2-19 no -> no 0.07s t2-20 no -> no 0.07s {"n": 40, "n_reached": 40, "n_labelled": 40, "refused": 0, "refused_by_reason": {}, "correct": 40, "errors": 0, "brier": 0.008130000000000002, "reliability": [{"lo": 0.0, "hi": 0.1, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.1, "hi": 0.2, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.2, "hi": 0.3, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.3, "hi": 0.4, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.4, " ANNOUNCE box benchbox · cards GPU-46890836-d8f · arm kev-4b · task kj · start 2026-09-22T10:35:13Z · expect ~8 min j-001 grounded -> grounded 0.29s j-002 grounded -> grounded 0.08s j-003 not_grounded -> not_grounded 0.07s j-004 not_grounded -> not_grounded 0.07s j-005 off_page|not_grounded -> off_page 0.07s j-006 grounded -> grounded 0.24s j-007 grounded -> grounded 0.07s j-008 not_grounded -> grounded 0.07s j-009 not_grounded -> not_grounded 0.07s j-010 off_page|not_grounded -> off_page 0.08s j-011 grounded -> grounded 0.20s j-012 grounded -> grounded 0.08s j-013 not_grounded -> not_grounded 0.08s j-014 not_grounded -> not_grounded 0.07s j-015 off_page|not_grounded -> off_page 0.07s j-016 grounded -> grounded 0.19s j-017 grounded -> grounded 0.07s j-018 not_grounded -> grounded 0.07s j-019 not_grounded -> not_grounded 0.07s j-020 off_page|not_grounded -> not_grounded 0.08s j-021 grounded -> grounded 0.27s j-022 grounded -> grounded 0.08s j-023 not_grounded -> not_grounded 0.07s j-024 not_grounded -> not_grounded 0.07s j-025 off_page|not_grounded -> off_page 0.08s j-026 grounded -> grounded 0.22s j-027 grounded -> grounded 0.08s j-028 not_grounded -> not_grounded 0.08s j-029 not_grounded -> not_grounded 0.08s j-030 off_page|not_grounded -> not_grounded 0.08s j-031 grounded -> grounded 0.37s j-032 grounded -> grounded 0.08s j-033 not_grounded -> not_grounded 0.08s j-034 not_grounded -> not_grounded 0.07s j-035 off_page|not_grounded -> off_page 0.08s j-036 grounded -> grounded 0.37s j-037 grounded -> grounded 0.08s j-038 not_grounded -> not_grounded 0.08s j-039 not_grounded -> not_grounded 0.08s j-040 off_page|not_grounded -> off_page 0.07s j-041 grounded -> grounded 0.21s j-042 grounded -> grounded 0.07s j-043 not_grounded -> not_grounded 0.07s j-044 not_grounded -> not_grounded 0.08s j-045 off_page|not_grounded -> off_page 0.08s j-046 grounded -> grounded 0.17s j-047 grounded -> grounded 0.08s j-048 not_grounded -> not_grounded 0.08s j-049 not_grounded -> not_grounded 0.07s j-050 off_page|not_grounded -> not_grounded 0.08s j-051 grounded -> grounded 0.17s j-052 grounded -> grounded 0.07s j-053 not_grounded -> not_grounded 0.07s j-054 not_grounded -> not_grounded 0.07s j-055 off_page|not_grounded -> off_page 0.08s j-056 grounded -> grounded 0.24s j-057 grounded -> grounded 0.08s j-058 not_grounded -> grounded 0.07s j-059 not_grounded -> not_grounded 0.08s j-060 off_page|not_grounded -> off_page 0.07s j-061 grounded -> grounded 0.24s j-062 grounded -> grounded 0.08s j-063 not_grounded -> not_grounded 0.08s j-064 not_grounded -> not_grounded 0.08s j-065 off_page|not_grounded -> off_page 0.07s j-066 grounded -> grounded 0.17s j-067 grounded -> grounded 0.08s j-068 not_grounded -> not_grounded 0.08s j-069 not_grounded -> not_grounded 0.08s j-070 off_page|not_grounded -> off_page 0.07s j-071 grounded -> grounded 0.17s j-072 grounded -> grounded 0.08s j-073 not_grounded -> grounded 0.08s j-074 not_grounded -> not_grounded 0.08s j-075 off_page|not_grounded -> off_page 0.08s j-076 grounded -> grounded 0.20s j-077 grounded -> grounded 0.07s j-078 not_grounded -> grounded 0.07s j-079 not_grounded -> not_grounded 0.08s j-080 off_page|not_grounded -> off_page 0.08s j-081 grounded -> grounded 0.17s j-082 grounded -> grounded 0.07s j-083 not_grounded -> grounded 0.08s j-084 not_grounded -> not_grounded 0.08s j-085 off_page|not_grounded -> off_page 0.07s j-086 grounded -> grounded 0.09s j-087 grounded -> grounded 0.09s j-088 not_grounded -> not_grounded 0.09s j-089 not_grounded -> not_grounded 0.09s j-090 off_page|not_grounded -> off_page 0.10s j-091 grounded -> grounded 0.25s j-092 grounded -> grounded 0.07s j-093 not_grounded -> grounded 0.08s j-094 not_grounded -> not_grounded 0.08s j-095 off_page|not_grounded -> off_page 0.08s j-096 grounded -> grounded 0.10s j-097 grounded -> grounded 0.10s j-098 not_grounded -> not_grounded 0.10s j-099 not_grounded -> not_grounded 0.10s j-100 off_page|not_grounded -> off_page 0.10s j-101 grounded -> grounded 0.38s j-102 grounded -> grounded 0.10s j-103 grounded -> grounded 0.20s j-104 grounded -> grounded 0.10s j-105 grounded -> grounded 0.13s j-106 grounded -> grounded 0.22s j-107 grounded -> grounded 0.27s j-108 grounded -> grounded 0.45s j-109 grounded -> grounded 0.20s j-110 grounded -> grounded 0.17s j-111 grounded -> grounded 0.17s j-112 grounded -> grounded 0.43s j-113 grounded -> grounded 0.20s j-114 grounded -> grounded 0.25s j-115 grounded -> grounded 0.17s j-116 grounded -> grounded 0.16s j-117 grounded -> grounded 0.17s j-118 grounded -> grounded 0.22s j-119 grounded -> grounded 0.30s {"n": 119, "n_reached": 119, "n_labelled": 119, "refused": 0, "refused_by_reason": {}, "correct": 92, "errors": 0, "brier": 0.19419915966386556, "reliability": [{"lo": 0.0, "hi": 0.1, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.1, "hi": 0.2, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.2, "hi": 0.3, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.3, "hi": 0.4, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.4, ANNOUNCE box benchbox · cards GPU-46890836-d8f · arm kev-4b · task ka · start 2026-09-22T10:35:38Z · expect ~20 min on-01 yes -> REFUSED: overlong — (None tokens) on-02 yes -> REFUSED: overlong — (None tokens) on-03 yes -> yes 0.59s on-04 yes -> yes 0.92s Traceback (most recent call last): File "/workshop/bench-jev-2026-09-21/run_kev.py", line 71, in kev_post with urllib.request.urlopen(req, timeout=timeout) as fh: ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/urllib/request.py", line 187, in urlopen return opener.open(url, data, timeout) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/urllib/request.py", line 493, in open response = meth(req, response) File "/usr/lib/python3.14/urllib/request.py", line 602, in http_response response = self.parent.error( 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.14/urllib/request.py", line 531, in error return self._call_chain(*args) ~~~~~~~~~~~~~~~~^^^^^^^ File "/usr/lib/python3.14/urllib/request.py", line 464, in _call_chain result = func(*args) File "/usr/lib/python3.14/urllib/request.py", line 611, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 422: Unprocessable Entity The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/workshop/bench-jev-2026-09-21/run_kev.py", line 340, in raise SystemExit(main()) ~~~~^^ File "/workshop/bench-jev-2026-09-21/run_kev.py", line 335, in main R.run_arm(R.ARMS[name], a.task, a.out) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workshop/bench-jev-2026-09-21/run.py", line 796, in run_arm for row in TASKS[t](arm, articles, overlong): ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workshop/bench-jev-2026-09-21/run_kev.py", line 191, in task_ka row = kev_decide(arm, art["state"], R.INSTR_A % item["question"], R.A_OPTIONS, noul_keys=("yes", "no")) File "/workshop/bench-jev-2026-09-21/run_kev.py", line 114, in kev_decide out = kev_post(arm.host + "/v1/systemone", body) File "/workshop/bench-jev-2026-09-21/run_kev.py", line 79, in kev_post raise R.SeatError("HTTP %s: %s" % (exc.code, detail)) from exc run.SeatError: HTTP 422: {"detail":"branch too long: 56"} [2026-09-22T10:35:50Z] kev-4b RC=1 [2026-09-22T10:35:50Z] stopping kev server pid 16892 GPU-7aa0be10-974f-6430-52fe-09018a0e2e08, 1 MiB, 30.58 W GPU-46890836-d8f7-e868-1977-c1ce28db0d7a, 1 MiB, 24.69 W [2026-09-22T10:36:00Z] === kev-4b-raw — the same checkpoint at KEV_TEMPERATURE=1.0, task (c) only [2026-09-22T10:36:11Z] kev-4b-raw server READY after 10s {"models":[{"id":"kev-latest","aliases":["jev-latest"],"run":"jaredpalmer/kev-4b","base":"Qwen/Qwen3.5-4B-Base","lora":16,"device":"cuda","temperature":1.0,"prefix_cache":{"size":4,"min_state_tokens":384,"hits":0,"misses":0,"cached_states":0}}]} [2026-09-22T10:36:11Z] warming kev-4b-raw (triton JIT; the first call compiles, the rest do not) warmup took 0.8s, discarded MODELS {"models":[{"id":"kev-latest","aliases":["jev-latest"],"run":"jaredpalmer/kev-4b","base":"Qwen/Qwen3.5-4B-Base","lora":16,"device":"cuda","temperature":1.0,"prefix_cache":{"size":4,"min_state_tokens":384,"hits":0,"misses":0,"cached_states":0}}]} ANNOUNCE box benchbox · cards GPU-46890836-d8f · arm kev-4b-raw · task kc · start 2026-09-22T10:36:11Z · expect ~6 min c-darwin-01-dessert-01 darwin -> sagan 0.08s c-darwin-01-dessert-01 darwin -> darwin 0.08s c-darwin-01-dessert-01 darwin -> sagan 0.07s c-darwin-01-dessert-02 darwin -> sagan 0.08s c-darwin-01-dessert-03 darwin -> hypatia 0.08s c-darwin-01-dessert-03 darwin -> darwin 0.08s c-darwin-01-main-01 darwin -> sagan 0.07s c-darwin-01-main-02 darwin -> socrates 0.07s c-darwin-01-main-02 darwin -> darwin 0.08s c-darwin-01-main-02 darwin -> darwin 0.08s c-darwin-01-main-02 darwin -> darwin 0.08s c-darwin-01-main-03 darwin -> darwin 0.08s c-darwin-01-starter-01 darwin -> sagan 0.07s c-darwin-01-starter-01 darwin -> darwin 0.07s c-darwin-01-starter-01 darwin -> sagan 0.07s c-darwin-01-starter-02 darwin -> darwin 0.07s c-darwin-01-starter-03 darwin -> sagan 0.08s c-darwin-01-starter-03 darwin -> darwin 0.08s c-einstein-01-dessert-01 einstein -> sagan 0.08s c-einstein-01-dessert-01 einstein -> sagan 0.08s c-einstein-01-dessert-02 einstein -> sagan 0.08s c-einstein-01-dessert-02 einstein -> ibn_sina 0.07s c-einstein-01-dessert-03 einstein -> sagan 0.08s c-einstein-01-main-01 einstein -> sagan 0.07s c-einstein-01-main-01 einstein -> einstein 0.08s c-einstein-01-main-02 einstein -> sagan 0.08s c-einstein-01-main-03 einstein -> sagan 0.07s c-einstein-01-main-03 einstein -> einstein 0.07s c-einstein-01-main-03 einstein -> einstein 0.08s c-einstein-01-main-03 einstein -> einstein 0.08s c-einstein-01-starter-01 einstein -> einstein 0.07s c-einstein-01-starter-02 einstein -> einstein 0.07s c-einstein-01-starter-02 einstein -> sagan 0.07s c-einstein-01-starter-03 einstein -> sagan 0.08s c-einstein-01-starter-03 einstein -> einstein 0.07s c-einstein-01-starter-03 einstein -> einstein 0.07s c-hypatia-01-dessert-01 hypatia -> socrates 0.08s c-hypatia-01-dessert-02 hypatia -> sagan 0.07s c-hypatia-01-dessert-02 hypatia -> sagan 0.07s c-hypatia-01-dessert-03 hypatia -> einstein 0.07s c-hypatia-01-main-01 hypatia -> hypatia 0.08s c-hypatia-01-main-01 hypatia -> ibn_sina 0.07s c-hypatia-01-main-01 hypatia -> sagan 0.08s c-hypatia-01-main-01 hypatia -> socrates 0.07s c-hypatia-01-main-01 hypatia -> einstein 0.07s c-hypatia-01-main-02 hypatia -> sagan 0.08s c-hypatia-01-main-03 hypatia -> hypatia 0.08s c-hypatia-01-main-03 hypatia -> ibn_sina 0.07s c-hypatia-01-main-03 hypatia -> hypatia 0.08s c-hypatia-01-starter-01 hypatia -> hypatia 0.07s c-hypatia-01-starter-01 hypatia -> ibn_sina 0.07s c-hypatia-01-starter-02 hypatia -> hypatia 0.07s c-hypatia-01-starter-02 hypatia -> hypatia 0.07s c-hypatia-01-starter-02 hypatia -> sagan 0.07s c-ibn_sina-01-dessert-01 ibn_sina -> sagan 0.07s c-ibn_sina-01-dessert-02 ibn_sina -> ibn_sina 0.07s c-ibn_sina-01-dessert-02 ibn_sina -> sagan 0.07s c-ibn_sina-01-dessert-02 ibn_sina -> sagan 0.07s c-ibn_sina-01-dessert-03 ibn_sina -> sagan 0.08s c-ibn_sina-01-dessert-03 ibn_sina -> ibn_sina 0.08s c-ibn_sina-01-main-02 ibn_sina -> sagan 0.07s c-ibn_sina-01-main-02 ibn_sina -> ibn_sina 0.08s c-ibn_sina-01-main-02 ibn_sina -> sagan 0.08s c-ibn_sina-01-main-03 ibn_sina -> ibn_sina 0.08s c-ibn_sina-01-main-03 ibn_sina -> socrates 0.08s c-ibn_sina-01-main-03 ibn_sina -> ibn_sina 0.08s c-ibn_sina-01-starter-01 ibn_sina -> ibn_sina 0.07s c-ibn_sina-01-starter-02 ibn_sina -> socrates 0.07s c-ibn_sina-01-starter-03 ibn_sina -> ibn_sina 0.08s c-ibn_sina-01-starter-03 ibn_sina -> ibn_sina 0.08s c-ibn_sina-01-starter-03 ibn_sina -> sagan 0.08s c-ibn_sina-01-starter-03 ibn_sina -> sagan 0.08s c-sagan-01-dessert-01 sagan -> sagan 0.08s c-sagan-01-dessert-01 sagan -> sagan 0.07s c-sagan-01-dessert-01 sagan -> sagan 0.07s c-sagan-01-dessert-03 sagan -> sagan 0.07s c-sagan-01-dessert-03 sagan -> sagan 0.08s c-sagan-01-dessert-03 sagan -> sagan 0.07s c-sagan-01-dessert-03 sagan -> sagan 0.08s c-sagan-01-dessert-03 sagan -> sagan 0.08s c-sagan-01-main-01 sagan -> einstein 0.07s c-sagan-01-main-02 sagan -> sagan 0.08s c-sagan-01-main-02 sagan -> hypatia 0.07s c-sagan-01-main-03 sagan -> sagan 0.08s c-sagan-01-main-03 sagan -> sagan 0.08s c-sagan-01-starter-01 sagan -> sagan 0.07s c-sagan-01-starter-02 sagan -> sagan 0.08s c-sagan-01-starter-02 sagan -> sagan 0.07s c-sagan-01-starter-03 sagan -> sagan 0.07s c-sagan-01-starter-03 sagan -> einstein 0.08s c-socrates-01-dessert-01 socrates -> socrates 0.08s c-socrates-01-dessert-01 socrates -> socrates 0.07s c-socrates-01-dessert-01 socrates -> sagan 0.08s c-socrates-01-dessert-02 socrates -> sagan 0.07s c-socrates-01-dessert-02 socrates -> socrates 0.08s c-socrates-01-dessert-02 socrates -> socrates 0.07s c-socrates-01-main-01 socrates -> socrates 0.07s c-socrates-01-main-02 socrates -> socrates 0.08s c-socrates-01-main-02 socrates -> socrates 0.07s c-socrates-01-main-02 socrates -> socrates 0.08s c-socrates-01-main-02 socrates -> socrates 0.07s c-socrates-01-main-03 socrates -> sagan 0.07s c-socrates-01-main-03 socrates -> ibn_sina 0.08s c-socrates-01-starter-01 socrates -> socrates 0.07s c-socrates-01-starter-02 socrates -> socrates 0.07s c-socrates-01-starter-03 socrates -> socrates 0.07s c-socrates-01-starter-03 socrates -> socrates 0.08s c-socrates-01-starter-03 socrates -> socrates 0.07s {"n": 108, "n_reached": 108, "n_labelled": 108, "refused": 0, "refused_by_reason": {}, "correct": 60, "errors": 0, "brier": 0.7020824074074073, "reliability": [{"lo": 0.0, "hi": 0.1, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.1, "hi": 0.2, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.2, "hi": 0.3, "n": 0, "sum_p": 0.0, "correct": 0, "mean_p": null, "accuracy": null, "gap": null}, {"lo": 0.3, "hi": 0.4, "n": 3, "sum_p": 1.0899999999999999, "correct": 1, "mean_p": 0.3633333333333333, "accuracy": 0.3 [2026-09-22T10:36:31Z] kev-4b-raw RC=0 [2026-09-22T10:36:31Z] stopping kev server pid 17806 GPU-7aa0be10-974f-6430-52fe-09018a0e2e08, 1 MiB, 30.62 W GPU-46890836-d8f7-e868-1977-c1ce28db0d7a, 1 MiB, 24.65 W [2026-09-22T10:36:41Z] === both boards, free: GPU-7aa0be10-974f-6430-52fe-09018a0e2e08, 1 MiB, 30.62 W GPU-46890836-d8f7-e868-1977-c1ce28db0d7a, 1 MiB, 24.65 W /dev/mapper/ubuntu--vg-ubuntu--lv 295G 196G 86G 70% / [2026-09-22T10:36:41Z] KEV CHAIN END