ANNOUNCE  box benchbox · card 0 GPU-46890836-d8f (one RTX 3090, 250 W cap; card 1 stays dark) · arm 12 · kev.serve jaredpalmer/kev-9b bf16 served · start 2026-09-22T10:16:03Z · expect ~5 min to ready, then ~40 min of task arms

Fetching 9 files:   0%|          | 0/9 [00:00<?, ?it/s]
Fetching 9 files: 100%|██████████| 9/9 [00:00<00:00, 2020.59it/s]
/workshop/kev/.venv/lib/python3.12/site-packages/fla/utils/_device.py:100: UserWarning: Triton is not supported on current platform, roll back to CPU.
  _cpu_device_warning()
/workshop/kev/.venv/lib/python3.12/site-packages/fla/utils/_device.py:177: UserWarning: Triton is not supported on current platform, roll back to CPU.
  _cpu_device_warning()

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 1875.60it/s]

Loading weights:   0%|          | 0/427 [00:00<?, ?it/s]
Loading weights:  57%|█████▋    | 243/427 [00:00<00:00, 2419.89it/s]
Loading weights: 100%|██████████| 427/427 [00:00<00:00, 2518.84it/s]
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
INFO:     Started server process [11652]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8009 (Press CTRL+C to quit)
serving jaredpalmer/kev-9b (/workshop/.cache/huggingface/hub/models--jaredpalmer--kev-9b/snapshots/2629c06a5aeb0feb3b9783bafed17ed8f39ecf5c) on cuda :8009
INFO:     127.0.0.1:43830 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:43832 - "GET /v1/models HTTP/1.1" 200 OK
[transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel.
INFO:     127.0.0.1:49782 - "POST /v1/systemone HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/workshop/kev/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 411, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 63, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1163, in __call__
    await super().__call__(scope, receive, send)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/starlette/applications.py", line 96, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/workshop/kev/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 88, in __call__
    await self.app(scope, receive, send)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/workshop/kev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 670, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 2734, in app
    await route.handle(scope, receive, send)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 1281, in handle
    await super().handle(scope, receive, send)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 280, in handle
    await self.app(scope, receive, send)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 158, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/workshop/kev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 144, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 706, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 354, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/starlette/concurrency.py", line 34, in run_in_threadpool
    return await anyio.to_thread.run_sync(func)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/anyio/to_thread.py", line 65, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2706, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1100, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/kev/serve.py", line 86, in systemone
    return server().answer(req)
           ^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/kev/serve.py", line 65, in answer
    ps, m = self.probs(rec)
            ^^^^^^^^^^^^^^^
  File "/workshop/kev/kev/serve.py", line 58, in probs
    ps = self.model.probs(enc)
         ^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/kev/model.py", line 244, in probs
    return [F.softmax(z, -1).cpu() for z in self.forward(enc)]
                                            ^^^^^^^^^^^^^^^^^
  File "/workshop/kev/kev/model.py", line 233, in forward
    return self.forward_batch([enc])[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/kev/model.py", line 238, in forward_batch
    if self.hybrid: return self.forward_rows_batch(encs)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/kev/model.py", line 225, in forward_rows_batch
    h = self.lm(input_ids=ids, position_ids=pos, attention_mask=att).last_hidden_state.float()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/peft/peft_model.py", line 3198, in forward
    return self.base_model(
           ^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/peft/tuners/tuners_utils.py", line 359, in forward
    return self.model(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 1068, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/transformers/utils/output_capturing.py", line 286, in wrapper
    outputs = func(self, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/transformers/models/qwen3_5/modeling_qwen3_5.py", line 1292, in forward
    hidden_states = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/transformers/modeling_layers.py", line 109, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/transformers/models/qwen3_5/modeling_qwen3_5.py", line 889, in forward
    hidden_states = self.linear_attn(
                    ^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/transformers/integrations/accelerate.py", line 1010, in wrapped
    output = forward_func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/transformers/models/qwen3_5/modeling_qwen3_5.py", line 639, in forward
    core_attn_out, last_recurrent_state = torch_chunk_gated_delta_rule(
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/transformers/integrations/hub_kernels.py", line 885, in wrapped
    return implementation(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fla/ops/gated_delta_rule/chunk.py", line 567, in chunk_gated_delta_rule
    o, final_state = ChunkGatedDeltaRuleFunction.apply(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/autograd/function.py", line 576, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fla/utils/_decorators.py", line 152, in wrapper
    return fn(*processed_args, **processed_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/amp/autocast_mode.py", line 517, in decorate_fwd
    return fwd(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fla/ops/gated_delta_rule/chunk.py", line 282, in forward
    q, q_rstd = l2norm_fwd(q)
                ^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fla/ops/backends/__init__.py", line 214, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fla/modules/l2norm.py", line 178, in l2norm_fwd
    l2norm_fwd_kernel[grid](
  File "/workshop/kev/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 390, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/fla/ops/utils/cache.py", line 378, in run
    return super().run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/triton/runtime/autotuner.py", line 237, in run
    used_cached_result = self.check_disk_cache(key, pruned_configs, benchmark)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/triton/runtime/autotuner.py", line 184, in check_disk_cache
    make_backend(driver.active.get_current_target()).hash(),
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/triton/runtime/driver.py", line 30, in __getattr__
    return getattr(self._initialize_obj(), name)
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/triton/runtime/driver.py", line 26, in _initialize_obj
    self._obj = self._init_fn()
                ^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/triton/runtime/driver.py", line 12, in _create_driver
    return active_drivers[0]()
           ^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 715, in __init__
    self.utils = CudaUtils()  # TODO: make static
                 ^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 62, in __init__
    mod = compile_module_from_src(
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/triton/runtime/build.py", line 88, in compile_module_from_src
    so = _build(name, src_path, tmpdir, library_dirs or [], include_dirs or [], libraries or [])
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workshop/kev/.venv/lib/python3.12/site-packages/triton/runtime/build.py", line 32, in _build
    raise RuntimeError(
RuntimeError: Failed to find C compiler. Please specify via CC environment variable or set triton.knobs.build.impl.
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [11652]