!eWOErHSaiddIbsUNsJ:nixos.org

NixOS CUDA

286 Members
CUDA packages maintenance and support in nixpkgs | https://github.com/orgs/NixOS/projects/27/ | https://nixos.org/manual/nixpkgs/unstable/#cuda58 Servers

Load older messages

SenderMessageTime
10 Sep 2025
@ss:someonex.netSomeoneSerge (back on matrix)
In reply to @hugo:okeso.eu

I managed to launch a test on my package with CUDA enabled, but I get an issue from triton not finding a C compiler. Does that ring a bell to someone ?

RuntimeError: Failed to find C compiler. Please specify via CC environment variable or set triton.knobs.build.impl.

I share my work in progress in a draft PR here: https://github.com/NixOS/nixpkgs/pull/441728

At the very least we recently stopped early-binding rocm libraries, maybe hard-coded compiler paths went with them. Try giving it a compiler at test time as suggested in the error? 		10:05:31
@hugo:okeso.euHugoI (vibe) tried to give it a compiler here in this commit https://github.com/NixOS/nixpkgs/pull/441728/commits/81f7997ca1ca37193f2f26fdbc85c586a92ba6dd but was unsuccessful. Any suggestion how to do that?10:06:58
@matthewcroughan:defenestrate.itmatthewcroughan @ 39c3 (DECT 94667 or 97340 or 67192) changed their display name from matthewcroughan @ nixcon to matthewcroughan.15:02:50
@lt1379:matrix.orgLunThat should only impact ROCm unless I messed it up! diff was https://github.com/NixOS/nixpkgs/commit/c74e5ffb6526ac1b4870504921b9ba9362189a1715:52:00
@layus:matrix.orglayus joined the room.18:26:24
@layus:matrix.orglayusIs this team involved in flox/nvidia partnership ? (See https://flox.dev/cuda/) I guess so since the nixos foundation also is, but there is no mention of this team or its amazing work.18:30:16
@matthewcroughan:defenestrate.itmatthewcroughan @ 39c3 (DECT 94667 or 97340 or 67192)

adrian-gierakowski:

!!! Exception during processing !!! HIP error: invalid device function
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

Traceback (most recent call last):
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/execution.py", line 496, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/execution.py", line 315, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/execution.py", line 289, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/execution.py", line 277, in process_inputs
    result = f(**inputs)
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/nodes.py", line 74, in encode
    return (clip.encode_from_tokens_scheduled(tokens), )
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/comfy/sd.py", line 170, in encode_from_tokens_scheduled
    pooled_dict = self.encode_from_tokens(tokens, return_pooled=return_pooled, return_dict=True)
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/comfy/sd.py", line 232, in encode_from_tokens
    o = self.cond_stage_model.encode_token_weights(tokens)
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/comfy/sd1_clip.py", line 689, in encode_token_weights
    out = getattr(self, self.clip).encode_token_weights(token_weight_pairs)
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/comfy/sd1_clip.py", line 45, in encode_token_weights
    o = self.encode(to_encode)
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/comfy/sd1_clip.py", line 291, in encode
    return self(tokens)
  File "/nix/store/jzm64j9dp50xs770h3w7n8h9pj6mpkjp-python3.13-torch-2.8.0/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/jzm64j9dp50xs770h3w7n8h9pj6mpkjp-python3.13-torch-2.8.0/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/comfy/sd1_clip.py", line 253, in forward
    embeds, attention_mask, num_tokens, embeds_info = self.process_tokens(tokens, device)
                                                      ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/comfy/sd1_clip.py", line 204, in process_tokens
    tokens_embed = self.transformer.get_input_embeddings()(tokens_embed, out_dtype=torch.float32)
  File "/nix/store/jzm64j9dp50xs770h3w7n8h9pj6mpkjp-python3.13-torch-2.8.0/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/jzm64j9dp50xs770h3w7n8h9pj6mpkjp-python3.13-torch-2.8.0/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/comfy/ops.py", line 270, in forward
    return self.forward_comfy_cast_weights(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/dg5g3ypdsjvy0274156l74klx4wr0nbx-comfyui-unstable-2025-09-06/lib/python3.13/site-packages/comfy/ops.py", line 266, in forward_comfy_cast_weights
    return torch.nn.functional.embedding(input, weight, self.padding_idx, self.max_norm, self.norm_type, self.scale_grad_by_freq, self.sparse).to(dtype=output_dtype)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jzm64j9dp50xs770h3w7n8h9pj6mpkjp-python3.13-torch-2.8.0/lib/python3.13/site-packages/torch/nn/functional.py", line 2546, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: HIP error: invalid device function
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.
19:53:26
@matthewcroughan:defenestrate.itmatthewcroughan @ 39c3 (DECT 94667 or 97340 or 67192)After a whole day recompiling torch :)19:53:32
@matthewcroughan:defenestrate.itmatthewcroughan @ 39c3 (DECT 94667 or 97340 or 67192)

Actually, with the HSA override to 11.0.0 it worked, but I get a different kind of error

loaded completely 30779.053608703613 1639.406135559082 True
  0%|                                    | 0/1 [00:00<?, ?it/s:0:rocdevice.cpp            :3020: 78074348282d us:  Callback: Queue 0x7f831c600000 aborting with error : HSA_STATUS_ERROR_INVALID_ISA: The instruction set architecture is invalid. code: 0x100f
19:58:53
@matthewcroughan:defenestrate.itmatthewcroughan @ 39c3 (DECT 94667 or 97340 or 67192) *

Actually, with the HSA override to 11.0.0 it worked, but I get a different kind of error

  0%|                                    | 0/1 [00:00<?, ?it/s:0:rocdevice.cpp            :3020: 78074348282d us:  Callback: Queue 0x7f831c600000 aborting with error : HSA_STATUS_ERROR_INVALID_ISA: The instruction set architecture is invalid. code: 0x100f
Aborted                    (core dumped) command nix "$@"```
19:59:14
@sporeray:matrix.orgRobbie Buxton
In reply to @layus:matrix.org
Is this team involved in flox/nvidia partnership ?
(See https://flox.dev/cuda/)
I guess so since the nixos foundation also is, but there is no mention of this team or its amazing work.
Ron mentioned them here https://discourse.nixos.org/t/nix-flox-nvidia-opening-up-cuda-redistribution-on-nix/69189/7 		20:01:06
@matthewcroughan:defenestrate.itmatthewcroughan @ 39c3 (DECT 94667 or 97340 or 67192)Is there a rocm room?20:54:56
@lt1379:matrix.orgLunhttps://matrix.to/#/#ROCm:nixos.org21:40:38
@connorbaker:matrix.orgconnor (burnt/out) (UTC-8) changed their display name from connor (he/him) (UTC+2) to connor (he/him) (UTC-7).22:20:37
@glepage:matrix.orgGaétan Lepage Well done guys for allowing this to happen (connor (he/him) (UTC-7) SomeoneSerge (back on matrix) stick...)
👏 		23:07:17
@glepage:matrix.orgGaétan Lepage * Well done guys for allowing this to happen (connor (he/him) (UTC-7) SomeoneSerge (back on matrix) stick Samuel Ainsworth...)
👏 		23:22:06
@ss:someonex.netSomeoneSerge (back on matrix)The negotiations with NVIDIA have been run by Flox (although in parallel with many other companies' simultaneous inquiries). Ron kept us, the Foundation, and the SC in the loop, and offered both legal help and workforce. The current idea roughly is that the CUDA team gets access to the relevant repo and infra, and work closely together with Flox to secure the position and a commx channel to NVIDIA. 23:26:05
@hexa:lossy.networkhexaWhat were the blockers for setting this up within the NixOS Foundation?23:54:32
11 Sep 2025
@rosscomputerguy:matrix.orgTristan RossFrom what I recall, it was something to do with having a legal entity in the US. If the foundation was registered in the US, then it would've been fine. This has been going on since at least Planet Nix, glad to see it finally pull through.00:03:41
@ihar.hrachyshka:matrix.org@ihar.hrachyshka:matrix.org joined the room.00:09:51
@connorbaker:matrix.orgconnor (burnt/out) (UTC-8)Will try to take a look later00:10:55
@connorbaker:matrix.orgconnor (burnt/out) (UTC-8)You’d probably need to override writeGpuTestPython to use the Python package set you specify. It’s inside cudaPackages so it has no way of knowing what scope you’re using it in.00:12:26
@connorbaker:matrix.orgconnor (burnt/out) (UTC-8) They’re still mostly okay; I’ve been exhausted recently so haven’t been awake too early (I was up at 3am this morning but that’s something else)
If we can do something closer to 8 that might be easier; Kevin Mittman how are you with morning meetings? It’d be good to catch up and discuss what’s been done so far with the database project Serge’s been working on. 		00:14:16
@connorbaker:matrix.orgconnor (burnt/out) (UTC-8)NVIDIA’s EULA effectively prohibits running CUDA binaries they release on non-NVIDIA hardware (see 1.2.8: https://docs.nvidia.com/cuda/eula/index.html#limitations)00:20:07
@justbrowsing:matrix.orgKevin Mittman (EOY sleep)https://developer.nvidia.com/blog/developers-can-now-get-cuda-directly-from-their-favorite-third-party-platforms/01:17:14
@ss:someonex.netSomeoneSerge (back on matrix)The fact remains: it was amd who shut it down, not nvidia?01:26:18
@ss:someonex.netSomeoneSerge (back on matrix)nvidia playing the "please submit this in paper by post, and attach proofs of your residence such as electricity bills delivered to your address" game (being the bureaucrat and coming up with arbitrary terms as they go)01:30:19
@ss:someonex.netSomeoneSerge (back on matrix)ah yeah, true01:33:21
@glepage:matrix.orgGaétan LepageWhen is the next one? I'll try to join10:16:38
@le-chat:matrix.orgle-chat I've updated the gist with the latest version, it seems to compile and run a pipeline with tensor_filter framework=pytorch accelerator=true:gpu ... ! fakesink, but I haven't got a time to check it really. 10:45:16

Show newer messages

Back to Room ListRoom Version: 9