In reply to @ss:someonex.net
I think I'd vote pro propagation if we could say with some certainty, that that is the only way to guarantee correctness for users of libcudart_static and of cmake's CUDA::cuda_driver (just because supporting that scope sounds doable)

Not specific to nixos, but just a rant from me: there's been a pretty large push around the cuda world for everyone to move to static libcudart... largely because with cuda 12 they introduced the minor version compatibility and "cuda enhanced compatibility" guarantees, and there's a lot of public statements (on github, etc.) from nvidia that suggests this is the safest way to distribute packages. All of this is really complicated and I don't fault projects for moving forward under this guidance, but i'm pretty confident that this does not cover all cases and you do still need to think about this stuff.

One example of where you still need to think about it: a lot of code uses the runtime API to resolve the driver API (through cudaGetDriverEntrypoint). The returned function pointers are given by min(linked_runtime_api_ver, actual_driver_version), exclusively. There's no automatic detection of another copy of libcudart in the same process that would allow for automatically matching the API version -- it's exclusively based on what you linked against compared to the driver version in use. (There's no way to implement API-level alignment between libraries in the same process; they would need a way to invalidate fnptrs they've already handed out when they suddenly encounter some new library in the process operating at a new version.)

This is a really easy way to run afoul of the cuda version mixing guidelines, and I feel like it's pretty underdiscussed and underdocumented. Those version mixing guidelines are still important, minor version compatibility does not save you, it's not the case that if they all start with "12" you don't have to think about it anymore.

In reply to @ss:someonex.net

a footgun people keep firing,

True

autoAddDriverRunpath

Yes and no. Yes because that'd definitely make one-off and our own contributions easier. No because once we start propagating it we lose the knowledge of which packages actually need to be patched. It still seems to me that most packages we don't have to patch because they call cudart and cudart is patchelfed. Maybe yes because I'm unsure what happens with libcudart_static.

autoPatchelfHook

I'd be rather strongly opposed to this one. Autopatchelf is a huge hammer, coarse and imprecise. It can actually erase correct runpaths from an originally correct binary. Let's reserve it for non

Another important thing to consider is (here we go again) whether we want to keep both backendStdenv and the hook and which of these things should be propagating what

In reply to @connorbaker:matrix.org
Had a learning moment today: was wondering why all the traffic between my remote builders, which are all networked together in the same room, kept going through the router instead of the switch. Well, turns out if you don't specify a mask with Address in a systemd.network configuration, it defaults to /32... so all the builders thought they were alone on their network, and all traffic to other IP addresses went through the gateway (my router).

In reply to @glepage:matrix.org
Good to know indeed...
Is your builder configuration public connor (he/him) (UTC-7) ?

Here’s an example of the interlaced build: https://github.com/ConnorBaker/cuda-packages/blob/main/cudaPackages-common/onnx-tensorrt.nix

Note that it does avoid building the library twice!