Should we do something like https://github.com/NixOS/nixpkgs/blob/5c1d2b0d06241279e6a70e276397a5e40e867499/pkgs/build-support/alternatives/blas/default.nix? Or should we ask upstream why don't they generate those?

For context, openmpi more or less maps onto cmake's expectations, whereas mpich just merges everything into a single file:

❯ ls /nix/store/nf8fqx39w8ib34hp24lrz48287xdbxd8-openmpi-4.1.6/lib/pkgconfig/
ompi-c.pc  ompi-cxx.pc  ompi-f77.pc  ompi-f90.pc  ompi-fort.pc  ompi.pc  orte.pc
❯ ls /nix/store/24sv27w3j1j3p7lxyh689bzbhmixxf35-mpich-4.1.2/lib/pkgconfig
mpich.pc
❯ cat /nix/store/24sv27w3j1j3p7lxyh689bzbhmixxf35-mpich-4.1.2/lib/pkgconfig/mpich.pc
...
Cflags:   -I${includedir}
...
# pkg-config does not understand Cxxflags, etc. So we allow users to
# query them using the --variable option

cxxflags=  -I${includedir}
fflags=-fallow-argument-mismatch -I${includedir}
fcflags=-fallow-argument-mismatch -I${includedir}

But apptainer doesn't seem to even run it:

❯ NVIDIA_VISIBLE_DEVICES=all strace ./result/bin/singularity exec --nv --nvccli writable.img python -c "" |& rg nvidia
futex(0x55fed1c4c888, FUTEX_WAIT_PRIVATE, 0, NULLINFO:    Setting --writable-tmpfs (required by nvidia-container-cli)

Although it claims it does:

❯ APPTAINER_MESSAGELEVEL=100000 NVIDIA_VISIBLE_DEVICES=all ./result/bin/singularity exec --nv --nvccli writable.img python -c ""
...
DEBUG   [U=1001,P=888790]  create()                      nvidia-container-cli
DEBUG   [U=1001,P=888822]  findOnPath()                  Found "nvidia-container-cli" at "/run/current-system/sw/bin/nvidia-container-cli"
DEBUG   [U=1001,P=888822]  findOnPath()                  Found "ldconfig" at "/run/current-system/sw/bin/ldconfig"
DEBUG   [U=1001,P=888822]  NVCLIConfigure()              nvidia-container-cli binary: "/run/current-system/sw/bin/nvidia-container-cli" args: ["--user" "configure" "--no-cgroups" "--device=all" "--compute" "--utility" "--ldconfig=@/run/current-system/sw/bin/ldconfig" "/nix/store/rzycmg66zpap6gjb5ylmvd8ymlfb7fag-apptainer-1.2.5/var/lib/apptainer/mnt/session/final"]
DEBUG   [U=1001,P=888822]  NVCLIConfigure()              Running nvidia-container-cli in user namespace
DEBUG   [U=1001,P=888790]  create()                      Chroot into /nix/store/rzycmg66zpap6gjb5ylmvd8ymlfb7fag-apptainer-1.2.5/var/lib/apptainer/mnt/session/final
...

Btw

❯ strace /run/current-system/sw/bin/nvidia-container-cli "--user" "configure" "--no-cgroups" "--device=all" "--compute" "--utility" "--ldconfig=@/run/current-system/sw/bin/ldconfig" "/nix/store/rzycmg66zpap6gjb5ylmvd8ymlfb7fag-apptainer-1.2.5/var/lib/apptainer/mnt/session/final"
...
capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=0, permitted=1<<CAP_CHOWN|1<<CAP_DAC_OVERRIDE|1<<CAP_DAC_READ_SEARCH|1<<CAP_FOWNER|1<<CAP_KILL|1<<CAP_SETGID|1<<CAP_SETUID|1<<CAP_SETPCAP|1<<CAP_NET_ADMIN|1<<CAP_SYS_CHROOT|1<<CAP_SYS_PTRACE|1<<CAP_SYS_ADMIN|1<<CAP_MKNOD, inheritable=1<<CAP_WAKE_ALARM}) = -1 EPERM (Operation not permitted)
...
nvidia-container-cli: permission error: capability change failed: operation not permitted

Was this even supposed to work without root?

In reply to @ss:someonex.net

Although it claims it does:

❯ APPTAINER_MESSAGELEVEL=100000 NVIDIA_VISIBLE_DEVICES=all ./result/bin/singularity exec --nv --nvccli writable.img python -c ""
...
DEBUG   [U=1001,P=888790]  create()                      nvidia-container-cli
DEBUG   [U=1001,P=888822]  findOnPath()                  Found "nvidia-container-cli" at "/run/current-system/sw/bin/nvidia-container-cli"
DEBUG   [U=1001,P=888822]  findOnPath()                  Found "ldconfig" at "/run/current-system/sw/bin/ldconfig"
DEBUG   [U=1001,P=888822]  NVCLIConfigure()              nvidia-container-cli binary: "/run/current-system/sw/bin/nvidia-container-cli" args: ["--user" "configure" "--no-cgroups" "--device=all" "--compute" "--utility" "--ldconfig=@/run/current-system/sw/bin/ldconfig" "/nix/store/rzycmg66zpap6gjb5ylmvd8ymlfb7fag-apptainer-1.2.5/var/lib/apptainer/mnt/session/final"]
DEBUG   [U=1001,P=888822]  NVCLIConfigure()              Running nvidia-container-cli in user namespace
DEBUG   [U=1001,P=888790]  create()                      Chroot into /nix/store/rzycmg66zpap6gjb5ylmvd8ymlfb7fag-apptainer-1.2.5/var/lib/apptainer/mnt/session/final
...

Propagating --debug to nvidia-container-cli: https://gist.github.com/SomeoneSerge/a4317ccec07e33324c588eb6f7c6f04a#file-gistfile0-txt-L310

Which is the same as if you manually run

❯ strace /run/current-system/sw/bin/nvidia-container-cli "--user" "configure" "--no-cgroups" "--device=all" "--compute" "--utility" "--ldconfig=@/run/current-system/sw/bin/ldconfig" "/nix/store/rzycmg66zpap6gjb5ylmvd8ymlfb7fag-apptainer-1.2.5/var/lib/apptainer/mnt/session/final"
...
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=0, permitted=0, inheritable=1<<CAP_WAKE_ALARM}) = 0
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=0, permitted=1<<CAP_CHOWN|1<<CAP_DAC_OVERRIDE|1<<CAP_DAC_READ_SEARCH|1<<CAP_FOWNER|1<<CAP_KILL|1<<CAP_SETGID|1<<CAP_SETUID|1<<CAP_SETPCAP|1<<CAP_NET_ADMIN|1<<CAP_SYS_CHROOT|1<<CAP_SYS_PTRACE|1<<CAP_SYS_ADMIN|1<<CAP_MKNOD, inheritable=1<<CAP_WAKE_ALARM}) = -1 EPERM (Operation not permitted)
write(2, "nvidia-container-cli: ", 22nvidia-container-cli: )  = 22
write(2, "permission error: capability cha"..., 67permission error: capability change failed: operation not permitted) = 67
write(2, "\n", 1
)                       = 1
exit_group(1)                           = ?
+++ exited with 1 +++

Propagating --debug to nvidia-container-cli: https://gist.github.com/SomeoneSerge/a4317ccec07e33324c588eb6f7c6f04a#file-gistfile0-txt-L310

Which is the same as if you manually run https://matrix.to/#/%23hpc%3Anixos.org/%24aCLdJvRqyXSNc0_LfuTb7tFxL3hBhMfXUzc13whct0U?via=someonex.net&via=matrix.org&via=kde.org&via=dodsorf.as

Btw

❯ strace /run/current-system/sw/bin/nvidia-container-cli "--user" "configure" "--no-cgroups" "--device=all" "--compute" "--utility" "--ldconfig=@/run/current-system/sw/bin/ldconfig" "/nix/store/rzycmg66zpap6gjb5ylmvd8ymlfb7fag-apptainer-1.2.5/var/lib/apptainer/mnt/session/final"
...
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=0, permitted=0, inheritable=1<<CAP_WAKE_ALARM}) = 0
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=0, permitted=1<<CAP_CHOWN|1<<CAP_DAC_OVERRIDE|1<<CAP_DAC_READ_SEARCH|1<<CAP_FOWNER|1<<CAP_KILL|1<<CAP_SETGID|1<<CAP_SETUID|1<<CAP_SETPCAP|1<<CAP_NET_ADMIN|1<<CAP_SYS_CHROOT|1<<CAP_SYS_PTRACE|1<<CAP_SYS_ADMIN|1<<CAP_MKNOD, inheritable=1<<CAP_WAKE_ALARM}) = -1 EPERM (Operation not permitted)
write(2, "nvidia-container-cli: ", 22nvidia-container-cli: )  = 22
write(2, "permission error: capability cha"..., 67permission error: capability change failed: operation not permitted) = 67
write(2, "\n", 1
)                       = 1
exit_group(1)                           = ?
+++ exited with 1 +++
...
nvidia-container-cli: permission error: capability change failed: operation not permitted

Was this even supposed to work without root?

Propagating --debug to nvidia-container-cli: https://gist.github.com/SomeoneSerge/a4317ccec07e33324c588eb6f7c6f04a#file-gistfile0-txt-L310

Which is the same as if you manually run https://matrix.to/#/%23hpc%3Anixos.org/%24aCLdJvRqyXSNc0_LfuTb7tFxL3hBhMfXUzc13whct0U?via=someonex.net&via=matrix.org&via=kde.org&via=dodsorf.as

Like, did it even require CAP_SYS_ADMIN before?

❯ ag eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee result-lib/ --search-binary
result-lib/lib/security/pam_slurm_adopt.la
41:libdir='/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-slurm-23.11.1.1/lib/security'

result-lib/lib/perl5/5.38.2/x86_64-linux-thread-multi/perllocal.pod
7:C<installed into: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-slurm-23.11.1.1/lib/perl5/site_perl/5.38.2>
29:C<installed into: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-slurm-23.11.1.1/lib/perl5/site_perl/5.38.2>

Binary file result-lib/lib/libslurm.so.40.0.0 matches.

result-lib/lib/security/pam_slurm.la
41:libdir='/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-slurm-23.11.1.1/lib/security'

Binary file result-lib/lib/slurm/libslurmfull.so matches.

Binary file result-lib/lib/slurm/mpi_pmi2.so matches.

Binary file result-lib/lib/slurm/libslurm_pmi.so matches
❯ strings result-lib/lib/slurm/mpi_pmi2.so | rg eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-slurm-23.11.1.1/bin/srun

arghhghhhghghghghghggh why