Btw

❯ strace /run/current-system/sw/bin/nvidia-container-cli "--user" "configure" "--no-cgroups" "--device=all" "--compute" "--utility" "--ldconfig=@/run/current-system/sw/bin/ldconfig" "/nix/store/rzycmg66zpap6gjb5ylmvd8ymlfb7fag-apptainer-1.2.5/var/lib/apptainer/mnt/session/final"
...
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=0, permitted=0, inheritable=1<<CAP_WAKE_ALARM}) = 0
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=0, permitted=1<<CAP_CHOWN|1<<CAP_DAC_OVERRIDE|1<<CAP_DAC_READ_SEARCH|1<<CAP_FOWNER|1<<CAP_KILL|1<<CAP_SETGID|1<<CAP_SETUID|1<<CAP_SETPCAP|1<<CAP_NET_ADMIN|1<<CAP_SYS_CHROOT|1<<CAP_SYS_PTRACE|1<<CAP_SYS_ADMIN|1<<CAP_MKNOD, inheritable=1<<CAP_WAKE_ALARM}) = -1 EPERM (Operation not permitted)
write(2, "nvidia-container-cli: ", 22nvidia-container-cli: )  = 22
write(2, "permission error: capability cha"..., 67permission error: capability change failed: operation not permitted) = 67
write(2, "\n", 1
)                       = 1
exit_group(1)                           = ?
+++ exited with 1 +++
...
nvidia-container-cli: permission error: capability change failed: operation not permitted

Was this even supposed to work without root?

Propagating --debug to nvidia-container-cli: https://gist.github.com/SomeoneSerge/a4317ccec07e33324c588eb6f7c6f04a#file-gistfile0-txt-L310

Which is the same as if you manually run https://matrix.to/#/%23hpc%3Anixos.org/%24aCLdJvRqyXSNc0_LfuTb7tFxL3hBhMfXUzc13whct0U?via=someonex.net&via=matrix.org&via=kde.org&via=dodsorf.as

Like, did it even require CAP_SYS_ADMIN before?

❯ ag eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee result-lib/ --search-binary
result-lib/lib/security/pam_slurm_adopt.la
41:libdir='/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-slurm-23.11.1.1/lib/security'

result-lib/lib/perl5/5.38.2/x86_64-linux-thread-multi/perllocal.pod
7:C<installed into: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-slurm-23.11.1.1/lib/perl5/site_perl/5.38.2>
29:C<installed into: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-slurm-23.11.1.1/lib/perl5/site_perl/5.38.2>

Binary file result-lib/lib/libslurm.so.40.0.0 matches.

result-lib/lib/security/pam_slurm.la
41:libdir='/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-slurm-23.11.1.1/lib/security'

Binary file result-lib/lib/slurm/libslurmfull.so matches.

Binary file result-lib/lib/slurm/mpi_pmi2.so matches.

Binary file result-lib/lib/slurm/libslurm_pmi.so matches
❯ strings result-lib/lib/slurm/mpi_pmi2.so | rg eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-slurm-23.11.1.1/bin/srun

arghhghhhghghghghghggh why

markuskowa

Linking slurm's libpmi2 seems to kind of work at aalto:

❯ ssh triton srun -N3 --mpi=pmi2 singularity exec cpi.sif cpi
srun: job 27525153 queued and waiting for resources
srun: job 27525153 has been allocated resources
...
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   csl13
  Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   csl2
  Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   csl11
  Local device: mlx5_0
--------------------------------------------------------------------------
Process 1 of 3 is on csl11.int.triton.aalto.fi
Process 2 of 3 is on csl13.int.triton.aalto.fi
Process 0 of 3 is on csl2.int.triton.aalto.fi
pi is approximately 3.1415926544231318, Error is 0.0000000008333387
wall clock time = 0.043120