| 16 Oct 2025 |
 raitobezarius | except if you are running in direct local store access | 10:33:48 |
 lillecarl | I'll resolve it eventually, i just need to setup /etc/{shadow,passwd,groups} properly in the container | 10:34:22 |
| raitobezarius | nothing comes to my mind to why you don't get the build fallback scenario, but if you say that --substituters '' works and that running with substituters don't lead to fallback, then this is a bug | 10:34:30 |
| raitobezarius | if you can minimize this to a minimal reproducer with Lix, we can take a look at it if you open an issue | 10:34:46 |
| lillecarl | I'll rewrite it quickly to it retries the build using --substituters '' if it fails (Does that still keep cache.nixos.org?) | 10:35:05 |
| raitobezarius | no, it removes cno | 10:35:13 |
| raitobezarius | --extra-substituters '' would be the right thing to do but it's an append variable, so putting '' is nop | 10:35:40 |
| raitobezarius | * --extra-substituters '' would be the right thing to do but it's an append variable, so putting '' is no-op | 10:35:42 |
| raitobezarius | you could move your unavailable substituter as an extra substituter though | 10:36:14 |
| raitobezarius | and decide to include it if the cache is available or not | 10:36:20 |
| lillecarl | I'll keep cno in the retry attempt then, otherwise this'll be an all-day thing. That should be valid? | 10:36:20 |
| raitobezarius | but all of that is a hack and Lix should do the right thing | 10:36:25 |
| lillecarl | I wanted to put it first to pull as much as possible cluster internal, i'll be back in a few minutes! | 10:36:56 |
| lillecarl | nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:19,922 DEBUG [nix-csi] Running command: nix build --refresh --impure --print-out-paths --out-link /nix/var/nix/gcroots/nix-csi/csi-fd1150e40cfc3dfd33ae3492ebed2247dc0138df14e48a6c47f8be234bc2ecc5 --file /tmp/tmp7ltio7vl.nix
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:19,940 DEBUG [nix-csi] fetching github input 'github:NixOS/nixpkgs/nixos-unstable'
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:20,895 DEBUG [nix-csi] fetching github input 'github:lillecarl/dinix/main'
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:21,519 DEBUG [nix-csi] warning: error: unable to download 'http://nix-cache.default.svc/87bdp04miawiffqc9k8n2m2p172fx6dj.narinfo': Failed to connect to nix-cache.default.svc port 80 after 0 ms: Could not connect to server (curl error code=7); retrying in 270 ms (attempt 1/5)
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:21,789 DEBUG [nix-csi] warning: error: unable to download 'http://nix-cache.default.svc/87bdp04miawiffqc9k8n2m2p172fx6dj.narinfo': Failed to connect to nix-cache.default.svc port 80 after 0 ms: Could not connect to server (curl error code=7); retrying in 661 ms (attempt 2/5)
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:22,451 DEBUG [nix-csi] warning: error: unable to download 'http://nix-cache.default.svc/87bdp04miawiffqc9k8n2m2p172fx6dj.narinfo': Failed to connect to nix-cache.default.svc port 80 after 0 ms: Could not connect to server (curl error code=7); retrying in 1261 ms (attempt 3/5)
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:23,712 DEBUG [nix-csi] warning: error: unable to download 'http://nix-cache.default.svc/87bdp04miawiffqc9k8n2m2p172fx6dj.narinfo': Failed to connect to nix-cache.default.svc port 80 after 0 ms: Could not connect to server (curl error code=7); retrying in 2347 ms (attempt 4/5)
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:26,068 DEBUG [nix-csi] error: unable to download 'http://nix-cache.default.svc/87bdp04miawiffqc9k8n2m2p172fx6dj.narinfo': Failed to connect to nix-cache.default.svc port 80 after 0 ms: Could not connect to server (curl error code=7)
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:26,090 DEBUG [nix-csi] Running command: nix build --substituters https://cache.nixos.org --refresh --impure --print-out-paths --out-link /nix/var/nix/gcroots/nix-csi/csi-fd1150e40cfc3dfd33ae3492ebed2247dc0138df14e48a6c47f8be234bc2ecc5 --file /tmp/tmp7ltio7vl.nix
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:26,106 DEBUG [nix-csi] fetching github input 'github:NixOS/nixpkgs/nixos-unstable'
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:26,849 DEBUG [nix-csi] fetching github input 'github:lillecarl/dinix/main'
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:27,491 DEBUG [nix-csi] this derivation will be built:
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:27,491 DEBUG [nix-csi] /nix/store/g81mfalchy3c729fm3sngljc7zr53nh8-binary-cache-env.drv
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:27,491 DEBUG [nix-csi] this path will be fetched (0.22 MiB download, 0.57 MiB unpacked):
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:27,491 DEBUG [nix-csi] /nix/store/87bdp04miawiffqc9k8n2m2p172fx6dj-ncdu-2.9.1
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:27,512 DEBUG [nix-csi] copying path '/nix/store/87bdp04miawiffqc9k8n2m2p172fx6dj-ncdu-2.9.1' from 'https://cache.nixos.org'...
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:27,643 DEBUG [nix-csi] building '/nix/store/g81mfalchy3c729fm3sngljc7zr53nh8-binary-cache-env.drv'...
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:27,747 DEBUG [nix-csi] /nix/store/gp4cz5k570q4g0swlakjvp1ffvafg0h6-binary-cache-env
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:29,108 DEBUG [nix-csi] Running command: mount --verbose --bind -o ro /nix/var/nix-csi/volumes/csi-fd1150e40cfc3dfd33ae3492ebed2247dc0138df14e48a6c47f8be234bc2ecc5/nix '/var/lib/kubelet/pods/1b3c3282-ce59-4e51-8471-4f45a4f2bed8/volumes/kubernetes.io~csi/nix-csi/mount'
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:29,110 DEBUG [nix-csi] mount: /nix/var/nix-csi/volumes/csi-fd1150e40cfc3dfd33ae3492ebed2247dc0138df14e48a6c47f8be234bc2ecc5/nix bound on /var/lib/kubelet/pods/1b3c3282-ce59-4e51-8471-4f45a4f2bed8/volumes/kubernetes.io~csi/nix-csi/mount.
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:30,364 ERROR [nix-csi] nix copy failed: nixCopy.returncode=1
nix-csi-node-6686w nix-csi-node nixCopy.combined="ssh: connect to host nix-cache.default.svc port 22: Connection refused\nerror: cannot connect to 'nix-cache'"
nix-csi-node-6686w nix-csi-node 2025-10-16 10:43:40,629 INFO [nix-csi] 406 paths copied to cache in 0.26 seconds
| 10:43:55 |
| lillecarl | (The error on the end is because the builder is trying to push to the cache that it's currently bootstrapping, it retries after 10s and succeeds pushing). | 10:45:00 |
| lillecarl | [12:45:05]❌130 ❯ kubectl get pods
NAME READY STATUS RESTARTS AGE
nix-cache-0 1/1 Running 0 107s
nix-csi-node-6686w 3/3 Running 0 2m10s
Yayyy 😄
| 10:45:30 |
 aloisw | It seems you're thinking of the online detection here. --offline works pretty reliably for what it does (it doesn't disable remote builds or FOD builds). | 10:49:36 |
| raitobezarius | you're right, it's just that this feature represent so many things for many people and it's bizarre somehow | 11:03:41 |
| raitobezarius | the notion of "offline"/"online" is not clear cut | 11:03:52 |
| lillecarl | raitobezarius: Thanks for the feedback. I implemented a workaround for now https://github.com/Lillecarl/nix-csi/commit/8a2a9cf38ad0ff2b05afdbbbd16105de848363b6
Would you like me to open an issue or will you? I've struggled with this issue previously and the more I read it seems to be "intended behaviour" with how the ~/.cache/nix works | 11:07:18 |
| raitobezarius | If you have a minimal reproducer independent of your CSI specific code, that'd be great, yes | 11:07:40 |
| lillecarl | I'll create a reproducer script 😄 | 11:16:48 |
| lillecarl | Thanks for confirming it's not "intended/expected behavior". I've been going mad about this | 11:17:06 |
| lillecarl | It's harder than you'd think to reproduce this on a single machine without reinventing containers 😆 | 13:44:13 |
| raitobezarius | unshare is just here if you need isolated network namespaces ;-) | 13:48:20 |
 QuadRadical (Ping) | sorry @raitobezarius i forgor to ping you yesterday when you said, so doing it today :3 | 13:49:56 |
|  sakura changed their display name from divine intellect to sakura. | 17:20:51 |
 Electria | Is there some option to change the log-format systemwide? Like an alternative to aliasing nix to nix --log-format multiline-with-logs
Seems like there should be a better way to do this, however I haven't been able to find much
| 20:17:30 |
 rosssmyth | sounds like something you could shove in nix.conf but I've never tried | 20:25:38 |
 kfiz | Is this something that can/should be ported to lix? https://github.com/NixOS/nix/pull/13301 | 23:28:55 |
