 | Nix: Cloud Native | 287 Members |
| 66 Servers |
| Nix: Cloud Native | 287 Members |
| 66 Servers |
| Sender | Message | Time |
|---|---|---|
| 2 Nov 2025 | ||
 lillecarl | * Still looking for Kubernetes users to try out nix-csi! 😄 It's got a in-cluster cache (ssh-ng) now and you can reuse CSI pods as your own build cluster. The cache pod maintains a /etc/nix/machines config you can SCP onto your machine and with some ssh_config you get all builder labeled nodes accessible from nix CLI on your machine: ^ Pretty much like this, the list is always up2date on the cache (watching pod nix-csi-node pod events). Works with aarch64-linux and x86_64-linux so for cross-building it's pretty neat. Still investigating the proper way to trigger cache population within the cluster when doing remote builds | 17:04:42 |
 Zhaofeng Li joined the room. | 17:30:41 | |
 Erik | @lillecarl:matrix.org: first time I have heard of nix-csi, definitively I will give a try. I found it really amazing! | 20:49:02 |
| lillecarl | Erik: It's still ~quite beta~, but I'm happy to provide some hand-holding 😄 | 20:49:58 |
| lillecarl | * Erik: It's still ~quite beta~, but I'm happy to provide some hand-holding 😄 The CSI bit works well, the cache bit works well if you hold it right-ish, hehe. There isn't an option to add your own caches and trust-keys currently so the beaten path is adding your pubkey and pushing to it, or providing expressions in the volumeAttributes | 20:51:34 |
| Erik | And how a container is invoked with nix-csi? | 20:53:53 |
| Erik | What I understand is that nix-csi avoids the necessity to an adapter like nixify to consume nix builds, or maybe i had misunderstood... | 20:56:01 |
| Erik | * What I understand is that nix-csi avoids the necessity of an adapter like nixify to consume nix builds, or maybe i had misunderstood... | 20:56:16 |
| lillecarl | Erik: Yeah, and it's zero copy too. you provide a volume to the pod. On it you put volumeAttributes. Either you put down ${system} = pkgs.whatever; or you put down expression = ''full independent nix expression''; In the first operating mode the storepath must be available on a cache, in the second one it'll build it in-cluster and store it on the built-in cache | 21:18:54 |
| lillecarl | The thing with running on the CSI layer is that you must still specify a container image (quay.io/nix-csi/scratch:1.0.1 which sets PATH to /nix/var/result) and you get the zero copy /nix from nix-csi 😄 So it's a bit different as you specify the "image" through volumes rather than image, but it's what you've got to do to operate on the CSI layer 😄 | 21:20:55 |
| lillecarl | There's nix-snapshotter which does the same thing on the CRI level but then you need their containerd and NixOS on the Kubernetes host, nix-csi can just be deployed anywhere Kubernetes runs any CRI which doesn't create VMs work 😄 | 21:21:54 |
| Erik | really interesting | 21:23:10 |
| Erik | I will give a try for sure | 21:23:40 |
| lillecarl | https://gist.github.com/Lillecarl/3104898680ff9bd6128b6f4ddc2072e4 <- example podspec | 21:24:45 |
| Erik | Yeaaaah, i have stalikng your repo already | 21:25:14 |
| Erik | i got the main idea | 21:25:20 |
| Erik | is really powerful | 21:25:32 |
| lillecarl | There's integration with "easykubenix" too that makes it very easy to build manifests and push them to a cache 😄 | 21:25:35 |
| lillecarl | Well, easykubenix just makes it easy to stringify manifests and has a "preDeployScript" that you can use to push the manifest you've built to a cache, and since the derivation in the volumeAttrs is a dependency of the manifest it's pushed | 21:26:19 |
| lillecarl | The projects are still "rough around the edges" but it's all working, I'm using it on a greenfield on-prem Kubernetes thingy I'm building for a client | 21:26:59 |
| lillecarl | But something like this would ofc work too | 21:28:36 |
| Erik | Obviously nix is the perfect glue | 21:29:40 |
| Erik | as always | 21:29:49 |
| lillecarl | In reality nix-csi is essentially "nix copy --to /a/cool/path && mount --bind /a/cool/path $targetpath" but instead of "nix copy" we rsync and initialize the DB separately, which makes it 0 storage overhead and sharing inodes which is cool if you wanna run 999999 of the same pod on the same machine 😄 | 21:30:42 |
| lillecarl | Haha yeah if Nix can't do it it can generate a script that does it | 21:31:08 |
| lillecarl | The only way to generate the nix-csi manifests currently is with easykubenix, it's poorly documented so when you're getting to please hit me up 😄 | 21:33:21 |
| lillecarl | The default.nix tries to build the CSI container image locally and ingest it into your containerd. You can ofc use easykubenix and just import the modules from $nix-csi/kubenix but I wouldn't expect most to grok that yet | 21:34:22 |
| lillecarl | * The default.nix tries to build the CSI container image locally and ingest it into your containerd. You can ofc use easykubenix and just import the modules from $nix-csi/kubenix but I wouldn't expect most to grok that yet Edit: Borderline gatekeeping / ADHD last 20% issues :P | 21:34:56 |
| Erik | Definitely I will hit you up! | 21:40:37 |
| Erik | When it is the building done? | 21:41:16 |