Nix: Cloud Native | 248 Members | |
| 56 Servers |
| Sender | Message | Time |
|---|---|---|
| 2 Nov 2025 | ||
| The thing with running on the CSI layer is that you must still specify a container image (quay.io/nix-csi/scratch:1.0.1 which sets PATH to /nix/var/result) and you get the zero copy /nix from nix-csi 😄 So it's a bit different as you specify the "image" through volumes rather than image, but it's what you've got to do to operate on the CSI layer 😄 | 21:20:55 | |
| There's nix-snapshotter which does the same thing on the CRI level but then you need their containerd and NixOS on the Kubernetes host, nix-csi can just be deployed anywhere Kubernetes runs any CRI which doesn't create VMs work 😄 | 21:21:54 | |
| really interesting | 21:23:10 | |
| I will give a try for sure | 21:23:40 | |
| https://gist.github.com/Lillecarl/3104898680ff9bd6128b6f4ddc2072e4 <- example podspec | 21:24:45 | |
| Yeaaaah, i have stalikng your repo already | 21:25:14 | |
| i got the main idea | 21:25:20 | |
| is really powerful | 21:25:32 | |
| There's integration with "easykubenix" too that makes it very easy to build manifests and push them to a cache 😄 | 21:25:35 | |
| Well, easykubenix just makes it easy to stringify manifests and has a "preDeployScript" that you can use to push the manifest you've built to a cache, and since the derivation in the volumeAttrs is a dependency of the manifest it's pushed | 21:26:19 | |
| The projects are still "rough around the edges" but it's all working, I'm using it on a greenfield on-prem Kubernetes thingy I'm building for a client | 21:26:59 | |
| But something like this would ofc work too
| 21:28:36 | |
| Obviously nix is the perfect glue | 21:29:40 | |
| as always | 21:29:49 | |
| In reality nix-csi is essentially "nix copy --to /a/cool/path && mount --bind /a/cool/path $targetpath" but instead of "nix copy" we rsync and initialize the DB separately, which makes it 0 storage overhead and sharing inodes which is cool if you wanna run 999999 of the same pod on the same machine 😄 | 21:30:42 | |
| Haha yeah if Nix can't do it it can generate a script that does it | 21:31:08 | |
| The only way to generate the nix-csi manifests currently is with easykubenix, it's poorly documented so when you're getting to please hit me up 😄 | 21:33:21 | |
| The default.nix tries to build the CSI container image locally and ingest it into your containerd. You can ofc use easykubenix and just import the modules from $nix-csi/kubenix but I wouldn't expect most to grok that yet | 21:34:22 | |
| * The default.nix tries to build the CSI container image locally and ingest it into your containerd. You can ofc use easykubenix and just import the modules from $nix-csi/kubenix but I wouldn't expect most to grok that yet Edit: Borderline gatekeeping / ADHD last 20% issues :P | 21:34:56 | |
| Definitely I will hit you up! | 21:40:37 | |
| When it is the building done? | 21:41:16 | |
| Easiest way actually is probably editing default.nix
Then running
| 21:41:39 | |
| The CSI part is "done", lately I've been working on strapping some more things to it. The cache node sets up a loadbalancer and exposes itself over port 22, it has a populated /etc/nix/machines and with some SSH config on your client you can use the Kubernetes cluster as your own remote builder farm 😄 | 21:43:17 | |
| There's more work to be done on the lifetime of paths in the cache, Nix SUCKS at garbage collection. There's only gcroots or go fuck yourself so I've patched it a bit so whenever the daemon does a path query it'll refresh "registraitonTime" in the nix database, then we use that to garbage collect instead of just killing everything that doesn't have a gcroot | 21:45:08 | |
| It's a bit hacky atm, I'm looking to improve it significantly 😄 (Code quality wise) | 21:45:45 | |
| * It's a bit hacky atm, I'm looking to improve it significantly 😄 (Code quality wise, and only doing the registrationTime update on certain Nix operations) | 21:46:15 | |
| wow | 21:47:10 | |
| I wasnt expecting to patch nix | 21:47:29 | |
| hahahahaha | 21:47:30 | |
| You don't have to, it's the one supplied in the nix-csi container 😄 | 21:48:03 | |