!VhbWwlUdjHkamKnfrK:nixos.org

Nix: Cloud Native

248 Members
56 Servers

Load older messages


SenderMessageTime
2 Nov 2025
@lillecarl:matrix.orglillecarlThe thing with running on the CSI layer is that you must still specify a container image (quay.io/nix-csi/scratch:1.0.1 which sets PATH to /nix/var/result) and you get the zero copy /nix from nix-csi 😄 So it's a bit different as you specify the "image" through volumes rather than image, but it's what you've got to do to operate on the CSI layer 😄21:20:55
@lillecarl:matrix.orglillecarlThere's nix-snapshotter which does the same thing on the CRI level but then you need their containerd and NixOS on the Kubernetes host, nix-csi can just be deployed anywhere Kubernetes runs any CRI which doesn't create VMs work 😄21:21:54
@erikeah:matrix.orgErikreally interesting21:23:10
@erikeah:matrix.orgErikI will give a try for sure21:23:40
@lillecarl:matrix.orglillecarlhttps://gist.github.com/Lillecarl/3104898680ff9bd6128b6f4ddc2072e4 <- example podspec21:24:45
@erikeah:matrix.orgErikYeaaaah, i have stalikng your repo already21:25:14
@erikeah:matrix.orgEriki got the main idea21:25:20
@erikeah:matrix.orgErikis really powerful21:25:32
@lillecarl:matrix.orglillecarlThere's integration with "easykubenix" too that makes it very easy to build manifests and push them to a cache 😄21:25:35
@lillecarl:matrix.orglillecarlWell, easykubenix just makes it easy to stringify manifests and has a "preDeployScript" that you can use to push the manifest you've built to a cache, and since the derivation in the volumeAttrs is a dependency of the manifest it's pushed21:26:19
@lillecarl:matrix.orglillecarlThe projects are still "rough around the edges" but it's all working, I'm using it on a greenfield on-prem Kubernetes thingy I'm building for a client21:26:59
@lillecarl:matrix.orglillecarl

But something like this would ofc work too

export STOREPATH=$(nix build --no-link --print-out-paths nixpkgs#hello)
nix copy $STOREPATH --to $somewhere
# run your YAML mangling bogus here
21:28:36
@erikeah:matrix.orgErikObviously nix is the perfect glue21:29:40
@erikeah:matrix.orgErikas always21:29:49
@lillecarl:matrix.orglillecarlIn reality nix-csi is essentially "nix copy --to /a/cool/path && mount --bind /a/cool/path $targetpath" but instead of "nix copy" we rsync and initialize the DB separately, which makes it 0 storage overhead and sharing inodes which is cool if you wanna run 999999 of the same pod on the same machine 😄21:30:42
@lillecarl:matrix.orglillecarlHaha yeah if Nix can't do it it can generate a script that does it 21:31:08
@lillecarl:matrix.orglillecarlThe only way to generate the nix-csi manifests currently is with easykubenix, it's poorly documented so when you're getting to please hit me up 😄21:33:21
@lillecarl:matrix.orglillecarlThe default.nix tries to build the CSI container image locally and ingest it into your containerd. You can ofc use easykubenix and just import the modules from $nix-csi/kubenix but I wouldn't expect most to grok that yet21:34:22
@lillecarl:matrix.orglillecarl* The default.nix tries to build the CSI container image locally and ingest it into your containerd. You can ofc use easykubenix and just import the modules from $nix-csi/kubenix but I wouldn't expect most to grok that yet Edit: Borderline gatekeeping / ADHD last 20% issues :P21:34:56
@erikeah:matrix.orgErikDefinitely I will hit you up! 21:40:37
@erikeah:matrix.orgErikWhen it is the building done?21:41:16
@lillecarl:matrix.orglillecarl

Easiest way actually is probably editing default.nix

            nix-csi = {
              namespace = "nix-csi";
              image = imageRef; # comment this out
              cache.storageClassName = "some nice CSI";
              authorizedKeys = [
                "YOUR KEY HERE :)"
              ];
              ctest = {
                enable = true;
                replicas = 1;
              };
            };

Then running

nix run --file . kubenixEval.deploymentScript -- --prune --yes
21:41:39
@lillecarl:matrix.orglillecarlThe CSI part is "done", lately I've been working on strapping some more things to it. The cache node sets up a loadbalancer and exposes itself over port 22, it has a populated /etc/nix/machines and with some SSH config on your client you can use the Kubernetes cluster as your own remote builder farm 😄21:43:17
@lillecarl:matrix.orglillecarlThere's more work to be done on the lifetime of paths in the cache, Nix SUCKS at garbage collection. There's only gcroots or go fuck yourself so I've patched it a bit so whenever the daemon does a path query it'll refresh "registraitonTime" in the nix database, then we use that to garbage collect instead of just killing everything that doesn't have a gcroot21:45:08
@lillecarl:matrix.orglillecarlIt's a bit hacky atm, I'm looking to improve it significantly 😄 (Code quality wise)21:45:45
@lillecarl:matrix.orglillecarl* It's a bit hacky atm, I'm looking to improve it significantly 😄 (Code quality wise, and only doing the registrationTime update on certain Nix operations)21:46:15
@erikeah:matrix.orgErikwow21:47:10
@erikeah:matrix.orgErikI wasnt expecting to patch nix21:47:29
@erikeah:matrix.orgErikhahahahaha21:47:30
@lillecarl:matrix.orglillecarlYou don't have to, it's the one supplied in the nix-csi container 😄21:48:03

Show newer messages


Back to Room ListRoom Version: 6