!VhbWwlUdjHkamKnfrK:nixos.org

Nix: Cloud Native

273 Members
65 Servers

Load older messages


SenderMessageTime
2 Nov 2025
@erikeah:matrix.orgErikis really powerful21:25:32
@lillecarl:matrix.orglillecarlThere's integration with "easykubenix" too that makes it very easy to build manifests and push them to a cache 😄21:25:35
@lillecarl:matrix.orglillecarlWell, easykubenix just makes it easy to stringify manifests and has a "preDeployScript" that you can use to push the manifest you've built to a cache, and since the derivation in the volumeAttrs is a dependency of the manifest it's pushed21:26:19
@lillecarl:matrix.orglillecarlThe projects are still "rough around the edges" but it's all working, I'm using it on a greenfield on-prem Kubernetes thingy I'm building for a client21:26:59
@lillecarl:matrix.orglillecarl

But something like this would ofc work too

export STOREPATH=$(nix build --no-link --print-out-paths nixpkgs#hello)
nix copy $STOREPATH --to $somewhere
# run your YAML mangling bogus here
21:28:36
@erikeah:matrix.orgErikObviously nix is the perfect glue21:29:40
@erikeah:matrix.orgErikas always21:29:49
@lillecarl:matrix.orglillecarlIn reality nix-csi is essentially "nix copy --to /a/cool/path && mount --bind /a/cool/path $targetpath" but instead of "nix copy" we rsync and initialize the DB separately, which makes it 0 storage overhead and sharing inodes which is cool if you wanna run 999999 of the same pod on the same machine 😄21:30:42
@lillecarl:matrix.orglillecarlHaha yeah if Nix can't do it it can generate a script that does it 21:31:08
@lillecarl:matrix.orglillecarlThe only way to generate the nix-csi manifests currently is with easykubenix, it's poorly documented so when you're getting to please hit me up 😄21:33:21
@lillecarl:matrix.orglillecarlThe default.nix tries to build the CSI container image locally and ingest it into your containerd. You can ofc use easykubenix and just import the modules from $nix-csi/kubenix but I wouldn't expect most to grok that yet21:34:22
@lillecarl:matrix.orglillecarl* The default.nix tries to build the CSI container image locally and ingest it into your containerd. You can ofc use easykubenix and just import the modules from $nix-csi/kubenix but I wouldn't expect most to grok that yet Edit: Borderline gatekeeping / ADHD last 20% issues :P21:34:56
@erikeah:matrix.orgErikDefinitely I will hit you up! 21:40:37
@erikeah:matrix.orgErikWhen it is the building done?21:41:16
@lillecarl:matrix.orglillecarl

Easiest way actually is probably editing default.nix

            nix-csi = {
              namespace = "nix-csi";
              image = imageRef; # comment this out
              cache.storageClassName = "some nice CSI";
              authorizedKeys = [
                "YOUR KEY HERE :)"
              ];
              ctest = {
                enable = true;
                replicas = 1;
              };
            };

Then running

nix run --file . kubenixEval.deploymentScript -- --prune --yes
21:41:39
@lillecarl:matrix.orglillecarlThe CSI part is "done", lately I've been working on strapping some more things to it. The cache node sets up a loadbalancer and exposes itself over port 22, it has a populated /etc/nix/machines and with some SSH config on your client you can use the Kubernetes cluster as your own remote builder farm 😄21:43:17
@lillecarl:matrix.orglillecarlThere's more work to be done on the lifetime of paths in the cache, Nix SUCKS at garbage collection. There's only gcroots or go fuck yourself so I've patched it a bit so whenever the daemon does a path query it'll refresh "registraitonTime" in the nix database, then we use that to garbage collect instead of just killing everything that doesn't have a gcroot21:45:08
@lillecarl:matrix.orglillecarlIt's a bit hacky atm, I'm looking to improve it significantly 😄 (Code quality wise)21:45:45
@lillecarl:matrix.orglillecarl* It's a bit hacky atm, I'm looking to improve it significantly 😄 (Code quality wise, and only doing the registrationTime update on certain Nix operations)21:46:15
@erikeah:matrix.orgErikwow21:47:10
@erikeah:matrix.orgErikI wasnt expecting to patch nix21:47:29
@erikeah:matrix.orgErikhahahahaha21:47:30
@lillecarl:matrix.orglillecarlYou don't have to, it's the one supplied in the nix-csi container 😄21:48:03
@lillecarl:matrix.orglillecarlhttps://github.com/Lillecarl/lix/commit/b5e5a56b93da57239feb192416291af21df8dfe8 <- This is the patch, it's really quite simple: Whenever anything queries a path update registrationTime. And to clarify: You don't have to patch your own client, this is only relevant within the nodes and they speak the same nix-daemon protocol as anyone 21:49:20
@lillecarl:matrix.orglillecarlAnd I apply the patch as an overlay on pkgs.lix so it's not stalled either. It will probably break in 2.94 when they're ripping ca-derivations out of Lix (but it's just that they're changing nearby code, nothing "actualy codey" that's relevant 😄21:51:16
@erikeah:matrix.orgErikNo problem with that21:52:44
@lillecarl:matrix.orglillecarl* And I apply the patch as an overlay on pkgs.lix so it's not stalled either. It will probably break in 2.94 when they're ripping ca-derivations out of Lix (but it's just that they're changing nearby code, nothing "actualy codey" that's relevant 😄 Edit: You could ofc not patch Lix/Nix too, it's just that there's no good way to track when a path was last "hot" in the cache without something like this. I've talked to Rickard @nixbuild.net and they're doing the same thing but.... Less hacky 😄21:52:49
@erikeah:matrix.orgErikGreat work21:53:12
@lillecarl:matrix.orglillecarlThanks :)21:53:16
@lillecarl:matrix.orglillecarlI've put a lot of effort into it tbh 😄21:53:35

Show newer messages


Back to Room ListRoom Version: 6