!VhbWwlUdjHkamKnfrK:nixos.org

Nix: Cloud Native

275 Members
61 Servers

Load older messages

SenderMessageTime
2 Nov 2025
@lillecarl:matrix.orglillecarl

Easiest way actually is probably editing default.nix

            nix-csi = {
              namespace = "nix-csi";
              image = imageRef; # comment this out
              cache.storageClassName = "some nice CSI";
              authorizedKeys = [
                "YOUR KEY HERE :)"
              ];
              ctest = {
                enable = true;
                replicas = 1;
              };
            };

Then running

nix run --file . kubenixEval.deploymentScript -- --prune --yes
21:41:39
@lillecarl:matrix.orglillecarlThe CSI part is "done", lately I've been working on strapping some more things to it. The cache node sets up a loadbalancer and exposes itself over port 22, it has a populated /etc/nix/machines and with some SSH config on your client you can use the Kubernetes cluster as your own remote builder farm 😄21:43:17
@lillecarl:matrix.orglillecarlThere's more work to be done on the lifetime of paths in the cache, Nix SUCKS at garbage collection. There's only gcroots or go fuck yourself so I've patched it a bit so whenever the daemon does a path query it'll refresh "registraitonTime" in the nix database, then we use that to garbage collect instead of just killing everything that doesn't have a gcroot21:45:08
@lillecarl:matrix.orglillecarlIt's a bit hacky atm, I'm looking to improve it significantly 😄 (Code quality wise)21:45:45
@lillecarl:matrix.orglillecarl* It's a bit hacky atm, I'm looking to improve it significantly 😄 (Code quality wise, and only doing the registrationTime update on certain Nix operations)21:46:15
@erikeah:matrix.orgErikwow21:47:10
@erikeah:matrix.orgErikI wasnt expecting to patch nix21:47:29
@erikeah:matrix.orgErikhahahahaha21:47:30
@lillecarl:matrix.orglillecarlYou don't have to, it's the one supplied in the nix-csi container 😄21:48:03
@lillecarl:matrix.orglillecarlhttps://github.com/Lillecarl/lix/commit/b5e5a56b93da57239feb192416291af21df8dfe8 <- This is the patch, it's really quite simple: Whenever anything queries a path update registrationTime. And to clarify: You don't have to patch your own client, this is only relevant within the nodes and they speak the same nix-daemon protocol as anyone 21:49:20
@lillecarl:matrix.orglillecarlAnd I apply the patch as an overlay on pkgs.lix so it's not stalled either. It will probably break in 2.94 when they're ripping ca-derivations out of Lix (but it's just that they're changing nearby code, nothing "actualy codey" that's relevant 😄21:51:16
@erikeah:matrix.orgErikNo problem with that21:52:44
@lillecarl:matrix.orglillecarl* And I apply the patch as an overlay on pkgs.lix so it's not stalled either. It will probably break in 2.94 when they're ripping ca-derivations out of Lix (but it's just that they're changing nearby code, nothing "actualy codey" that's relevant 😄 Edit: You could ofc not patch Lix/Nix too, it's just that there's no good way to track when a path was last "hot" in the cache without something like this. I've talked to Rickard @nixbuild.net and they're doing the same thing but.... Less hacky 😄21:52:49
@erikeah:matrix.orgErikGreat work21:53:12
@lillecarl:matrix.orglillecarlThanks :)21:53:16
@lillecarl:matrix.orglillecarlI've put a lot of effort into it tbh 😄21:53:35
@erikeah:matrix.orgErikSince nix, i do not fear patches any more21:53:40
@lillecarl:matrix.orglillecarl *

Easiest way actually is probably editing default.nix

            nix-csi = {
              namespace = "nix-csi";
              image = imageRef; # comment this out
              cache.storageClassName = "some nice CSI";
              authorizedKeys = [
                "YOUR KEY HERE :)"
              ];
              ctest = {
                enable = false;
                replicas = 0;
              };
            };

Then running

nix run --file . kubenixEval.deploymentScript -- --prune --yes
21:54:55
@erikeah:matrix.orgErikI will give try and if get stuck i will get to you21:55:06
@lillecarl:matrix.orglillecarlI'm up for another 3-4 hours at least (CET)21:56:18
@erikeah:matrix.orgErikToday I will not be able, where I live is midnight22:00:44
@lillecarl:matrix.orglillecarl *

Easiest way actually is probably editing default.nix

            nix-csi = {
              namespace = "nix-csi";
              image = imageRef; # comment this out
              cache.storageClassName = "some nice CSI";
              authorizedKeys = [
                "YOUR KEY HERE :) (Only required if you wanna push to the cluster cache)"
              ];
              ctest = {
                enable = false;
                replicas = 0;
              };
            };

Then running

nix run --file . kubenixEval.deploymentScript -- --prune --yes
22:00:48
@lillecarl:matrix.orglillecarlNp! There's always another day, nuclear Armageddon is not on my agenda22:01:40
@erikeah:matrix.orgErikBuajajajajjaja22:02:07
@erikeah:matrix.orgErik @lillecarl:matrix.org: thanks for the hints! 22:04:24
3 Nov 2025
@jennydaman:fedora.im@jennydaman:fedora.im left the room.18:53:49
4 Nov 2025
@cafkafk:gitter.imcafkafk changed their profile picture.08:23:35
7 Nov 2025
@mahmoudk1000:matrix.orgMahmoud joined the room.19:06:56
@emma:rory.gay@emma:rory.gay left the room.22:41:28
9 Nov 2025
@lillecarl:matrix.orglillecarl

Erik: I reworked where patched Lix updates paths and made it update recursively so now it sucks less ™️

https://github.com/Lillecarl/lix/commit/9ac72bbd0c7802ca83a907d1fec135f31aab6d24

I'm gonna add a script on a timer to the CSI pods that runs nix path-info --all | nix copy --to nix-cache. This way the cache will always hold any storePaths alive in the cluster while not requiring any complex bookkeeping (nix copy acts as a cache refresher) 😄

It's shaping up pretty darn nice, especially the feature of using the CSI pods as a build farm

22:02:01

Show newer messages

Back to Room ListRoom Version: 6