@kalbasit:matrix.org nix-csi enables you to tag nodes as builders, it's pretty barely tested but it works. It completely bypasses resourc constraints.

It works by using the in-cluster cache pod as SSH jump box into the nodes which can run builds.

It'll be developed further :)

@kalbasit:matrix.org I've been thinking about how to architect the in-kube building. The way it works now is pretty dumb since you can't constrain it properly. My current idea is to use a Deployment for building that mounts the same hostPath (optionally maybe) as the DaemonSet. I'll have to verify that I can make sure builds run in the Deployment instead of talking to the Nix daemon running in the DS. I've also been considering if I should stop running nix daemon altogether since it's effectively a single user system anyways. nix-csi is privileged.

I'd love to hear ideas. Im pretty sure I'll dump "dinix" and run multiple containers like Kube wants you to too. I'm aiming for 1.0 Q1 where all the things not just work, but work "like they should". Building in Kubernetes won't be as good as nixbuild.net but it'll be good enough ™️

@kalbasit:matrix.org My plan is to stay entirely cloud agnostic so AWS is out of the question. I'll have to come up with different solutions for RWO and RWX builds. If you have RWX storage the in-cluster cache and the builders can just share a store but I'll also support non RWX, I have ideas to watch the builder stores and push paths to cache as soon as they're registered in the DB.

I'm already patching Nix(Lix) to update "registrationtime" when a path is referenced which makes the GC in nix-csi pretty awesome (nodes keep their paths for an hour, cache for configurable time).

By eagerly pushing to cache I think performance will be quite good, in-cluster speeds should be good enough to not matter much.

The registrationtime updates are recursive too meaning we keep all deps and build-deps hot always (well once it's implemented :p)

kalbasit: I've hit a limitation to running the builds in Kubernetes: Either you run privileged or you disable the sandbox. I added an option that easily configures these settings in tandem for the builders but it doesn't seem to be reasonable to work around. There's user namespaces but they require node configuration which I've banned from nix-csi (it must be deployable everywhere).

Other than that it's chugging along nicely 😄

kalbasit: I've hit a limitation to running the builds in Kubernetes: Either you run privileged or you disable the sandbox. I added an option that easily configures these settings in tandem for the builders because it doesn't seem to be reasonable to work around. There's user namespaces but they require node configuration which I've banned from nix-csi (it must be deployable everywhere).

Other than that it's chugging along nicely 😄