In reply to @lillecarl:matrix.org
https://asciinema.org/a/I6RXVGmCyCcHnXInvh5DKph4C
read-write nix store in kubernetes with overlayfs 😄 yayy

In reply to @claesatwork:matrix.org
It would be interesting to see the data you pipe into kubectl apply in the first command, it seems to be the magic sauce :-)

https://gist.github.com/Lillecarl/cbd2e037f1fba7c145a37c3f33d5f926

Essentially everything except the daemonset,csidriver and the ubuntu pod is reduntant(in the current state). Currently the volume is just "ephemeral" so the expression metadata is straight in the podspec under volumeAttributes rather than through the expression CRD.

The reason for having a CRD to stick nix expressions into is so that they can be built ahead of pod-creation time, but I've reprioritized to just make sure it works reliably building on pod creation. It'll still be "kewl enough" to hopefully attract more eyes and ideas 😄

Claes: The entire CSI driver is in this file: https://github.com/Lillecarl/cknix/blob/main/cknix_csi/cknix.py

I've cleaned it up now and honestly it's so dumb and simple now. The "coolest" part is probably the execline script (Don't wanna ship a full shell) that pipes nix-store --dump-db to nix-store --load-db 😸
https://github.com/Lillecarl/cknix/blob/main/flake.nix#L63

The nix code quality is still garbage so don't judge too hard

In reply to @claesatwork:matrix.org
Cool!

In the example you exec into the pod to show that the nix store is present. Do you have a plan for how it will work for a pod to execute the right process at start?

I assume all pods running in a cluster will all have the same nix store mounted so the loose end here is to distinguish which pods run what?

Nix makes the "top derivation" available under /nix/var/result/bin/binaryname in the container, which can be set as the command in podspec to execute straight from the volume. Volumes are mounted before the container is created so it works out (You should use "tini" as the entrypoint which then executes your code to reap zombie processes, same thing with any container really).

How the stores work: There's a daemonset (run once per node) that is this CSI driver, it mounts HOST/var/lib/cknix/nix to /nix2 in the daemonset initcontainer, copies /nix to /nix2. Then the actual CSI container runs and mounts HOST/var/lib/cknix/nix to /nix. This store is is the "source store" for all pods on that NODE. But to create an isolated "store view" per pod so they can't see other pods storepaths we create "fake stores" using hardlinks only containing the paths from nix path-info into the "fake store" the "fake store" is then either bind mounted (shares page cache and is amazing but readonly) into the pod, or overlayfs mounted (means you can run nix commands within the workload pod and RW the entire Nix store :)

RW mounting should only be in development, but it's cool because it allows you to edit any storepaths (since the workload pods aren't privileged we can't setup a nix-daemon and do the RO bind mount thing on /nix/store so the entire store is read-write so you can do shenanigans you've never before imagined without strangling your dog

In reply to @lillecarl:matrix.org
The part where we hardlink from the source store to a new "fake store" is where it's unique, just sharing a global /nix with all pods on the host would work, but definitely not for "production" since any application in the cluster could read any other application in the clusters code by travesting the store.

ok this part I misunderstood first

will need more time to look through but definitely innovative!

In reply to @lillecarl:matrix.org
markuskowa: I notice you haven't scheduled a next meetup ;)