| 28 Jul 2025 |
 magic_rb | Id say no firewall works well with them because interoperability at the iptables/nftables level is grossly at the wrong level, but thats another separate rant | 13:37:02 |
 Molly Miller | getting docker to work with scripted networking and iptables is a bit painful but doable | 13:38:20 |
| Molly Miller | (ask me how i know, etc etc) | 13:38:30 |
| magic_rb | I personally gave up, i put k3s (containerd) into a separate network namespace and do the firewalling on the outside | 13:39:17 |
 Marcel | Ifstate is not in the release phase for V2 so I am preparing to upstream the nix module to nixpkgs. The question is if the options should be under networking.ifstate or services.ifstate or somewhere different? That's hot it's currently done in my flake: https://search.nüschtos.de/?scope=IfState.nix | 16:01:33 |
| Marcel | * | 16:01:58 |
| Marcel | * | 16:02:08 |
| Marcel | * | 16:02:33 |
| Marcel | * Ifstate is now in the release candidate phase for V2 so I am preparing to upstream the nix module to nixpkgs. The question is if the options should be under networking.ifstate or services.ifstate or somewhere different? That's how it's currently done in my flake: https://search.nüschtos.de/?scope=IfState.nix
Services.ifstate might not be optimal because it's not a daemon and only runs on boot or rebuild
| 16:03:46 |
 adamcstephens | networking.ifstate seems reasonable to me | 16:06:06 |
 Zhaofeng Li | interesting, what's your setup like? I might do something similar, but for the wrong reasons :p | 16:13:06 |
| Zhaofeng Li | (launch k3s from k8s) | 16:13:19 |
| magic_rb | In reply to @zhaofeng:zhaofeng.li
interesting, what's your setup like? I might do something similar, but for the wrong reasons :p uh, i use systemd-nspawn with some convincing and i wrote a simple k3s module for NixNG | 16:17:21 |
| magic_rb | https://git.sr.ht/~magic_rb/uk3s.nix/tree/master/item/nixos/modules these modules | 16:17:52 |
| magic_rb | https://git.sr.ht/~magic_rb/uk3s.nix/tree/master/item/nixos/modules/uk3s.nix#L341 this specifically is what you need to run k3s in a nspawn container | 16:18:40 |
| magic_rb | or itll complain | 16:18:44 |
| magic_rb | the two env vars are reverse engineered from systemd source code and a lot of trial and error, im still using this setup but im hoping to migrate away, not from nixng+ucontainer but throw out the k3s | 16:19:26 |
| Zhaofeng Li | ok, so k3s and flannel basically just work inside a network namespace, good to know | 16:23:35 |
| Zhaofeng Li | going to attempt kind of the same thing but with cilium (probably not soon tbh) | 16:25:47 |
| magic_rb | In reply to @zhaofeng:zhaofeng.li
ok, so k3s and flannel basically just work inside a network namespace, good to know Im using istio, problem with cilium is that their own test suite is broken, and has been for months, when i tried it, so i couldnt know if it was my problem or their problem when i was debugging it | 16:27:42 |
| magic_rb | So i gave up, went to istio | 16:27:48 |
| magic_rb | But istio is insanely slow, envoy has huge overheads | 16:29:01 |
| magic_rb | I can see envoy burning cpu time when im copying from my nix cache, so im throwing the whole thing out | 16:29:55 |
| Zhaofeng Li | hmm, that doesn't sound too good... basically https://spot.rackspace.com provides cheap compute but their control plane is garbage, so I want to just shove a daemonset up there and run my k3s 🙃 | 16:31:24 |
| magic_rb | Yeah i wouldnt, pain | 16:33:01 |
| magic_rb | What you save on hardware cost youll spend double on your sanity because kubernetes and istio/cilium | 16:33:20 |
| Zhaofeng Li | (with a bit of wireguard and bird magic maybe I can make them join my existing cluster, but yeah, extremely cursedness) | 16:33:29 |
| magic_rb | I went into it enthusiastically trying to make it work, but the whole thing is rotten from the core. | 16:33:42 |
| Zhaofeng Li | * (with a bit of wireguard and bird magic maybe I can make them join my existing cluster, but yeah, extreme cursedness) | 16:34:39 |
| magic_rb | etcd fucking sucks, its slow as hell. The manifests gets huge very quickly, i can never remember all of the obscure options, networking is a mess, if you want to use the new gateway api youll end up reading issues trying to figure what is or isnt supported, eventually reading tests and source code like me. | 16:34:44 |
