!coeAONBrWyDJnYMbMi:nixos.org

NixOS System Operations

603 Members
About system administration for running NixOS systems in production. Declaratively manage your operations. | Room recommendations: #networking:nixos.org164 Servers

Load older messages


SenderMessageTime
8 Apr 2026
@johnhamelink:matrix.orgjohnhamelink *

Hey folks, I wrote a nix flake for https://crowci.dev/ (a woodpeckerCI fork). My flake uses podman quadlets using quadlet-nix. I'm coming across an issue when building that I'm hoping someone might be able to shed light on: when an agent (runner) container runs nix build, it seems to be able to surpass resource restrictions set in the quadlet configuration (using PodmanArgs). The result is that long builds get OOM killed. What I really want is for the container to be constrained to its resource requirements. My /etc/containers/systemd container configuration (generated by quadlet-nix) looks like this: https://gist.github.com/johnhamelink/80995130d2afc1cedee31b501cb3e689

My nix flake is here in case you are interested https://codefloe.com/crowci/crowci-flake

07:52:40
@johnhamelink:matrix.orgjohnhamelinkWith the above container configuration, you can see here that the nix process run by conmon bursts right past 2G of memory:08:50:29
@johnhamelink:matrix.orgjohnhamelinkscreenshot-20260408-09:48:09.png
Download screenshot-20260408-09:48:09.png
08:50:32
@johnhamelink:matrix.orgjohnhamelink* With the above container configuration, you can see here that the nix process run by conmon bursts right past 2G of memory (PID 133965):08:51:09
@johnhamelink:matrix.orgjohnhamelinkscreenshot-20260408-09:51:29.png
Download screenshot-20260408-09:51:29.png
08:52:01
@johnhamelink:matrix.orgjohnhamelinkMeanwhile podman stats shows only 17-19MB of memory usage08:52:12
@johnhamelink:matrix.orgjohnhamelinkOK! I figured it out: The agent container uses the docker.socket to spin up its own containers - which is why the nix build process isn't a direct child of the container - and that container wasn't receiving the resource limitation. The authors thought ahead and added configuration for this, which when applied kills the container when it reaches the limit. Now I just need to figure out how to throttle the process instead of kill it outright10:35:03
@johnhamelink:matrix.orgjohnhamelink* OK! I figured it out: The agent container uses the docker.socket to spin up its own containers - which is why the nix build process isn't a direct child of the container - and that container wasn't receiving the resource limitation. The authors thought ahead and added configuration for this, which when applied kills the container when it reaches the limit. Now I just need to figure out how to throttle the spawned container instead of kill it outright10:35:27
@johnhamelink:matrix.orgjohnhamelink I was able to resolve the ram problem with zramSwap.enable = true; Problem solved :) 12:12:08
@jaredmontoya:matrix.orgjaredmontoyaDoes anyone know what to do if promtail is gone? My use case includes using promtail to scrape journald on a 1GB ram raspberry pi. promtail used 23-32MB of RAM but the supposed alternatives (both grafana alloy and fluent-bit) use more than 600MB of RAM12:14:25
@jaredmontoya:matrix.orgjaredmontoyaand I can't give up 60% of my raspberry pi ram just to send it's logs to loki12:14:54
@goeranh:matrix.orggoeranhmaybe just https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-remote.html, or rsyslog?12:19:11
@sandro:supersandro.deSandro 🐧I wanted to look into victoria metrics because it is supposed to be fast and memory efficient 13:55:45
@sandro:supersandro.deSandro 🐧so opposite of opensearch13:55:54
@apernaah:matrix.orgaparna changed their profile picture.14:20:59
@tomasharkema:matrix.orgteumaauss joined the room.16:19:45
@noradtux:tnxip.denoradtux uhh, victria logs .. interesting 16:57:42
@noradtux:tnxip.denoradtux uhh, victoria logs .. interesting 16:57:57
@magic_rb:matrix.redalder.orgmagic_rbIm relatively happy with postgres, it get very big but thats primarily cause i havent configured any resampling17:57:32
@noradtux:tnxip.denoradtux I currently use graylog to collect logs from .. everything. But that is sooooo ressource heavy 17:59:37
@noradtux:tnxip.denoradtux I currently use graylog to collect syslog from .. everything. But that is sooooo ressource heavy 17:59:56
@magic_rb:matrix.redalder.orgmagic_rbTelegraf + postgres here, works okay18:01:44
@magic_rb:matrix.redalder.orgmagic_rbI dont notice it running. But i also have 2 cpus and 64gb of memory18:02:01
@magic_rb:matrix.redalder.orgmagic_rbI can check the memory use when i get to my laptop later today18:02:36
9 Apr 2026
@monokles:matrix.monokles.eumonoklesIf it is just monitoring for a home setup without the need for metrics, you could also just pipe journald entries of interest to something by writing a small systemd service yourself. For example, I push ssh login events on my systems to private e2ee matrix rooms. 10:36:35
@setkeh_tksp:matrix.orgJames (setkeh) Griffis joined the room.10:44:52
@phaer:matrix.orgphaerI just checked because that seemed to be rather high: alloy on my hosts eats between 40 and 65M of RAM. Granted, none of those is suuuper busy with log generation, but the journal of a private rasperry pi shouldn't be either. Maybe it's specific rules or so? I can at least tell you it does not NEED to use that much RAM if you are otherwise happy with one of them15:11:54
@mrappard:matrix.orgMatthew Rappard joined the room.17:51:39
@mrappard:matrix.orgMatthew RappardFeel free to loudly tell me it's a bad. But has anyone got NixOS working on a Google Cloud Workstations https://cloud.google.com/workstations17:52:19
@blades:interestingt.imblades
In reply to @mrappard:matrix.org
Feel free to loudly tell me it's a bad.

But has anyone got NixOS working on a Google Cloud Workstations
https://cloud.google.com/workstations
Not nixos, but our internal equivalent uses a lot of the same tooling and it was trivial to install nix (via determinate installer) and devenv into the the workspaces.
(I would have installed nixos, but the infra team in charge begged me not to add more distros that they’d have to test.)
18:44:41

There are no newer messages yet.


Back to Room ListRoom Version: 10