!coeAONBrWyDJnYMbMi:nixos.org

NixOS System Operations

601 Members
About system administration for running NixOS systems in production. Declaratively manage your operations. | Room recommendations: #networking:nixos.org164 Servers

Load older messages


SenderMessageTime
6 Apr 2026
@elisaado:elisaado.comEli Saado changed their profile picture.11:04:08
@elisaado:elisaado.comEli Saado changed their profile picture.11:05:02
7 Apr 2026
@oleg20082009:matrix.org@oleg20082009:matrix.org joined the room.21:17:34
@oleg20082009:matrix.org@oleg20082009:matrix.org left the room.21:38:35
8 Apr 2026
@johnhamelink:matrix.orgjohnhamelink joined the room.07:31:59
@johnhamelink:matrix.orgjohnhamelink Hey folks, I wrote a nix flake for https://crowci.dev/ (a woodpeckerCI fork). My flake uses podman quadlets using quadlet-nix. I'm coming across an issue when building that I'm hoping someone might be able to shed light on: when an agent (runner) container runs nix build, it seems to be able to surpass resource restrictions set in the quadlet configuration (using PodmanArgs). The result is that long builds get OOM killed. What I really want is for the container to be constrained to its resource requirements. My /etc/containers/systemd container configuration (generated by quadlet-nix) looks like this: https://gist.github.com/johnhamelink/80995130d2afc1cedee31b501cb3e689 07:51:29
@johnhamelink:matrix.orgjohnhamelink *

Hey folks, I wrote a nix flake for https://crowci.dev/ (a woodpeckerCI fork). My flake uses podman quadlets using quadlet-nix. I'm coming across an issue when building that I'm hoping someone might be able to shed light on: when an agent (runner) container runs nix build, it seems to be able to surpass resource restrictions set in the quadlet configuration (using PodmanArgs). The result is that long builds get OOM killed. What I really want is for the container to be constrained to its resource requirements. My /etc/containers/systemd container configuration (generated by quadlet-nix) looks like this: https://gist.github.com/johnhamelink/80995130d2afc1cedee31b501cb3e689

My nix flake is here in case you are interested https://codefloe.com/crowci/crowci-flake

07:52:40
@johnhamelink:matrix.orgjohnhamelinkWith the above container configuration, you can see here that the nix process run by conmon bursts right past 2G of memory:08:50:29
@johnhamelink:matrix.orgjohnhamelinkscreenshot-20260408-09:48:09.png
Download screenshot-20260408-09:48:09.png
08:50:32
@johnhamelink:matrix.orgjohnhamelink* With the above container configuration, you can see here that the nix process run by conmon bursts right past 2G of memory (PID 133965):08:51:09
@johnhamelink:matrix.orgjohnhamelinkscreenshot-20260408-09:51:29.png
Download screenshot-20260408-09:51:29.png
08:52:01
@johnhamelink:matrix.orgjohnhamelinkMeanwhile podman stats shows only 17-19MB of memory usage08:52:12
@johnhamelink:matrix.orgjohnhamelinkOK! I figured it out: The agent container uses the docker.socket to spin up its own containers - which is why the nix build process isn't a direct child of the container - and that container wasn't receiving the resource limitation. The authors thought ahead and added configuration for this, which when applied kills the container when it reaches the limit. Now I just need to figure out how to throttle the process instead of kill it outright10:35:03
@johnhamelink:matrix.orgjohnhamelink* OK! I figured it out: The agent container uses the docker.socket to spin up its own containers - which is why the nix build process isn't a direct child of the container - and that container wasn't receiving the resource limitation. The authors thought ahead and added configuration for this, which when applied kills the container when it reaches the limit. Now I just need to figure out how to throttle the spawned container instead of kill it outright10:35:27
@johnhamelink:matrix.orgjohnhamelink I was able to resolve the ram problem with zramSwap.enable = true; Problem solved :) 12:12:08
@jaredmontoya:matrix.orgjaredmontoyaDoes anyone know what to do if promtail is gone? My use case includes using promtail to scrape journald on a 1GB ram raspberry pi. promtail used 23-32MB of RAM but the supposed alternatives (both grafana alloy and fluent-bit) use more than 600MB of RAM12:14:25
@jaredmontoya:matrix.orgjaredmontoyaand I can't give up 60% of my raspberry pi ram just to send it's logs to loki12:14:54
@goeranh:matrix.orggoeranhmaybe just https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-remote.html, or rsyslog?12:19:11
@sandro:supersandro.deSandro 🐧I wanted to look into victoria metrics because it is supposed to be fast and memory efficient 13:55:45
@sandro:supersandro.deSandro 🐧so opposite of opensearch13:55:54
@apernaah:matrix.orgaparna changed their profile picture.14:20:59
@tomasharkema:matrix.orgteumaauss joined the room.16:19:45
@noradtux:tnxip.denoradtux uhh, victria logs .. interesting 16:57:42
@noradtux:tnxip.denoradtux uhh, victoria logs .. interesting 16:57:57
@magic_rb:matrix.redalder.orgmagic_rbIm relatively happy with postgres, it get very big but thats primarily cause i havent configured any resampling17:57:32
@noradtux:tnxip.denoradtux I currently use graylog to collect logs from .. everything. But that is sooooo ressource heavy 17:59:37
@noradtux:tnxip.denoradtux I currently use graylog to collect syslog from .. everything. But that is sooooo ressource heavy 17:59:56
@magic_rb:matrix.redalder.orgmagic_rbTelegraf + postgres here, works okay18:01:44
@magic_rb:matrix.redalder.orgmagic_rbI dont notice it running. But i also have 2 cpus and 64gb of memory18:02:01
@magic_rb:matrix.redalder.orgmagic_rbI can check the memory use when i get to my laptop later today18:02:36

There are no newer messages yet.


Back to Room ListRoom Version: 10