!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

244 Members
Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.84 Servers

Load older messages


SenderMessageTime
23 Mar 2024
@olafkfreund:matrix.orgOlaf Krasicki-Freund joined the room.07:47:38
25 Mar 2024
@vcunat:matrix.orgvcunat

Today we don't have enough RAM to evaluate release-23.11

[root@rhea:~]# journalctl -u hydra-evaluator.service -S today | grep oom-kill
Mar 25 10:20:14 rhea systemd[1]: hydra-evaluator.service: Failed with result 'oom-kill'.
Mar 25 11:16:03 rhea systemd[1]: hydra-evaluator.service: Failed with result 'oom-kill'.
11:32:20
@vcunat:matrix.orgvcunatWe only have 100G of zramctl.11:33:10
@vcunat:matrix.orgvcunat * We "only" have 100G of zramctl.11:33:16
@vcunat:matrix.orgvcunatI can't see anything else taking lots of RAM there.11:41:39
@vcunat:matrix.orgvcunatSo I expect is eval needing more; maybe we've been close to limit for longer time already.11:42:39
@vcunat:matrix.orgvcunat * So I expect it's eval needing more; maybe we've been close to limit for longer time already.11:42:46
@rick:matrix.ciphernetics.nlMindaviI've been looking at lots of RAM usage on my hydra instance, which is relatively small. So I think something happened or has been happening where memory usage of nix is going up.16:13:17
26 Mar 2024
@k900:0upti.meK900 ⚡️Booped unstable-small09:51:37
@k900:0upti.meK900 ⚡️It had two evals queued up09:51:44
@hexa:lossy.networkhexa (signing key rotation when)I merged firefox 124.0.1 into release-23.11 on 2024-03-22 and it has still not appeared in nixos-23.1118:19:31
@hexa:lossy.networkhexa (signing key rotation when)eval interval was changed to 1.5 days18:19:58
@hexa:lossy.networkhexa (signing key rotation when)last channel bump was 4 days ago18:20:14
@hexa:lossy.networkhexa (signing key rotation when)I have people contacting me, that they're not allowed to use 124.0 due to the security issues in it 🫠18:21:01
@hexa:lossy.networkhexa (signing key rotation when) * I have people contacting me, that they're not allowed to use 124.0 anymore due to the security issues in it 🫠18:21:10
@hexa:lossy.networkhexa (signing key rotation when)the eval from 3 days ago still had 124.0 … wat18:21:53
@vcunat:matrix.orgvcunatIt's the RAM issues I posted.18:22:17
@vcunat:matrix.orgvcunatMost eval attempts result into OOM.18:22:36
@hexa:lossy.networkhexa (signing key rotation when)when 3 days is actually 3 days 23 hours and 20 minutes18:22:39
@k900:0upti.meK900 ⚡️It did have a successful eval18:23:22
@k900:0upti.meK900 ⚡️Just now18:23:35
@hexa:lossy.networkhexa (signing key rotation when)1h ago18:23:45
@hexa:lossy.networkhexa (signing key rotation when)I'm bumping the builds18:23:47
@hexa:lossy.networkhexa (signing key rotation when)

8958 builds have been bumped to the front of the queue.

18:23:51
@hexa:lossy.networkhexa (signing key rotation when)context: https://www.mozilla.org/en-US/security/advisories/mfsa2024-15/18:24:13
@hexa:lossy.networkhexa (signing key rotation when)image.png
Download image.png
18:28:01
@hexa:lossy.networkhexa (signing key rotation when)could also be a process not freeing up allocatec memory18:29:23
@hexa:lossy.networkhexa (signing key rotation when) * could also be a process not freeing up allocatecdmemory18:30:26
@hexa:lossy.networkhexa (signing key rotation when) * could also be a process not freeing up allocated memory18:30:29
@k900:0upti.meK900 ⚡️I've also bumped unstable-small because we had eval breakage and I want to see the tarball job now20:22:47

There are no newer messages yet.


Back to Room ListRoom Version: 6