23 Mar 2024 |
| Olaf Krasicki-Freund joined the room. | 07:47:38 |
25 Mar 2024 |
vcunat | Today we don't have enough RAM to evaluate release-23.11
[root@rhea:~]# journalctl -u hydra-evaluator.service -S today | grep oom-kill
Mar 25 10:20:14 rhea systemd[1]: hydra-evaluator.service: Failed with result 'oom-kill'.
Mar 25 11:16:03 rhea systemd[1]: hydra-evaluator.service: Failed with result 'oom-kill'.
| 11:32:20 |
vcunat | We only have 100G of zramctl. | 11:33:10 |
vcunat | * We "only" have 100G of zramctl. | 11:33:16 |
vcunat | I can't see anything else taking lots of RAM there. | 11:41:39 |
vcunat | So I expect is eval needing more; maybe we've been close to limit for longer time already. | 11:42:39 |
vcunat | * So I expect it's eval needing more; maybe we've been close to limit for longer time already. | 11:42:46 |
Mindavi | I've been looking at lots of RAM usage on my hydra instance, which is relatively small. So I think something happened or has been happening where memory usage of nix is going up. | 16:13:17 |
26 Mar 2024 |
K900 ⚡️ | Booped unstable-small | 09:51:37 |
K900 ⚡️ | It had two evals queued up | 09:51:44 |
hexa (signing key rotation when) | I merged firefox 124.0.1 into release-23.11 on 2024-03-22 and it has still not appeared in nixos-23.11 | 18:19:31 |
hexa (signing key rotation when) | eval interval was changed to 1.5 days | 18:19:58 |
hexa (signing key rotation when) | last channel bump was 4 days ago | 18:20:14 |
hexa (signing key rotation when) | I have people contacting me, that they're not allowed to use 124.0 due to the security issues in it 🫠 | 18:21:01 |
hexa (signing key rotation when) | * I have people contacting me, that they're not allowed to use 124.0 anymore due to the security issues in it 🫠 | 18:21:10 |
hexa (signing key rotation when) | the eval from 3 days ago still had 124.0 … wat | 18:21:53 |
vcunat | It's the RAM issues I posted. | 18:22:17 |
vcunat | Most eval attempts result into OOM. | 18:22:36 |
hexa (signing key rotation when) | when 3 days is actually 3 days 23 hours and 20 minutes | 18:22:39 |
K900 ⚡️ | It did have a successful eval | 18:23:22 |
K900 ⚡️ | Just now | 18:23:35 |
hexa (signing key rotation when) | 1h ago | 18:23:45 |
hexa (signing key rotation when) | I'm bumping the builds | 18:23:47 |
hexa (signing key rotation when) |
8958 builds have been bumped to the front of the queue.
| 18:23:51 |
hexa (signing key rotation when) | context: https://www.mozilla.org/en-US/security/advisories/mfsa2024-15/ | 18:24:13 |
hexa (signing key rotation when) | Download image.png | 18:28:01 |
hexa (signing key rotation when) | could also be a process not freeing up allocatec memory | 18:29:23 |
hexa (signing key rotation when) | * could also be a process not freeing up allocatecdmemory | 18:30:26 |
hexa (signing key rotation when) | * could also be a process not freeing up allocated memory | 18:30:29 |
K900 ⚡️ | I've also bumped unstable-small because we had eval breakage and I want to see the tarball job now | 20:22:47 |