| 19 Mar 2026 |
 Mic92 | how did you came up with this number, so I can monitor? | 12:49:53 |
 hexa (signing key rotation when) | journalctl -u hydra-queue-runner --since "2 days ago" --grep="unable to upload" | wc -l | 12:50:38 |
| Mic92 | okay, we don't have any upload error since: Thu Mar 19 01:01:45 PM UTC 2026 | 13:02:00 |
| Mic92 | Let's see | 13:02:05 |
| Mic92 | journalctl -u hydra-queue-runner --since "3 hour ago" --grep="unable to upload" is also empty | 13:02:13 |
| Mic92 | while true; do journalctl -u hydra-queue-runner --since "2026-03-19 13:01:45" --grep="unable to upload"; sleep 180; done | 13:06:19 |
| Mic92 | * while true; do journalctl -u hydra-queue-runner --since "2026-03-19 13:01:45" --grep="unable to upload"; sleep 180; done for monitoring | 13:06:29 |
| Mic92 | so far only getting a bunch of 500er:
Mar 19 13:11:55 mimas hydra-queue-runner[2800047]: warning: unable to upload 'https://nix-cache.s3.us-east-1.amazonaws.com/nar/0a8jn6fw74pdnpa10viam4wb4jmpszkfw19zrrm1h1n40zlpnlcx.nar.xz': HTTP error 503
Mar 19 13:11:55 mimas hydra-queue-runner[2800047]: warning: unable to upload 'https://nix-cache.s3.us-east-1.amazonaws.com/nar/0a8jn6fw74pdnpa10viam4wb4jmpszkfw19zrrm1h1n40zlpnlcx.nar.xz': HTTP error 503
| 13:16:20 |
| Mic92 | * so far only getting a bunch of one 500er:
Mar 19 13:11:55 mimas hydra-queue-runner[2800047]: warning: unable to upload 'https://nix-cache.s3.us-east-1.amazonaws.com/nar/0a8jn6fw74pdnpa10viam4wb4jmpszkfw19zrrm1h1n40zlpnlcx.nar.xz': HTTP error 503 | 13:16:40 |
| Mic92 | Still only one error. | 15:24:49 |
| Mic92 | Backport of the fix: https://github.com/NixOS/nix/pull/15525 | 15:58:59 |
| Mic92 | hexa (signing key rotation when): vcunat Jeremy Fleischman (jfly) dgrig https://meet.cccda.de/nix-osin-fra starts in 5min according to my calendar Arian already implied he can't join today. | 16:56:00 |
| hexa (signing key rotation when) | huh, arian implied he could demo the mdm stuff earlier today | 16:57:43 |
 Arian | Just wrote in the infra team chat. I have a monthly team meeting that overlaps :( | 16:59:08 |
| hexa (signing key rotation when) | ok | 16:59:17 |
|  Miles Dyson set a profile picture. | 18:05:54 |
| 20 Mar 2026 |
|  irrelevancy changed their display name from irrelevancyisthygoal One Confused Being to irrelevancy. | 06:46:39 |
| 21 Mar 2026 |
 vcunat | maximum-snail.mac.nixos.org was stuck, all job slots in "sending inputs" for several hours. Rebooted now, seems to be building again. | 05:55:09 |
| hexa (signing key rotation when) | that means it ran out of disk space | 09:23:41 |
| vcunat | As it tends to happen recently, what's the best action when I see it again? (assuming we can't easily avoid it) | 09:30:21 |
| hexa (signing key rotation when) | reboot the builder or restart the queue-runner | 09:34:32 |
| hexa (signing key rotation when) | I chose the former because it is less disruptive | 09:34:40 |
| vcunat | I want to have long uninterrupted queue-runner periods. | 09:35:59 |
| vcunat | Big builds suffer, especially chromium and derivatives on aarch64-linux. The aarch64-linux:big-parallel queue doesn't manage to catch up too well lately. | 09:36:49 |
| vcunat | (a chromium often runs for half a day in there) | 09:37:23 |
| vcunat | * (a chromium build often runs for half a day in there) | 09:37:30 |
| vcunat | * (a single chromium build often runs for half a day in there) | 09:37:48 |
| hexa (signing key rotation when) | yeah, so another machine would be useful | 09:41:20 |
| vcunat | A dedicated machine for all the chromiums and electrons 😆 | 09:49:36 |
|  Mindavi joined the room. | 17:26:41 |
