| 28 May 2025 |
 vcunat | "Steps completed per minute" isn't much lower than usually, I'd say. | 10:07:16 |
 uep | i'm going by https://hydra.nixos.org/jobset/nixos/trunk-combined, and the fact that for the last.. quite a few days, it seems the build from the previous day hasn't finished by the time the next one starts. And even with relatively low numbers of jobs to run (8k, vs 120k) it's taking more than a day to finish.
I realise this isn't the total picture, i'm not looking at overall throughput or jobs | 10:11:42 |
 hexa (signing key rotation when) | this relates to the "build ingestion" in that we don't currently get enough runnables to saturate our linux builders | 10:14:30 |
| hexa (signing key rotation when) | usually restarting the queue-runner can tip the balance again | 10:15:03 |
| vcunat | Redacted or Malformed Event | 10:15:16 |
| uep | oh, build ingestion is on-boarding of jobs to run, not ingestion of build results... makes more sense now | 10:24:33 |
| hexa (signing key rotation when) | https://github.com/NixOS/hydra/commit/720db63d52ebcbda617603e7aa5b5c750cc6afec was the last change to the scheduling logic | 10:29:52 |
| hexa (signing key rotation when) | we've run with that since around april 29th I would say | 10:31:51 |
| hexa (signing key rotation when) | but that's in the dispatcher, not the ingestor | 10:32:15 |
| hexa (signing key rotation when) | * but that's in the dispatcher, not the queue-monitor | 10:32:41 |
| hexa (signing key rotation when) | and ingestion business is determined here https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/queue-monitor.cc#L54-L82 | 10:33:22 |
| hexa (signing key rotation when) | * and ingestion busyness is determined here https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/queue-monitor.cc#L54-L82 | 10:33:34 |
| uep | i dearly love postres notify, but the fact that it's bundled into a transaction sometimes really hurts | 10:36:45 |
| uep | * i dearly love postgres notify, but the fact that it's bundled into a transaction sometimes really hurts | 10:37:04 |
 K900 | The entire queue setup is one huge crime against postgres | 10:37:29 |
| K900 | Unfortunately | 10:37:47 |
| vcunat | It often feels like restarting the queue runner helps. I tried that again during this discussion, and it does seem like getting through trunk-combined faster (including the Steps completed per minute graph). No idea why this happens, and yes - during the restart we lose already ingested builds that haven't finished, and startup takes some time, so it's not always a win. | 11:02:13 |
 emily | is the queue logic so complicated that it's not obvious how to reimplement it with some more normal queueing thing or is it just a "no free time that anyone wants to spend patching scary C++" thing? | 11:08:01 |
| emily | (I mean I know using Postgres for everything including queueing is popular these days but I'm going to assume that however Hydra does it is not how you would do it in 2025.) | 11:08:36 |
| emily | the code in hydra-queue-runner looks shorter/less terrifying than I expected but I'm guessing the gremlins are just deeper than the surface. | 11:09:49 |
| K900 | It's not complicated at all | 11:10:08 |
| K900 | It's just extremely cursed | 11:10:12 |
| K900 | And no one wants to touch it | 11:10:18 |
| 29 May 2025 |
 Winter | does channels.nixos.org have rate limiting, and if so, is it worse than GitHub where if you sneeze on it without an auth token it'll 429 you? | 14:48:58 |
| K900 | It shouldn't I think | 14:50:27 |
| K900 | Unless Fastly is doing somethign | 14:50:31 |
| K900 | * Unless Fastly is doing something | 14:50:33 |
| Winter | thanks! | 14:51:14 |
| 30 May 2025 |
 SomeoneSerge (back on matrix) | =\ | 13:42:54 |
| SomeoneSerge (back on matrix) | Redacted or Malformed Event | 13:43:12 |
