| 18 Jun 2026 |
 hexa | thanks a ton for you work! | 14:58:44 |
| hexa | the really slow part is now uploading to s3 | 15:25:02 |
| hexa | where the smallest outputs take minutes in "Receiving outputs" | 15:25:24 |
| hexa | e.g. systemd services, or rendered text files | 15:25:37 |
| hexa | so the next step that we're looking at it is to move ingestion to a new bucket in de-fra | 15:26:03 |
| hexa | and reconfigure fastly to fallback to us-east1 only when it hits 404 on the de-fra bucket | 15:26:39 |
| hexa | de-fra because currently 100% of our build farm sits in Germany | 15:27:35 |
| hexa | (I blame hetzner for that) | 15:28:19 |
| hexa | Redacted or Malformed Event | 15:28:22 |
 Mic92 | i think compression also still might be not the fastest in this setup | 15:34:32 |
| hexa | did anything relevant about the s3 connection change? like, did we have long-lived keep alive before and no we don't? | 15:34:48 |
| hexa | oh, so receiving uploads is compression + upload | 15:34:57 |
| hexa | god we need to split that up for better visibility | 15:35:09 |
| hexa | Redacted or Malformed Event | 15:35:24 |
 K900 | I assume it's streaming compression? | 15:35:44 |
| hexa | zstd should still be plenty fast for very small files | 15:35:44 |
| hexa | https://hydra.nixos.org/build/331667029 | 15:35:59 |
| hexa | 
Download | 15:36:08 |
| hexa | e.g. activate is fairly small | 15:36:18 |
| K900 | But yeah uploads for small outputs look really bad now | 15:36:21 |
| K900 | Significantly worse than before | 15:36:30 |
| hexa | the old queue runner could do unstable-small in 30 minutes with fewest rebuilds | 15:37:13 |
| hexa | Redacted or Malformed Event | 15:37:21 |
| hexa | as a comparison | 15:37:39 |
| K900 | How good is the otel instrumentation currently on the runner/workers? | 15:38:57 |
| K900 | It might be good to spin up a Tempo instance so we can have proper metrics for this | 15:39:17 |
| K900 | Should be fairly easy to do | 15:39:27 |
| Mic92 | You can't really stream zstd with s3 unfortunally because s3 wants to know the size up-front. In niks3 I solved this by having smallish multi-part upload. I just yesterday introduced something similar in queue-runner because before it was just buffering everything in memory. | 15:40:14 |
| Mic92 | But I can try to make the parts a bit smaller. | 15:40:40 |
| hexa | I don't immediately see an option to set up an otel endpoint in the module | 15:46:17 |
