| 3 Jan 2025 |
 hexa | and whatever is in hydra's queeu | 19:24:17 |
| hexa | * and whatever is in hydra's queue | 19:24:22 |
| 4 Jan 2025 |
 @emma:rory.gay | youre still actively maintaining 32bit packages? | 05:15:17 |
 Vladimír Čunát | When looking at the graph of Hydra's steps completed per minute, I believe I see a gradual decrease after each restart of the queue-runner service. Interesting.
https://grafana.nixos.org/d/MJw9PcAiz/hydra-jobs?orgId=1&from=2024-12-18T15:49:39.948Z&to=2025-01-04T08:18:21.662Z&refresh=1m&var-machine=$__all&viewPanel=panel-20 | 08:21:27 |
| hexa | yeah, that is some spike in runnables for once | 15:49:46 |
| hexa | 22 slots waiting to receive | 17:50:22 |
| hexa | 144 receiving | 17:50:31 |
| hexa | so the build capacity can saturate the compress slots still | 17:50:45 |
| hexa | so we might still want to look into zstd 🙂 | 17:51:42 |
| Vladimír Čunát | Looks like pretty rare occasions so far. And PSI cpusome wasn't even high. | 18:05:14 |
| Vladimír Čunát | * Looks like pretty rare occasions so far. And PSI cpusome wasn't even high. (something like 10%) | 18:05:44 |
| Vladimír Čunát | So increasing threads might still help, too, though I'm not sure why. | 18:06:23 |
| hexa | the compressor are often starved waiting on data from the builders | 18:20:31 |
| hexa | * the compressors are often starved waiting on data from the builders | 18:21:29 |
| 5 Jan 2025 |
| hexa | contemplating banning certain user agents and making people who scrape hydra set an explicit one | 03:07:10 |
| hexa | looking at curl, wget and most of all scrapy | 03:07:20 |
| hexa | when hydra-server gets busy we don't get any metrics any more from it | 03:07:47 |
 @adam:robins.wtf | Is it possible to put Hydra behind the Fastly cache | 11:58:08 |
| @adam:robins.wtf | Would that help here? | 11:58:26 |
 emily | many pages seem too dynamic for that? | 14:03:07 |
| emily | (the expensive ones, I'd assume) | 14:03:12 |
| @adam:robins.wtf | yeah i guess it depends on what they're scraping | 14:55:50 |
 K900 | They're not scraping anything | 14:57:51 |
| @adam:robins.wtf | then what is happening? because hexa said "people who scrape hydra" | 15:01:08 |
| hexa | there are gaps in our graphs on prometheus, and when that happens I also can't reach h.n.o. | 15:03:13 |
| hexa | I browse the access.log, and yes, there are some high frequency scrapers in there | 15:03:33 |
| hexa | we could probably evaluate access logs besser | 15:03:48 |
| hexa | Hits h% Vis. v% Tx. Amount Data
18111 20.20% 4 0.05% 763.06 MiB 2a01:4f9:3070:15e0::1 (pluto.nixos.org)
16250 18.13% 1 0.01% 1.69 GiB 99.245. (random rogers customer)
4059 4.53% 1 0.01% 1.91 MiB 34.44 (google cloud)
2683 2.99% 2 0.02% 2.00 MiB 81.200
| 15:06:18 |
| hexa | this is the last 75.5h | 15:07:32 |
| hexa | estimated from the prometheus scraper, who runs every 15s | 15:07:51 |
