| 5 Jan 2025 |
 hexa | looking at curl, wget and most of all scrapy | 03:07:20 |
| hexa | when hydra-server gets busy we don't get any metrics any more from it | 03:07:47 |
 @adam:robins.wtf | Is it possible to put Hydra behind the Fastly cache | 11:58:08 |
| @adam:robins.wtf | Would that help here? | 11:58:26 |
 emily | many pages seem too dynamic for that? | 14:03:07 |
| emily | (the expensive ones, I'd assume) | 14:03:12 |
| @adam:robins.wtf | yeah i guess it depends on what they're scraping | 14:55:50 |
 K900 | They're not scraping anything | 14:57:51 |
| @adam:robins.wtf | then what is happening? because hexa said "people who scrape hydra" | 15:01:08 |
| hexa | there are gaps in our graphs on prometheus, and when that happens I also can't reach h.n.o. | 15:03:13 |
| hexa | I browse the access.log, and yes, there are some high frequency scrapers in there | 15:03:33 |
| hexa | we could probably evaluate access logs besser | 15:03:48 |
| hexa | Hits h% Vis. v% Tx. Amount Data
18111 20.20% 4 0.05% 763.06 MiB 2a01:4f9:3070:15e0::1 (pluto.nixos.org)
16250 18.13% 1 0.01% 1.69 GiB 99.245. (random rogers customer)
4059 4.53% 1 0.01% 1.91 MiB 34.44 (google cloud)
2683 2.99% 2 0.02% 2.00 MiB 81.200
| 15:06:18 |
| hexa | this is the last 75.5h | 15:07:32 |
| hexa | estimated from the prometheus scraper, who runs every 15s | 15:07:51 |
| hexa | * | 15:13:29 |
 raitobezarius | reaction has something to block on the L3 level the scrapers | 15:19:37 |
| raitobezarius | https://reaction.ppom.me/filters/ai-crawlers.html | 15:20:39 |
| K900 | Oh sorry I meant Fastly | 15:21:47 |
| @adam:robins.wtf | that makes more sense :) | 15:22:36 |
| @adam:robins.wtf | i didn't mean to imply fastly was scraping us. i was just wondering if we could leverage fastly to protect hydra | 15:22:55 |
| hexa | tbh, the hydra-server needs to be more robust | 16:29:19 |
| hexa | it can't just lock up | 16:29:24 |
 Vladimír Čunát | Maybe we should separate the external-facing web somehow. | 16:31:53 |
| emily | the web UI runs on the same machine that holds the signing key, right? | 16:33:26 |
| hexa | yes | 16:37:59 |
| emily | scary | 16:39:16 |
| emily | signing key rotation when | 16:39:28 |
| hexa | different user | 16:39:33 |
| raitobezarius | the power of unix perm isolation | 16:39:50 |
