| 30 May 2025 |
 SomeoneSerge (back on matrix) | * Mozilla/5.0 (X11; Linux x86\_64; rv:136.0) Gecko/20100101 Firefox/136.0 | 13:47:29 |
 hexa | 136.0 would be from 2025-03-04 | 13:48:11 |
| hexa | and 136.0.4 from 2025-03-27 | 13:48:27 |
| SomeoneSerge (back on matrix) | Why? | 13:49:06 |
| hexa | because bots are advertising all kinds of random browser versions | 13:49:27 |
| SomeoneSerge (back on matrix) | Alright, lynx seems to work, at least that | 13:49:29 |
| hexa | so we only allow semi recent ones to reduce the set of bots we allow through | 13:49:48 |
| hexa | we deployed anubis some time ago, but it broke nix.dev reverse proxying the manual from hydra 🥲 | 13:50:18 |
| hexa | so that was rolled back | 13:50:35 |
| SomeoneSerge (back on matrix) | 👌 I wonder if scrapers were creating significant load, but I'll read the chat history later | 13:52:21 |
| hexa | they called every possible api endpoint and eventually broke hydra-server, so that a manual restart was required | 13:52:52 |
| hexa | * they called every possible endpoint and eventually broke hydra-server, so that a manual restart was required | 13:53:03 |
| hexa | ideally hydra-server would be slightly more robust, but until then we're employing simpler measures | 13:53:42 |
| SomeoneSerge (back on matrix) | What was the name of that zero knowledge hcapcha pass extension? We could do that but with discourse/github id instead of capcha... | 14:00:10 |
 K900 | We can't do anything as long as nix.dev exists as configured | 14:01:42 |
| K900 | Because it can't do auth | 14:01:47 |
| K900 | Any kind of auth | 14:01:54 |
| K900 | It's just a braindead HTTP proxy | 14:02:00 |
 VladimÃr ÄŒunát | We could exempt parts needed by nix.dev, as a compromise. | 14:02:20 |
| VladimÃr ÄŒunát | (at least in theory) | 14:02:45 |
 adamcstephens | can it be configured to send a header? | 14:03:18 |
| K900 | I don't know | 14:05:21 |
| SomeoneSerge (back on matrix) | In reply to @k900:0upti.me
We can't do anything as long as nix.dev exists as configured It doesn't just passthrough headers? | 14:07:29 |
| K900 | It does not | 14:08:32 |
| adamcstephens | https://github.com/NixOS/nix.dev/blob/42eba9d2324918ddac16144399de501d58c9247d/_redirects#L40-L41 | 14:09:20 |
 @emma:rory.gay | https://cgit.rory.gay/BlockScrapersC.git/tree/template/p_user_agent_legacy_browser.txt
https://cgit.rory.gay/BlockScrapersC.git/tree/template/p_user_agent_legacy_os.txt
can confirm :), this is all based on real observations | 14:09:27 |
| adamcstephens | it looks like changing 200 to 301 in the _redirects file would make them actual redirects instead of proxies. i can't tell whether the production is deployed to cloudflare pages or netlify, but this is the same regardless | 14:10:10 |
| K900 | This will create a lot of unnecessary load on Hydra | 14:10:34 |
| hexa | netlify | 14:10:39 |
| K900 | Because Hydra is terribly slow at pulling out artifacts | 14:10:45 |
