| 30 May 2025 |
 K900 | I don't know | 14:05:21 |
 SomeoneSerge (back on matrix) | In reply to @k900:0upti.me
We can't do anything as long as nix.dev exists as configured It doesn't just passthrough headers? | 14:07:29 |
| K900 | It does not | 14:08:32 |
 adamcstephens | https://github.com/NixOS/nix.dev/blob/42eba9d2324918ddac16144399de501d58c9247d/_redirects#L40-L41 | 14:09:20 |
 @emma:rory.gay | https://cgit.rory.gay/BlockScrapersC.git/tree/template/p_user_agent_legacy_browser.txt
https://cgit.rory.gay/BlockScrapersC.git/tree/template/p_user_agent_legacy_os.txt
can confirm :), this is all based on real observations | 14:09:27 |
| adamcstephens | it looks like changing 200 to 301 in the _redirects file would make them actual redirects instead of proxies. i can't tell whether the production is deployed to cloudflare pages or netlify, but this is the same regardless | 14:10:10 |
| K900 | This will create a lot of unnecessary load on Hydra | 14:10:34 |
 hexa | netlify | 14:10:39 |
| K900 | Because Hydra is terribly slow at pulling out artifacts | 14:10:45 |
 Alyssa Ross | Somebody suggested just checking if a Netlify header is set and bypassing anubis if so somewhere | 14:11:18 |
| 23 Sep 2025 |
|  Rick (Mindavi) left the room. | 07:36:31 |
| 30 May 2025 |
| adamcstephens | Then redirect them to nixos.org manal? | 14:11:21 |
| adamcstephens | Why is it going directly to hydra anyway... | 14:11:38 |
| hexa | yeah, it could just use the latest version from the cache | 14:13:09 |
| adamcstephens | ahh, this is the nix manual. https://nixos.org/manual/nix/unstable/ redirects to nix.dev 🫠| 14:13:48 |
| hexa | it is just that hydra provides a stable link and that made it easy to proxy that | 14:13:49 |
| adamcstephens | https://docs.netlify.com/routing/redirects/rewrites-proxies/#proxy-to-another-service | 14:17:14 |
| adamcstephens | yes, custom headers are an option for proxying | 14:18:24 |
| adamcstephens | or the request can be signed, but not sure what that entails on the other end | 14:18:51 |
| adamcstephens | In theory, something like this would give us a header we could trust on the hydra side: https://github.com/NixOS/nix.dev/compare/master...adamcstephens:nix.dev:random-header?expand=1 | 15:00:36 |
| Alyssa Ross | I don't think header forging should matter very much — if the scraper bots were smart they'd just use a User-Agent that doesn't look like a browser to anubis. | 15:26:18 |
| Alyssa Ross | (If I'm understanding what you mean by trust) | 15:26:43 |
| hexa | the anubis module unfortunately looks like … use the default bot policy or write your own | 15:28:52 |
| hexa | not sure how if it offers a knob to extend it | 15:29:04 |
| hexa | also not sure where botPolicy gets used https://github.com/NixOS/nixpkgs/blob/96ec055edbe5ee227f28cdbc3f1ddf1df5965102/nixos/modules/services/networking/anubis.nix#L58 | 15:37:52 |
| hexa | I can't find it referenced anywhere | 15:38:10 |
| hexa | ah yeah, https://github.com/NixOS/nixpkgs/pull/401622 | 15:38:57 |
| adamcstephens | maybe "check" or "whitelist" would have been better terms. i'm not too worried about bots forging, but was thinking a known header we could explicitly validate is set on the anubis side. | 17:29:55 |
| adamcstephens | any header would probably work. i put a random string in my example because 🤷 | 17:30:48 |
| hexa | yeah, we can get more creative once bots adapt to these things | 17:31:40 |
