| 24 Jul 2025 |
 cleverca22 | my hydra is being swamped by some kind of scrapper/bot/ddos
they are just recursively following every link on every page, ignoring robots.txt, and causing performance problems
user-agents are total garbage, even claiming to be things like macos on ppc, or linux on ppc
no cookies
every ip hits me up once, and then never comes back!
what can be done to block this kind of garbage? | 15:21:14 |
 ma27 | I essentially stole https://github.com/NixOS/infra/blob/7ee3f5c95beda825b742580178f84034ec48aa9c/non-critical-infra/hosts/staging-hydra/hydra-proxy.nix#L9 | 15:25:41 |
| ma27 | and it's working surprisingly well | 15:25:49 |
| cleverca22 | hmmm, i'm getting some like:
Mozilla/5.0 (iPod; U; CPU iPhone OS 4_3 like Mac OS X; as-IN) AppleWebKit/532.43.7 (KHTML, like Gecko) Version/3.0.5 Mobile/8B119 Safari/6532.43.7
Opera/9.82.(X11; Linux i686; mg-MG) Presto/2.9.176 Version/11.00
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_11_5; rv:1.9.5.20) Gecko/3574-09-12 00:50:37.474434 Firefox/3.6.2
Mozilla/5.0 (X11; U; Linux ppc; fr; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.10 (maverick) Firefox/3.6.12
| 15:27:13 |
| cleverca22 | i do see ipod in the list you linked | 15:28:05 |
| cleverca22 | let me see what happens if i apply that..... | 15:28:30 |
| cleverca22 | ma27: yep, i can confirm, traffic is now 403'ing and going to abuse.log! | 15:34:48 |
| cleverca22 | some still slips thru though, but i could tweak the regex to improve that | 15:35:06 |
| cleverca22 | even with the small bits that slip thru, the machine is basically idle now, so its all good | 16:05:22 |
 Sandro | That list at least put Samsung Mobile Browsers out because they are on Chromium 130 🙃 | 23:13:55 |
| Sandro | If you open Developer Tools and choose Android, you get a Nexus with Android 6 | 23:14:37 |
| Sandro | Also I quickly found some friends which should have updated their Firefox more often | 23:14:55 |
| 25 Jul 2025 |
|  Luke joined the room. | 03:22:27 |
| Luke | I posted this in the Terranix channel, but am hoping to get more eyes on it - sorry for double posting!
The high level idea I have is as follows:
- Orchestrator flake: sources all other flakes using input following. Hydra builds this flake, and some cron job regularly runs
flake update to pick up new changes to all packages in the pipeline.
- Pipeline flake: a set of impure builds with a sequential dependency chain that are responsible for executing terraform plan and apply for each stage. These builds would use something like fixed output derivations with a fake hash to skirt network sandboxing. Integration tests would also be represented this way.
- Infrastructure flake: defines the terranix infrastructure, gets used by the various pipeline stages for deployment.
- Application flakes: flakes that define how to build and package application code, these get used by the infrastructure flake.
Then configure Hydra, and one machine should be able to act as a full devops pipeline?
This should let developers easily work on any subset of the system in a local workspace by modifying the flake inputs, as well as allowing things like manual deployments for rollbacks or emergency situations. There might be flaws with this idea, but I am curious to hear what folks think
| 03:24:57 |
| Luke | This is for a declarative CI/CD deployment pipeline running from a single machine | 03:26:21 |
 hexa | cleverca22, ma27, Sandro 🐧 we have moved to anubis since | 12:47:44 |
| Sandro | I know | 12:48:02 |
| hexa | https://grafana.nixos.org/d/fejx5cl0i0s1sb/anubis?orgId=1&from=now-6d&to=now&timezone=utc&var-site=hydra.nixos.org:9001&viewPanel=panel-3 | 12:48:19 |
| Sandro | All those live sucking AI and SEO companies 😒 | 12:48:59 |
|  Vuks joined the room. | 18:04:27 |
| 26 Jul 2025 |
| cleverca22 | i was considering doing that, got a link to how its all configured? | 01:45:12 |
| hexa | https://github.com/NixOS/infra/blob/main/build/hydra-proxy.nix#L14 | 01:46:51 |
| hexa | we currently have a map for some exceptions, but you likely don't need that | 01:47:12 |
| hexa | just bypass the map and point the proxyPass directly at anubis | 01:47:23 |
| Sandro | I do proxy auth but that's more complicated | 11:46:58 |
| Sandro | And I also patch anubis to not lie about the http status code | 11:47:11 |
| hexa | yeah, I looked into that (realistic status codes) shortly and you have to configure the whole bot policy yourself if you want to override these | 13:29:33 |
| hexa | so patching them in the source is more appealing | 13:29:39 |
| hexa | same with go-away, which has various yaml policy definitions | 13:30:00 |
| hexa | I absolutely don't want to manage my own policy list, if I can get away with it | 13:30:15 |
