| 19 Nov 2025 |
 adamcstephens | Are there log events from before this stopped working? | 16:08:09 |
| adamcstephens | It looks like redirects should be followed if I'm reading the go docs correctly. https://pkg.go.dev/net/http#Client.Do and I'd expect a different error if the download itself failed https://github.com/lxc/incus/blob/09e0f36dd5f02aeda5c97a4b27060cef338e1d76/shared/util/net.go#L52 | 16:14:25 |
| adamcstephens | * It looks like redirects should be followed if I'm reading the go docs correctly. https://pkg.go.dev/net/http#Client.Do and I'd expect a different error if the download itself failed (aka not 200) https://github.com/lxc/incus/blob/09e0f36dd5f02aeda5c97a4b27060cef338e1d76/shared/util/net.go#L52 | 16:14:50 |
| adamcstephens | yes this is all too convoluted, and i have some ideas on how to avoid distrobuilder for this but they will take more time and convincing upstream to accept them. unfortunately if we don't fix this before 10 days after the failure started our images will expire and users won't be able to pull them down from the LXC image server. | 16:18:45 |
| adamcstephens | i'll dig into distrobuilder some more to see if i can at the result it's failing to unpack | 16:28:28 |
 Vladimír Čunát | Based on the logs, the "distrobuilder" agent does not follow the redirect. | 16:28:37 |
| Vladimír Čunát | Oh wait. | 16:29:08 |
| Vladimír Čunát | Wrong command at first, but correct statement, I think. | 16:31:53 |
| Vladimír Čunát | grep -F download-by-type/file/squashfs-image /var/log/nginx/access.log | grep -F distrobuilder | grep -F /build/
This command shows empty. Without the last grep I see them. i.e. distrobuild agents only look at the /job/nixos/... URLs and not the followup /build/... URLs.
| 16:33:11 |
| adamcstephens | 👍 thanks | 16:34:02 |
| adamcstephens | It's anubis | 16:54:26 |
| adamcstephens | which also explains why it's not following redirects. it gets the anubis page returned and doesn't even know about them | 16:55:39 |
| adamcstephens | <p>This website is running Anubis version <code>v1.23.1</code>.</p>
| 16:57:43 |
| Vladimír Čunát | OK, so you need an exception to bypass Anubis for this use case. | 17:06:33 |
| Vladimír Čunát | adamcstephens: can you retry now? | 17:10:24 |
| Vladimír Čunát | (just trying a quick prototype) | 17:10:43 |
| adamcstephens | still got the anubis page | 17:11:14 |
| Vladimír Čunát | I suppose someone more capable than me in this would have a look 😅 | 17:17:20 |
| adamcstephens | In theory a bot policy as below should work, but it isn't scoped at all if that is desired. I don't see managed policies already in nixos/infra though, and not sure how the defaults are handled by adding one...
- name: distrobuilder
user_agent_regex: distrobuilder
action: ALLOW
| 17:21:57 |
| Vladimír Čunát | I thought we have these here
https://github.com/NixOS/infra/blob/main/build/hydra-proxy.nix#L96 | 17:23:15 |
| Vladimír Čunát | But maybe I'm wrong. | 17:23:24 |
| Vladimír Čunát | i.e. the approach is to allow some URLs, not some agents. | 17:24:06 |
| adamcstephens | ahh | 17:24:59 |
 hexa | I assume an anubis update was backported and that caused this change in behavior | 17:31:19 |
| adamcstephens | that rule should handle the intermediate and final redirects, but not the initial request /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image | 17:31:27 |
| adamcstephens | yeah, i kinda guessed it was an anubis update given the infra git history | 17:31:43 |
| adamcstephens | how about /job/[^/]+/[^/]+/[^/]+/latest/download-by-type/[^/]+ ? | 17:39:47 |
| adamcstephens | maybe a bit more explicit than the others, so could drop the last match | 17:42:04 |
| Vladimír Čunát | 🤷 deployed
locations."~ ^/job/[^/]+/[^/]+/[^/]+/latest/download-by-type/[^/]+" = {
proxyPass = "http://hydra-server";
};
(temporarily)
| 17:42:55 |
| adamcstephens | hmm, still no luck | 17:44:24 |
