| 19 Nov 2025 |
 vcunat | But you can see it in the log above. | 16:05:08 |
| vcunat | They used to be 200, now they're 302. | 16:05:17 |
 adamcstephens | I can do some digging on the redirects | 16:07:55 |
| adamcstephens | Are there log events from before this stopped working? | 16:08:09 |
| adamcstephens | It looks like redirects should be followed if I'm reading the go docs correctly. https://pkg.go.dev/net/http#Client.Do and I'd expect a different error if the download itself failed https://github.com/lxc/incus/blob/09e0f36dd5f02aeda5c97a4b27060cef338e1d76/shared/util/net.go#L52 | 16:14:25 |
| adamcstephens | * It looks like redirects should be followed if I'm reading the go docs correctly. https://pkg.go.dev/net/http#Client.Do and I'd expect a different error if the download itself failed (aka not 200) https://github.com/lxc/incus/blob/09e0f36dd5f02aeda5c97a4b27060cef338e1d76/shared/util/net.go#L52 | 16:14:50 |
| adamcstephens | yes this is all too convoluted, and i have some ideas on how to avoid distrobuilder for this but they will take more time and convincing upstream to accept them. unfortunately if we don't fix this before 10 days after the failure started our images will expire and users won't be able to pull them down from the LXC image server. | 16:18:45 |
| adamcstephens | i'll dig into distrobuilder some more to see if i can at the result it's failing to unpack | 16:28:28 |
| vcunat | Based on the logs, the "distrobuilder" agent does not follow the redirect. | 16:28:37 |
| vcunat | Oh wait. | 16:29:08 |
| vcunat | Wrong command at first, but correct statement, I think. | 16:31:53 |
| vcunat | grep -F download-by-type/file/squashfs-image /var/log/nginx/access.log | grep -F distrobuilder | grep -F /build/
This command shows empty. Without the last grep I see them. i.e. distrobuild agents only look at the /job/nixos/... URLs and not the followup /build/... URLs.
| 16:33:11 |
| adamcstephens | 👍 thanks | 16:34:02 |
| adamcstephens | It's anubis | 16:54:26 |
| adamcstephens | which also explains why it's not following redirects. it gets the anubis page returned and doesn't even know about them | 16:55:39 |
| adamcstephens | <p>This website is running Anubis version <code>v1.23.1</code>.</p>
| 16:57:43 |
| vcunat | OK, so you need an exception to bypass Anubis for this use case. | 17:06:33 |
| vcunat | adamcstephens: can you retry now? | 17:10:24 |
| vcunat | (just trying a quick prototype) | 17:10:43 |
| adamcstephens | still got the anubis page | 17:11:14 |
| vcunat | I suppose someone more capable than me in this would have a look 😅 | 17:17:20 |
| adamcstephens | In theory a bot policy as below should work, but it isn't scoped at all if that is desired. I don't see managed policies already in nixos/infra though, and not sure how the defaults are handled by adding one...
- name: distrobuilder
user_agent_regex: distrobuilder
action: ALLOW
| 17:21:57 |
| vcunat | I thought we have these here
https://github.com/NixOS/infra/blob/main/build/hydra-proxy.nix#L96 | 17:23:15 |
| vcunat | But maybe I'm wrong. | 17:23:24 |
| vcunat | i.e. the approach is to allow some URLs, not some agents. | 17:24:06 |
| adamcstephens | ahh | 17:24:59 |
 hexa (signing key rotation when) | I assume an anubis update was backported and that caused this change in behavior | 17:31:19 |
| adamcstephens | that rule should handle the intermediate and final redirects, but not the initial request /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image | 17:31:27 |
| adamcstephens | yeah, i kinda guessed it was an anubis update given the infra git history | 17:31:43 |
| adamcstephens | how about /job/[^/]+/[^/]+/[^/]+/latest/download-by-type/[^/]+ ? | 17:39:47 |
