| 19 Nov 2025 |
 adamcstephens | the user agent should be exactly distrobuilder as set by this call https://github.com/lxc/distrobuilder/blob/39f431652247798dfd6935b79a658935ca0672cf/sources/common.go#L148 | 16:01:44 |
| adamcstephens | https://github.com/lxc/incus/blob/09e0f36dd5f02aeda5c97a4b27060cef338e1d76/shared/util/net.go#L21 | 16:02:21 |
 Vladimír Čunát | [16/Nov/2025:01:01:53 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1384 "-" "distrobuilder"
[16/Nov/2025:01:07:36 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1385 "-" "distrobuilder"
[16/Nov/2025:01:23:13 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1531 "-" "distrobuilder"
[16/Nov/2025:01:39:20 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1386 "-" "distrobuilder"
[17/Nov/2025:01:02:14 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1537 "-" "distrobuilder"
[17/Nov/2025:01:07:56 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1384 "-" "distrobuilder"
[17/Nov/2025:01:23:30 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1384 "-" "distrobuilder"
[17/Nov/2025:01:39:09 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1388 "-" "distrobuilder"
[17/Nov/2025:02:15:59 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1538 "-" "distrobuilder"
[17/Nov/2025:02:22:49 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1384 "-" "distrobuilder"
[17/Nov/2025:02:58:58 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1377 "-" "distrobuilder"
[18/Nov/2025:01:00:34 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1386 "-" "distrobuilder"
[18/Nov/2025:01:08:00 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1537 "-" "distrobuilder"
[18/Nov/2025:01:23:45 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1386 "-" "distrobuilder"
[18/Nov/2025:01:39:22 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1383 "-" "distrobuilder"
[19/Nov/2025:01:00:34 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1384 "-" "distrobuilder"
[19/Nov/2025:01:08:09 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1385 "-" "distrobuilder"
[19/Nov/2025:01:23:44 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1385 "-" "distrobuilder"
[19/Nov/2025:01:39:25 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1384 "-" "distrobuilder"
[19/Nov/2025:03:29:30 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/1.1" 302 360 "-" "Wget/1.25.0"
[19/Nov/2025:04:00:39 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1534 "-" "distrobuilder"
[19/Nov/2025:04:00:47 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/1.1" 302 360 "-" "Wget/1.25.0"
[19/Nov/2025:15:45:07 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 302 360 "-" "curl/8.16.0"
[19/Nov/2025:15:45:12 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 302 360 "-" "curl/8.16.0"
[19/Nov/2025:15:45:37 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 302 360 "-" "ditrobuilder (not actually, just a test)"
[19/Nov/2025:15:45:43 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 302 360 "-" "ditrobuilder (not actually, just a test)"
[19/Nov/2025:15:45:44 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 302 360 "-" "ditrobuilder (not actually, just a test)"
[19/Nov/2025:15:45:46 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 302 360 "-" "ditrobuilder (not actually, just a test)"
| 16:03:55 |
 emily | does distrobuilder follow redirects? that could explain the < 1 sec download
# curl https://hydra.nixos.org/job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image -i
HTTP/2 302
server: nginx
date: Wed, 19 Nov 2025 16:03:35 GMT
content-type: text/html; charset=utf-8
content-length: 360
location: https://hydra.nixos.org/build/314008482/download-by-type/file/squashfs-image
set-cookie: hydra_session=e7dae4552403e83c008193eb14e44bfb37180ae4; path=/; expires=Wed, 26-Nov-2025 16:03:35 GMT; HttpOnly; SameSite=Lax
vary: Accept
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Moved</title>
</head>
<body>
<p>This item has moved <a href="https://hydra.nixos.org/build/314008482/download-by-type/file/squashfs-image">here</a>.</p>
</body>
</html>
| 16:04:06 |
| adamcstephens | I'd have to verify, but I don't think these redirects are new are they? e.g. we didn't add them 6 days ago? | 16:04:40 |
| Vladimír Čunát | That's what I wonder. | 16:04:55 |
| Vladimír Čunát | But you can see it in the log above. | 16:05:08 |
| Vladimír Čunát | They used to be 200, now they're 302. | 16:05:17 |
| adamcstephens | I can do some digging on the redirects | 16:07:55 |
| adamcstephens | Are there log events from before this stopped working? | 16:08:09 |
| adamcstephens | It looks like redirects should be followed if I'm reading the go docs correctly. https://pkg.go.dev/net/http#Client.Do and I'd expect a different error if the download itself failed https://github.com/lxc/incus/blob/09e0f36dd5f02aeda5c97a4b27060cef338e1d76/shared/util/net.go#L52 | 16:14:25 |
| adamcstephens | * It looks like redirects should be followed if I'm reading the go docs correctly. https://pkg.go.dev/net/http#Client.Do and I'd expect a different error if the download itself failed (aka not 200) https://github.com/lxc/incus/blob/09e0f36dd5f02aeda5c97a4b27060cef338e1d76/shared/util/net.go#L52 | 16:14:50 |
| adamcstephens | yes this is all too convoluted, and i have some ideas on how to avoid distrobuilder for this but they will take more time and convincing upstream to accept them. unfortunately if we don't fix this before 10 days after the failure started our images will expire and users won't be able to pull them down from the LXC image server. | 16:18:45 |
| adamcstephens | i'll dig into distrobuilder some more to see if i can at the result it's failing to unpack | 16:28:28 |
| Vladimír Čunát | Based on the logs, the "distrobuilder" agent does not follow the redirect. | 16:28:37 |
| Vladimír Čunát | Oh wait. | 16:29:08 |
| Vladimír Čunát | Wrong command at first, but correct statement, I think. | 16:31:53 |
| Vladimír Čunát | grep -F download-by-type/file/squashfs-image /var/log/nginx/access.log | grep -F distrobuilder | grep -F /build/
This command shows empty. Without the last grep I see them. i.e. distrobuild agents only look at the /job/nixos/... URLs and not the followup /build/... URLs.
| 16:33:11 |
| adamcstephens | 👍 thanks | 16:34:02 |
| adamcstephens | It's anubis | 16:54:26 |
| adamcstephens | which also explains why it's not following redirects. it gets the anubis page returned and doesn't even know about them | 16:55:39 |
| adamcstephens | <p>This website is running Anubis version <code>v1.23.1</code>.</p>
| 16:57:43 |
| Vladimír Čunát | OK, so you need an exception to bypass Anubis for this use case. | 17:06:33 |
| Vladimír Čunát | adamcstephens: can you retry now? | 17:10:24 |
| Vladimír Čunát | (just trying a quick prototype) | 17:10:43 |
| adamcstephens | still got the anubis page | 17:11:14 |
| Vladimír Čunát | I suppose someone more capable than me in this would have a look 😅 | 17:17:20 |
| adamcstephens | In theory a bot policy as below should work, but it isn't scoped at all if that is desired. I don't see managed policies already in nixos/infra though, and not sure how the defaults are handled by adding one...
- name: distrobuilder
user_agent_regex: distrobuilder
action: ALLOW
| 17:21:57 |
| Vladimír Čunát | I thought we have these here
https://github.com/NixOS/infra/blob/main/build/hydra-proxy.nix#L96 | 17:23:15 |
| Vladimír Čunát | But maybe I'm wrong. | 17:23:24 |
