| 19 Nov 2025 |
 adamcstephens | there are two downloads. one for the VM, which is a curl and can be seen in the job, and another for the container image which is handled by distrobuilder. I'd need to instrument distrobuilder more to troubleshoot further what's being returned, but it returns within a second which is clearly not downloading the 274MB container image. | 15:52:57 |
| adamcstephens | whatever hydra is returning is not the image. specifically in these lines
time="2025-11-19T01:39:31Z" level=info msg="Downloading source"
Error: Error while downloading source: Failed unpacking rootfs: Unsupported compression
time="2025-11-19T01:39:31Z" level=error msg="Failed running distrobuilder" err="Error while downloading source: Failed unpacking rootfs: Unsupported compression"
| 15:54:43 |
| adamcstephens | https://github.com/lxc/distrobuilder/blob/main/sources/nixos-http.go | 15:55:06 |
| adamcstephens | A wget/curl on the same image URL downloads correctly. | 15:57:28 |
 vcunat | Do you know what kind of user agent it's sending? | 15:59:12 |
| vcunat | 🤔 or a precise recent timestamp. | 15:59:35 |
| adamcstephens | DEBUG [2025-11-18T23:00:38-05:00] Downloading hydra output: https://hydra.nixos.org/job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image
Error: Error while downloading source: Failed unpacking rootfs: Unsupported compression
ERROR [2025-11-18T23:00:39-05:00] Failed running distrobuilder err="Error while downloading source: Failed unpacking rootfs: Unsupported compression
| 16:00:17 |
| adamcstephens | (i instrumented distrobuilder enough to print the URL so I could verify) | 16:00:31 |
| adamcstephens | the user agent should be exactly distrobuilder as set by this call https://github.com/lxc/distrobuilder/blob/39f431652247798dfd6935b79a658935ca0672cf/sources/common.go#L148 | 16:01:44 |
| adamcstephens | https://github.com/lxc/incus/blob/09e0f36dd5f02aeda5c97a4b27060cef338e1d76/shared/util/net.go#L21 | 16:02:21 |
| vcunat | [16/Nov/2025:01:01:53 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1384 "-" "distrobuilder"
[16/Nov/2025:01:07:36 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1385 "-" "distrobuilder"
[16/Nov/2025:01:23:13 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1531 "-" "distrobuilder"
[16/Nov/2025:01:39:20 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1386 "-" "distrobuilder"
[17/Nov/2025:01:02:14 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1537 "-" "distrobuilder"
[17/Nov/2025:01:07:56 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1384 "-" "distrobuilder"
[17/Nov/2025:01:23:30 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1384 "-" "distrobuilder"
[17/Nov/2025:01:39:09 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1388 "-" "distrobuilder"
[17/Nov/2025:02:15:59 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1538 "-" "distrobuilder"
[17/Nov/2025:02:22:49 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1384 "-" "distrobuilder"
[17/Nov/2025:02:58:58 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1377 "-" "distrobuilder"
[18/Nov/2025:01:00:34 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1386 "-" "distrobuilder"
[18/Nov/2025:01:08:00 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1537 "-" "distrobuilder"
[18/Nov/2025:01:23:45 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1386 "-" "distrobuilder"
[18/Nov/2025:01:39:22 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1383 "-" "distrobuilder"
[19/Nov/2025:01:00:34 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1384 "-" "distrobuilder"
[19/Nov/2025:01:08:09 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1385 "-" "distrobuilder"
[19/Nov/2025:01:23:44 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1385 "-" "distrobuilder"
[19/Nov/2025:01:39:25 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1384 "-" "distrobuilder"
[19/Nov/2025:03:29:30 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/1.1" 302 360 "-" "Wget/1.25.0"
[19/Nov/2025:04:00:39 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 200 1534 "-" "distrobuilder"
[19/Nov/2025:04:00:47 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/1.1" 302 360 "-" "Wget/1.25.0"
[19/Nov/2025:15:45:07 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 302 360 "-" "curl/8.16.0"
[19/Nov/2025:15:45:12 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 302 360 "-" "curl/8.16.0"
[19/Nov/2025:15:45:37 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 302 360 "-" "ditrobuilder (not actually, just a test)"
[19/Nov/2025:15:45:43 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 302 360 "-" "ditrobuilder (not actually, just a test)"
[19/Nov/2025:15:45:44 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 302 360 "-" "ditrobuilder (not actually, just a test)"
[19/Nov/2025:15:45:46 +0000] "GET /job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image HTTP/2.0" 302 360 "-" "ditrobuilder (not actually, just a test)"
| 16:03:55 |
 emily | does distrobuilder follow redirects? that could explain the < 1 sec download
# curl https://hydra.nixos.org/job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux/latest/download-by-type/file/squashfs-image -i
HTTP/2 302
server: nginx
date: Wed, 19 Nov 2025 16:03:35 GMT
content-type: text/html; charset=utf-8
content-length: 360
location: https://hydra.nixos.org/build/314008482/download-by-type/file/squashfs-image
set-cookie: hydra_session=e7dae4552403e83c008193eb14e44bfb37180ae4; path=/; expires=Wed, 26-Nov-2025 16:03:35 GMT; HttpOnly; SameSite=Lax
vary: Accept
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Moved</title>
</head>
<body>
<p>This item has moved <a href="https://hydra.nixos.org/build/314008482/download-by-type/file/squashfs-image">here</a>.</p>
</body>
</html>
| 16:04:06 |
| adamcstephens | I'd have to verify, but I don't think these redirects are new are they? e.g. we didn't add them 6 days ago? | 16:04:40 |
| vcunat | That's what I wonder. | 16:04:55 |
| vcunat | But you can see it in the log above. | 16:05:08 |
| vcunat | They used to be 200, now they're 302. | 16:05:17 |
| adamcstephens | I can do some digging on the redirects | 16:07:55 |
| adamcstephens | Are there log events from before this stopped working? | 16:08:09 |
| adamcstephens | It looks like redirects should be followed if I'm reading the go docs correctly. https://pkg.go.dev/net/http#Client.Do and I'd expect a different error if the download itself failed https://github.com/lxc/incus/blob/09e0f36dd5f02aeda5c97a4b27060cef338e1d76/shared/util/net.go#L52 | 16:14:25 |
| adamcstephens | * It looks like redirects should be followed if I'm reading the go docs correctly. https://pkg.go.dev/net/http#Client.Do and I'd expect a different error if the download itself failed (aka not 200) https://github.com/lxc/incus/blob/09e0f36dd5f02aeda5c97a4b27060cef338e1d76/shared/util/net.go#L52 | 16:14:50 |
| adamcstephens | yes this is all too convoluted, and i have some ideas on how to avoid distrobuilder for this but they will take more time and convincing upstream to accept them. unfortunately if we don't fix this before 10 days after the failure started our images will expire and users won't be able to pull them down from the LXC image server. | 16:18:45 |
| adamcstephens | i'll dig into distrobuilder some more to see if i can at the result it's failing to unpack | 16:28:28 |
| vcunat | Based on the logs, the "distrobuilder" agent does not follow the redirect. | 16:28:37 |
| vcunat | Oh wait. | 16:29:08 |
| vcunat | Wrong command at first, but correct statement, I think. | 16:31:53 |
| vcunat | grep -F download-by-type/file/squashfs-image /var/log/nginx/access.log | grep -F distrobuilder | grep -F /build/
This command shows empty. Without the last grep I see them. i.e. distrobuild agents only look at the /job/nixos/... URLs and not the followup /build/... URLs.
| 16:33:11 |
| adamcstephens | 👍 thanks | 16:34:02 |
| adamcstephens | It's anubis | 16:54:26 |
| adamcstephens | which also explains why it's not following redirects. it gets the anubis page returned and doesn't even know about them | 16:55:39 |
| adamcstephens | <p>This website is running Anubis version <code>v1.23.1</code>.</p>
| 16:57:43 |
