| 27 Oct 2021 |
 Vladimír Čunát | (i.e. move that revert to the staging branch) | 12:25:58 |
 das_j | can do | 12:26:10 |
 moritz.hedtke | Goddammit Vladimír Čunát or somebody else can you try https://gist.githubusercontent.com/mohe2015/23407b8f83ae8cca0948629789d08518/raw/264e85a4a879ae4195c32756f5473373beab844a/goddammit.diff on top of staging-next please? Probably the esbuild version is just broken. | 13:44:52 |
| Vladimír Čunát | In reply to @moritz.hedtke:matrix.org
Goddammit Vladimír Čunát or somebody else can you try https://gist.githubusercontent.com/mohe2015/23407b8f83ae8cca0948629789d08518/raw/264e85a4a879ae4195c32756f5473373beab844a/goddammit.diff on top of staging-next please? Probably the esbuild version is just broken. It builds for me. (I didn't really inspect what it does and the implications.) | 14:33:06 |
| moritz.hedtke | Thanks for testing. It just updates to the latest version of peertube (with the minimal effort). Also we missed a place where we used esbuild instead of esbuild_locked (the yarn.lock pinned version) but I tested and that didn't matter. | 14:34:14 |
| moritz.hedtke | As I read a bugreport about that specific esbuild version (with an unrelated bug) I just assume its either golang that breaks the build or something else. But I'm 95% sure its esbuild even though other tools got updated too. Maybe I will investigate more later but now its way more managable. Also das_j should probably update the coreutils bug report if you agree with my assessment | 14:35:38 |
| Vladimír Čunát | I gather that some "versions" of peertube build are affected by the coreutils bug and some aren't affected? | 14:37:10 |
| Vladimír Čunát | * I gather that some "versions" of peertube build system are affected by the coreutils bug and some aren't affected? | 14:37:19 |
| moritz.hedtke | Maybe its not a coreutils bug but just a different -frandom-seed and we could change the derivation hash of literally any dependent derivation of golang or peertube | 14:37:43 |
| moritz.hedtke | I'm pretty sure I got the root cause: https://github.com/mohe2015/nixpkgs/tree/esbuild-bug look at the last commit - I think its a bug in the golang compiler because for me the previous commit .#esbuild/bin/esbuild segfaults but for the last one it doesn't. Please verify so we can be confident it has nothing to do with coreutils. | 15:23:54 |
| moritz.hedtke | Looking at the diff https://gist.github.com/mohe2015/2a0aeef7d1e7217aec2b25cd8db40775 I'm actually not sure any more and it could possibly be a copy-failure. Do you know where golang gets its random-seed similar data from so I could override it. Or is it dependent on the derivation hash which you can't change. Because I would like to build it on top of master with the same random-seed similar data. | 15:35:59 |
| moritz.hedtke | Checked with xxd and diff and lots of data is overwritten with zeros | 15:42:46 |
| moritz.hedtke | Are that symptoms of coreutils cp bug? | 15:42:57 |
| Vladimír Čunát | That sounds plausible. Holes are long consecutive zeroed ranges. | 15:44:34 |
| Vladimír Čunát | And the identified commit did change something around hole-seeking. | 15:45:09 |
| moritz.hedtke | Ok, so now I don't have to find the non-existent bug in golang any more and we are pretty confident that the coreutils bug is actually this bug. Interesting that changing the derivation fixes it so the bug is probably really extremely rare but if you're building a lot you are bound to experience it. | 15:51:09 |
| das_j | We should move as much of the discussion as possible to https://github.com/NixOS/nixpkgs/issues/143208 since that is more open (and at least one coreutils maintainer is currently following the staging-next PR) and it doesn't clutter the channels that could also be used for other issues | 18:55:24 |
| moritz.hedtke | Should I add a summary of my conclusions of the peertube issue there or somewhere else? | 18:57:22 |
| das_j | Probably the best place is there so everything is in one issue | 19:01:06 |
| das_j | We can hide comments later if they get outdated by newer findings (not deleting them so they are still available for future reference) | 19:01:39 |
| moritz.hedtke | Does anybody know why that happens reproducibly (with esbuild from peertube) at least and changing the derivation usually results in a fine binary. Is the cp behavior so closely dependent on the contents of the file or what? | 19:17:01 |
| moritz.hedtke | Also the binary is copied at https://github.com/NixOS/nixpkgs/blob/b63e29ebad88ffaa579c83d0879310378457a925/pkgs/development/go-modules/generic/default.nix#L252 for golang and not anywhere else? | 19:25:16 |
| moritz.hedtke | because simply copying the binary (across filesystem boundaries) doesn't work so there seems to be something else too (or I'm doing something wrong). Also should I put this discussion also on the issue because this is more of a thought-process writing out | 19:28:47 |
| moritz.hedtke | The binary cache now has the broken binary but for some reason I now fail to reproduce locally. (With nix develop .#esbuild to get dependencies and then nix build --option substitute false .#esbuild). Weird. | 21:42:31 |
| 28 Oct 2021 |
| das_j | Vladimír Čunát: is it possible that your ext4 failures are related to your nix substituting intermediate results from the cache? | 08:35:23 |
| Vladimír Čunát | In reply to @janne.hess:helsinki-systems.de
Vladimír Čunát: is it possible that your ext4 failures are related to your nix substituting intermediate results from the cache? I think almost all inputs were substituted on those attempts. | 08:44:11 |
| das_j | In reply to @vcunat:matrix.org
I think almost all inputs were substituted on those attempts. So is it possible that you just substituted the bad inputs from hydra? That would also explain why changing anything would force you to build locally and not show the issue | 08:46:52 |
| Vladimír Čunát | In reply to @janne.hess:helsinki-systems.de
So is it possible that you just substituted the bad inputs from hydra? That would also explain why changing anything would force you to build locally and not show the issue In both succeeding and failing cases I only retried the final build multiple times, not any dependencies. | 08:48:52 |
| das_j | In reply to @vcunat:matrix.org
In both succeeding and failing cases I only retried the final build multiple times, not any dependencies. so it's likely you just got broken tooling that was built on ZFS and there is no issue with ext4? | 08:52:11 |
| Vladimír Čunát | In reply to @janne.hess:helsinki-systems.de
so it's likely you just got broken tooling that was built on ZFS and there is no issue with ext4? It's ceratinly an option, assuming we have ZFS among Hydra workers. | 08:52:44 |
