| 27 Oct 2021 |
 Vladimír Čunát | I gather that some "versions" of peertube build are affected by the coreutils bug and some aren't affected? | 14:37:10 |
| Vladimír Čunát | * I gather that some "versions" of peertube build system are affected by the coreutils bug and some aren't affected? | 14:37:19 |
 moritz.hedtke | Maybe its not a coreutils bug but just a different -frandom-seed and we could change the derivation hash of literally any dependent derivation of golang or peertube | 14:37:43 |
| moritz.hedtke | I'm pretty sure I got the root cause: https://github.com/mohe2015/nixpkgs/tree/esbuild-bug look at the last commit - I think its a bug in the golang compiler because for me the previous commit .#esbuild/bin/esbuild segfaults but for the last one it doesn't. Please verify so we can be confident it has nothing to do with coreutils. | 15:23:54 |
| moritz.hedtke | Looking at the diff https://gist.github.com/mohe2015/2a0aeef7d1e7217aec2b25cd8db40775 I'm actually not sure any more and it could possibly be a copy-failure. Do you know where golang gets its random-seed similar data from so I could override it. Or is it dependent on the derivation hash which you can't change. Because I would like to build it on top of master with the same random-seed similar data. | 15:35:59 |
| moritz.hedtke | Checked with xxd and diff and lots of data is overwritten with zeros | 15:42:46 |
| moritz.hedtke | Are that symptoms of coreutils cp bug? | 15:42:57 |
| Vladimír Čunát | That sounds plausible. Holes are long consecutive zeroed ranges. | 15:44:34 |
| Vladimír Čunát | And the identified commit did change something around hole-seeking. | 15:45:09 |
| moritz.hedtke | Ok, so now I don't have to find the non-existent bug in golang any more and we are pretty confident that the coreutils bug is actually this bug. Interesting that changing the derivation fixes it so the bug is probably really extremely rare but if you're building a lot you are bound to experience it. | 15:51:09 |
 das_j | We should move as much of the discussion as possible to https://github.com/NixOS/nixpkgs/issues/143208 since that is more open (and at least one coreutils maintainer is currently following the staging-next PR) and it doesn't clutter the channels that could also be used for other issues | 18:55:24 |
| moritz.hedtke | Should I add a summary of my conclusions of the peertube issue there or somewhere else? | 18:57:22 |
| das_j | Probably the best place is there so everything is in one issue | 19:01:06 |
| das_j | We can hide comments later if they get outdated by newer findings (not deleting them so they are still available for future reference) | 19:01:39 |
| moritz.hedtke | Does anybody know why that happens reproducibly (with esbuild from peertube) at least and changing the derivation usually results in a fine binary. Is the cp behavior so closely dependent on the contents of the file or what? | 19:17:01 |
| moritz.hedtke | Also the binary is copied at https://github.com/NixOS/nixpkgs/blob/b63e29ebad88ffaa579c83d0879310378457a925/pkgs/development/go-modules/generic/default.nix#L252 for golang and not anywhere else? | 19:25:16 |
| moritz.hedtke | because simply copying the binary (across filesystem boundaries) doesn't work so there seems to be something else too (or I'm doing something wrong). Also should I put this discussion also on the issue because this is more of a thought-process writing out | 19:28:47 |
| moritz.hedtke | The binary cache now has the broken binary but for some reason I now fail to reproduce locally. (With nix develop .#esbuild to get dependencies and then nix build --option substitute false .#esbuild). Weird. | 21:42:31 |
| 28 Oct 2021 |
| das_j | Vladimír Čunát: is it possible that your ext4 failures are related to your nix substituting intermediate results from the cache? | 08:35:23 |
| Vladimír Čunát | In reply to @janne.hess:helsinki-systems.de
Vladimír Čunát: is it possible that your ext4 failures are related to your nix substituting intermediate results from the cache? I think almost all inputs were substituted on those attempts. | 08:44:11 |
| das_j | In reply to @vcunat:matrix.org
I think almost all inputs were substituted on those attempts. So is it possible that you just substituted the bad inputs from hydra? That would also explain why changing anything would force you to build locally and not show the issue | 08:46:52 |
| Vladimír Čunát | In reply to @janne.hess:helsinki-systems.de
So is it possible that you just substituted the bad inputs from hydra? That would also explain why changing anything would force you to build locally and not show the issue In both succeeding and failing cases I only retried the final build multiple times, not any dependencies. | 08:48:52 |
| das_j | In reply to @vcunat:matrix.org
In both succeeding and failing cases I only retried the final build multiple times, not any dependencies. so it's likely you just got broken tooling that was built on ZFS and there is no issue with ext4? | 08:52:11 |
| Vladimír Čunát | In reply to @janne.hess:helsinki-systems.de
so it's likely you just got broken tooling that was built on ZFS and there is no issue with ext4? It's ceratinly an option, assuming we have ZFS among Hydra workers. | 08:52:44 |
| Vladimír Čunát | In reply to @janne.hess:helsinki-systems.de
so it's likely you just got broken tooling that was built on ZFS and there is no issue with ext4? * It's ceratinly a possibility, assuming we have ZFS among Hydra workers. | 08:52:53 |
| moritz.hedtke | In reply to @moritz.hedtke:matrix.org
The binary cache now has the broken binary but for some reason I now fail to reproduce locally. (With nix develop .#esbuild to get dependencies and then nix build --option substitute false .#esbuild). Weird. Considering this I believe the zfs workers from hydra to be the culprit. If should be reproducible with these steps assuming cp fails in the esbuild derivation and substituting the go compiler should not matter. I will update them that I strongly believe this to be a misattribution if the issue. Also the bug tracker should be updated if you agree. | 10:24:10 |
| moritz.hedtke | * Considering this I also believe the zfs workers from hydra to be the culprit. If should be reproducible with these steps assuming cp fails in the esbuild derivation and substituting the go compiler should not matter. I will update them that I strongly believe this to be a misattribution if the issue. Also the bug tracker should be updated if you agree. | 10:24:23 |
| moritz.hedtke | The issue mostly was that we both probably paid attention that peertube got built but that was not the broken derivation. Identifying the real broken derivation took a while | 10:25:22 |
| moritz.hedtke | Probably not worth the effort but is it possible to check in which builder it got built and check whether it uses zfs? The first step may even be possible on the web interface I believe | 10:38:39 |
| Vladimír Čunát | In reply to @moritz.hedtke:matrix.org
Probably not worth the effort but is it possible to check in which builder it got built and check whether it uses zfs? The first step may even be possible on the web interface I believe Here, I think: https://hydra.nixos.org/build/156099030#tabs-buildsteps | 10:47:50 |
