| 25 Nov 2024 |
 John Ericson | emily: check out https://github.com/NixOS/nix/issues/11954 and https://github.com/NixOS/nix/issues/11955 re those CA derivation Nixpkgs PRs btw | 19:09:59 |
| John Ericson | I know finally understand what trofi was doing long ago in mkDerivation, and I agree that's the right thing to do for now | 19:10:24 |
| John Ericson | I think I have a decent plan for making things optimized in the CA world, but it should wait until after CA stabilization | 19:10:58 |
 emily | that quasi-derivation stuff makes me think about Ninja validations (probably because I was just talking about them rather than because they're super directly related, but they do involve tweaking build graph semantics), do you know about those? | 19:27:28 |
| John Ericson | emily: oh I don't know those | 19:28:23 |
| John Ericson | sounds like a good thing to read about ;) | 19:28:28 |
| John Ericson | * sounds like a good thing to read about :) | 19:28:32 |
| emily | In reply to @Ericson2314:matrix.org
emily: oh I don't know those https://ninja-build.org/manual.html#validations basically, we could extend the build graph with nodes that you can attach to another node that
(a) run after that node's completion and get to use its outputs;
(b) don't block the builds of things downstream of that node; but
(c) if they fail, the node they're attached to itself is considered to have failed | 19:29:35 |
| emily | in a world where we split up check phases from main derivations, this would mean that you could run LLVM's tests in parallel with building things that use that LLVM | 19:29:54 |
| John Ericson | oh interesting point via speculative execution | 19:30:03 |
| emily | and have it and everything downstream be considered to have failed if the tests fail | 19:30:07 |
| John Ericson | that's great | 19:30:11 |
| emily | currently lots of things are blocked on slow tests | 19:30:13 |
| emily | we could gain a huge amount of parallelization if tests were taken out of the critical path entirely like this | 19:30:24 |
| John Ericson | for tests I was just thinking "well, depend on the underlying builds at your own risk" | 19:30:33 |
| John Ericson | but yes this could be used with that | 19:30:40 |
| emily | yeah, but if you avoid depending on the underlying derivation it still means that tests become a blocking thing :) | 19:30:56 |
| emily | (this came up in the context of a long discussion about dynamic library stubs / relinking to avoid mass rebuilds from ABI-compatible library changes, https://discourse.nixos.org/t/why-does-the-nixos-infrastructure-have-to-be-hosted-in-a-centralized-way/46789/32 onwards but it's quite long) | 19:31:49 |
| emily | anyway, +1 to everything that gets rid of the phantom dependencies at the Nix level | 19:32:04 |
|  @apurba007:matrix.org left the room. | 19:34:49 |
 p14 | emily: I did a small poc of this over the weekend. And looked into the effects. It looks like a plausible thing to me. Interesting question of how/whether it can be wired into nixpkgs in general without it requiring a big-bang rewrite. | 19:48:30 |
| p14 | It is very cool watching a dependency rebuild but seeing an instant relink for an expensive to build dependent. | 19:49:33 |
| emily | nice | 19:50:26 |
| emily | would be cool to see your prototype | 19:50:29 |
| p14 | Afk right now, will get back to you on that. | 19:51:19 |
| John Ericson | @emilazy:matrix.org I like your thinking in that thread. I agree the interesting thing is not perfect minimal rebuilds, but separating concerns purely at scale | 19:58:17 |
| emily | my ideal is to get away from staging entirely and do everything as a bors-style merge queue where every single change leaves everything green, but even applying all the ideas in that thread to the maximum extent probably still leaves a very substantial gap away from making that viable :) | 19:59:50 |
| emily | but it's frustrating that we at least have a way to deploy one-byte security patches to libraries at scale that doesn't involve compromising the model like replaceDependencies but – we don't do it | 20:00:19 |
| emily | (for understandable reasons, given Nixpkgs legacy, but still.) | 20:00:29 |
| John Ericson | If we make sure we don't starve the remaining mass rebuilds, I think we can do it! | 20:00:41 |
