| 10 Apr 2026 |
 atemu12 | The hard way would be to build soongnix and run it inside a drv to get per-project drvs (100s) | 23:06:51 |
| 11 Apr 2026 |
 jaen | Yeah, It Depends™ | 01:48:50 |
| jaen | The MVP is writing a ninja parser that doesn't choke on how big AOSP is (that I have mostly mapped out), welding it onto nix-ninja (with a few CLI oddities Soong insists on) and seeing what falls out. Which will probably be "congrats, you now have loads of dyndrv profiling to do" and it is a bit of an open question how far you can go with optimisations. I don't think it's fundamentally impossible to get it to scale (bazel, for all of it being a giant PITA, somehow does it), but may end up not being practical. | 01:49:42 |
| jaen | Which is where the second part of it comes in - I'm trying to write it in a modular enough way, so you could decide to take out the ninja chunks, write some Soong piece and keep the rest of niceties for free. Or Grade, or Cmake, or Kbuild or whatever. I just picked AOSP as the first target because I was annoyed at how painful robotnix was to iterate on (until CyclicPentane came along and showed me I just sucked xD), but the idea is to provide something all whatever2nix tools could use. | 01:52:30 |
| jaen | And if it turns out that there's a hard ceiling to dyndrvs that makes the simple & dumb approach not work, then I guess I can always pivot to Sonngnix 2: Electric Bogaloo. | 01:54:43 |
| jaen | * | 01:55:04 |
| jaen | As far as I understand it, the biggest difference between the two is Soong(nix) being applicative (as far as I can tell) and dyndrvs (and thus shinobi) being monadic. It doesn't change much for AOSP build since applicative is strictly less powerful (you have to know the build graph ahead of time), but is in general more powerful (because you can modify the build graph during the build, when you discover new dependencies), so it doesn't hurt to try and leverage that, if practicable. | 02:02:24 |
| jaen | But I guess that's enough of off-topic for now | 02:02:43 |
 pentane | fwiw i think it's perfectly on-topic, and I'm also interested in this | 06:45:11 |
 magic_rb | Not yet enough offtopic. I never thought of dyndrv as being monadic while normally were in applicative land, but damn youre right | 06:55:49 |
| magic_rb | Does bazel do sandbox setup? Cause assuming you dont use stdenv for your dyndrvs, then the only real thing nix does except for a fork exec is the sandbox setup | 07:37:36 |
| magic_rb | Maybe if we could reuse the sandbox for a group of dyndrvs? | 07:37:50 |
| jaen | It does have sandboxing, but it is less strict about it than Nix. With Nix you have to really try to make something non-reproducible, while with Bazel it's the other way around - it's not too hard to end up with "FFS why it does break with remote execution". | 07:40:51 |
| jaen | I guess it's historically because for Nix hermeticity was the point, while for Bazel it was just means for an end (hard to do webscale builds if you can't reliably cache) | 07:42:10 |
| jaen | Possibly? But then the trade-off becomes, I think, cache granularity. | 07:47:06 |
| jaen | One of the few downsides of Nix as opposed to Bazel is the default build granularity - for Bazel you are encouraged to go as granular as a single file, for Nix is more natural to go at a package granularity. And depending on how you chunk you, end up with different incrementality and early cut-off trade-offs, I think. But "Builds Systems a la Carte" is probably going to explain this better than me. | 07:49:30 |
| jaen | I haven't tested this specifically for building things together, but at least for planning (that is, creating dyndrvs as you go) the exact trade-off is you either rebuild more or you eat more of the overhead. | 07:51:01 |
| jaen | My hope is that there was something obvious missed in the PoC and now that I'll get paid to write it myself, there will be some easy fix lurking. Or if it not, I'll just keep pestering Ericson for fixes/guidance xD | 08:01:34 |
 neobrain | Thanks for the explanation, that's super interesting. I didn't even know about nix-ninja, sounds like plenty exciting stuff is happening in this space :) | 08:25:17 |
| atemu12 | @jaen:matrix.org I'd highly recommend you try to dyndrv a bunch of stubs to test scaling before you try to make it work with a real build system | 10:09:35 |
| atemu12 | IIRC I did something similar once and discovered sandbox creation overhead to be so massive that this granularity is simply infeasible | 10:10:45 |
| atemu12 | And sandbox creation isn't something you can optimise that much further AFAIK | 10:12:17 |
| atemu12 | (Ah, referring to per-file drvs here.) | 10:13:05 |
| magic_rb | If we could reuse the sandbox, as in cleaning out the build directory ought to be quick, we wouldnt have to set up a new sandbox from scratch | 10:16:34 |
| magic_rb | But idk | 10:16:38 |
| jaen | Yeah, what I was currently "doing" (or rather forcing Claude to do it for me - not going to say sorry for it, the only way I could reliably progress the PoCs between executive dysfunction and $DAY_JOB) and while it works well enough at the scale of Nix, at scale of Chromium and AOSP it buckles under itself in two different ways and the fixes seem to be in opposition to each other - having bigger chunks (so you rebuild more if something changes) vs. nesting more (so you pay more overhead, because you have more derivations). Now that I will hopefully be paid to do this full time, maybe I'll just find something the LLM was too dumb to chance upon. | 10:41:57 |
| jaen | Naively this goes with "bigger chunks" so you need to rebuild more on cache miss, I think. At least I couldn't side-step that so far. Alternatively, bazel seems to have a concept of persistent worker processes (https://bazel.build/remote/persistent), at worst maybe Nix could get something like that? | 10:43:57 |
| magic_rb | i mean not neccessarily | 10:44:15 |
| magic_rb | you can clean the sandbox, but not throw it away | 10:44:21 |
| magic_rb | if you kno youll be building 500 derivations you setup one sandbox and build everything in that sandbox with separate build directories. If i were to guess what takes the most time, its setting up the filesystem tree, but you can do that once with the union of all the closures. | 10:45:13 |
