!VRULIdgoKmKPzJZzjj:nixos.org

Nix Hackers

845 Members
For people hacking on the Nix package manager itself186 Servers

Load older messages


SenderMessageTime
1 Jul 2025
@roberthensing:matrix.orgroberth Yes, but only the things you bring into the derivation. You could use a source filter to avoid some unneeded stuff.
Currently that's all moot because fetchTree copies everything it could return, but we'll change that, and make source filtering a solution for this problem.
11:49:19
@magic_rb:matrix.redalder.orgmagic_rb
In reply to @roberthensing:matrix.org
Yes, but only the things you bring into the derivation. You could use a source filter to avoid some unneeded stuff.
Currently that's all moot because fetchTree copies everything it could return, but we'll change that, and make source filtering a solution for this problem.
Right which is why im saying that i think each annexed file should become its own store path, so that youre not copying about 40GB for each build
11:50:14
@roberthensing:matrix.orgroberth If it's not clear which is better, we could make this behavior configurable. Making the right parts of .git/annex available to derivations would be a pain. 11:50:38
@magic_rb:matrix.redalder.orgmagic_rbSay im working on a game, then to nix build it, i need essentially all the annexed files, so the for every build im copying all the assets which can be arbitrarily huge11:51:13
@roberthensing:matrix.orgroberthThat's designing for the current Nix, not the Nix we're promising, fwiw11:51:33
@magic_rb:matrix.redalder.orgmagic_rbEven with lazy tree, if im using 40GB, i have to copy 40GB every time no?11:52:15
@magic_rb:matrix.redalder.orgmagic_rbThats how i understood it11:52:25
@roberthensing:matrix.orgroberth

So I guess we have three possible behaviors:

  • deference it completely and return the file contents
  • put it in the store and return store references somehow (more complicated than it seems, but potentially lazy under more circumstances)
  • generate a .git/annex which only has the relevant entries (probably annoying to work with, unless your build makes assumptions about .git/annex)
11:53:59
@roberthensing:matrix.orgroberthOnly if you dereference every annexed file11:54:31
@roberthensing:matrix.orgroberth

To make fetching the annexed files lazy involves:

  1. changing fetchTree to not copy everything to the store, returning a path value instead of a store path (or a lazy-trees store path that produces the correct hash part, which has never been done)
  2. implementing the annex support in an "event based" manner - not eagerly fetching everything
11:58:34
@emilazy:matrix.orgemilyif the build of a game requires processing every source asset – which I believe is normal – then this does not stop copying every asset anew for every build, even if lazy trees were fully implemented, right?11:59:20
@emilazy:matrix.orgemily(maybe you can cut down on it if you can split it into one asset build per derivation since the store paths would remain the same if the individual asset doesn't change?)11:59:55
@emilazy:matrix.orgemily(and I guess integration could avoid re-hashing the file to determine that?)12:00:03
@emilazy:matrix.orgemily just checking I understand correctly that if all the assets are needed in the same derivation as part of a src = ./.; type thing it would still amount to copying the entire thing every time 12:00:24
@magic_rb:matrix.redalder.orgmagic_rb Yeah this is what im saying. Say youre building a godot game, you end up with src = ./., you cant do much with uh, whats was it filterTree 12:03:33
@roberthensing:matrix.orgroberthThe store references choice does make that easier to achieve. Otherwise, we're looking representations of store objects that don't reside in a real filesystem but in a FUSE store, and an underlying ca-store like tvix/snix. That would be great to have, but it's a lot of work12:03:44
@emilazy:matrix.orgemilyI think the practical solution is just to poke holes in the sandbox to access the underlying content-addressed annex store12:05:47
@emilazy:matrix.orgemilyand rewrite symlinks to point to it before you run your build12:05:56
@roberthensing:matrix.orgroberth Right, so to clarify, if your main goal is to distribute a built game, you should not use builtin fetchers for anything but Nix expressions, because those will be intermediate results, and only in case of FODs it's possible to optimize those away by virtue of the dependent being available and having no reference to the source 12:05:59
@roberthensing:matrix.orgroberthSo what we're left with is the "developer" use case12:06:15
@emilazy:matrix.orgemilyit is "reasonably" pure since things are still addressed by hash but it seems to me the only way to get builds involving >40 GiB of assets without years of experimental work on the Nix end12:06:25
@roberthensing:matrix.orgroberthFor development, presumably you want a quick build, so making asset processing fine grained with a derivation per asset would be the thing to do12:06:59
@magic_rb:matrix.redalder.orgmagic_rbTo clarify, i dont have an immediate use case for this. It would be nice, but so far i dont have a game id be developing lol12:07:17
@sinan:sinanmohd.comsinan changed their profile picture.12:07:52
@magic_rb:matrix.redalder.orgmagic_rb So this was more something i wanted to look at personally, but it seems i severely underestimated the complexity 12:09:11
@roberthensing:matrix.orgroberthIf you're ok with some eagerness for now, the complexity isn't so bad12:09:57
@roberthensing:matrix.orgroberthThis one's the easiest to implement, and it can follow the LFS pattern12:10:20
@magic_rb:matrix.redalder.orgmagic_rbHonestly even the eager version would be nice, currently it just leaves a bunch of dangling symlinks12:10:45
@magic_rb:matrix.redalder.orgmagic_rbSo option one, "deference it completely and return the file contents"12:11:07
@roberthensing:matrix.orgroberthyep12:11:16

Show newer messages


Back to Room ListRoom Version: 6