| 1 Jul 2025 |
 magic_rb | Okay im lost lol | 11:45:17 |
| magic_rb | We do have to copy for a build still no? | 11:45:50 |
 Robert Hensing (roberth) | So we have the sources accessor objects that behave like very simple virtual file systems, and we plan to use them directly instead of copying everything to the store all the time | 11:46:11 |
| Robert Hensing (roberth) | They can implement operations like readDirectory or readFile as they please, so the git accessor with annex enabled could add a .git and .git/annex to what it returns, and then do whatever is necessary to return the contents of that | 11:47:31 |
| Robert Hensing (roberth) | Yes, but only the things you bring into the derivation. You could use a source filter to avoid some unneeded stuff.
Currently that's all moot because fetchTree copies everything it could return, but we'll change that, and make source filtering a solution for this problem. | 11:49:19 |
| magic_rb | In reply to @roberthensing:matrix.org
Yes, but only the things you bring into the derivation. You could use a source filter to avoid some unneeded stuff.
Currently that's all moot because fetchTree copies everything it could return, but we'll change that, and make source filtering a solution for this problem. Right which is why im saying that i think each annexed file should become its own store path, so that youre not copying about 40GB for each build | 11:50:14 |
| Robert Hensing (roberth) | If it's not clear which is better, we could make this behavior configurable. Making the right parts of .git/annex available to derivations would be a pain. | 11:50:38 |
| magic_rb | Say im working on a game, then to nix build it, i need essentially all the annexed files, so the for every build im copying all the assets which can be arbitrarily huge | 11:51:13 |
| Robert Hensing (roberth) | That's designing for the current Nix, not the Nix we're promising, fwiw | 11:51:33 |
| magic_rb | Even with lazy tree, if im using 40GB, i have to copy 40GB every time no? | 11:52:15 |
| magic_rb | Thats how i understood it | 11:52:25 |
| Robert Hensing (roberth) | So I guess we have three possible behaviors:
- deference it completely and return the file contents
- put it in the store and return store references somehow (more complicated than it seems, but potentially lazy under more circumstances)
- generate a
.git/annex which only has the relevant entries (probably annoying to work with, unless your build makes assumptions about .git/annex)
| 11:53:59 |
| Robert Hensing (roberth) | Only if you dereference every annexed file | 11:54:31 |
| Robert Hensing (roberth) | To make fetching the annexed files lazy involves:
- changing fetchTree to not copy everything to the store, returning a path value instead of a store path (or a lazy-trees store path that produces the correct hash part, which has never been done)
- implementing the annex support in an "event based" manner - not eagerly fetching everything
| 11:58:34 |
