| 1 Jul 2025 |
 magic_rb | Damn didnt know there was that too | 11:41:54 |
| magic_rb | We could enforce a blessed set of backends yeah | 11:42:08 |
 Robert Hensing (roberth) | Partial support can be great if virtually everyone uses the SHA256 stuff | 11:42:15 |
| Robert Hensing (roberth) | yea | 11:42:19 |
| magic_rb | If its blake2 cant we sha256 hash the blake2 hash, or is that a horrible idea | 11:42:45 |
| Robert Hensing (roberth) | Probably a symlink to a synthetic .git/annex directory is preferable | 11:42:52 |
| magic_rb | In reply to @roberthensing:matrix.org
Probably a symlink to a synthetic .git/annex directory is preferable Id say that each annexed object should be a separate store path | 11:43:15 |
| magic_rb | You dont want to be copying 40GB every time a single image changes | 11:43:25 |
| Robert Hensing (roberth) | True, but we shouldn't be copying those store paths in the first place | 11:43:50 |
| Robert Hensing (roberth) | And if we do, they should match the non-Nix layouts | 11:44:02 |
| Robert Hensing (roberth) | Currently there's not much of a connection between libfetchers fetchers and the store layer, so putting it in the store is more complicated than putting it in .git/annex, fwiw | 11:44:27 |
| magic_rb | Wait so we just tell nix to include .git/annex during eval? | 11:44:57 |
| Robert Hensing (roberth) | (copying to the store only happens at the end of fetchTree and not within the individual source accessors) | 11:45:02 |
| magic_rb | Okay im lost lol | 11:45:17 |
| magic_rb | We do have to copy for a build still no? | 11:45:50 |
| Robert Hensing (roberth) | So we have the sources accessor objects that behave like very simple virtual file systems, and we plan to use them directly instead of copying everything to the store all the time | 11:46:11 |
| Robert Hensing (roberth) | They can implement operations like readDirectory or readFile as they please, so the git accessor with annex enabled could add a .git and .git/annex to what it returns, and then do whatever is necessary to return the contents of that | 11:47:31 |
| Robert Hensing (roberth) | Yes, but only the things you bring into the derivation. You could use a source filter to avoid some unneeded stuff.
Currently that's all moot because fetchTree copies everything it could return, but we'll change that, and make source filtering a solution for this problem. | 11:49:19 |
| magic_rb | In reply to @roberthensing:matrix.org
Yes, but only the things you bring into the derivation. You could use a source filter to avoid some unneeded stuff.
Currently that's all moot because fetchTree copies everything it could return, but we'll change that, and make source filtering a solution for this problem. Right which is why im saying that i think each annexed file should become its own store path, so that youre not copying about 40GB for each build | 11:50:14 |
| Robert Hensing (roberth) | If it's not clear which is better, we could make this behavior configurable. Making the right parts of .git/annex available to derivations would be a pain. | 11:50:38 |
| magic_rb | Say im working on a game, then to nix build it, i need essentially all the annexed files, so the for every build im copying all the assets which can be arbitrarily huge | 11:51:13 |
| Robert Hensing (roberth) | That's designing for the current Nix, not the Nix we're promising, fwiw | 11:51:33 |
| magic_rb | Even with lazy tree, if im using 40GB, i have to copy 40GB every time no? | 11:52:15 |
| magic_rb | Thats how i understood it | 11:52:25 |
| Robert Hensing (roberth) | So I guess we have three possible behaviors:
- deference it completely and return the file contents
- put it in the store and return store references somehow (more complicated than it seems, but potentially lazy under more circumstances)
- generate a
.git/annex which only has the relevant entries (probably annoying to work with, unless your build makes assumptions about .git/annex)
| 11:53:59 |
| Robert Hensing (roberth) | Only if you dereference every annexed file | 11:54:31 |
| Robert Hensing (roberth) | To make fetching the annexed files lazy involves:
- changing fetchTree to not copy everything to the store, returning a path value instead of a store path (or a lazy-trees store path that produces the correct hash part, which has never been done)
- implementing the annex support in an "event based" manner - not eagerly fetching everything
| 11:58:34 |
 emily | if the build of a game requires processing every source asset – which I believe is normal – then this does not stop copying every asset anew for every build, even if lazy trees were fully implemented, right? | 11:59:20 |
