| 1 Jul 2025 |
 magic_rb | If the annexed file isnt in the local annex store, the symlink will be dangling and then nix should throw an error ideally. To make it not do that one much do git annex get path/to/symlink which will copy the file into the local annex | 11:40:47 |
 roberth | I started with this very high level description here, which suggests that the key can be anything | 11:40:52 |
| roberth | if it's always sha256, or at least in practice, we can rely on that property to do the same thing we do for LFS | 11:41:12 |
| roberth | (or similar if we want it to be a symlink) | 11:41:27 |
| magic_rb | Oh yeah okay backends | 11:41:45 |
| magic_rb | Damn didnt know there was that too | 11:41:54 |
| magic_rb | We could enforce a blessed set of backends yeah | 11:42:08 |
| roberth | Partial support can be great if virtually everyone uses the SHA256 stuff | 11:42:15 |
| roberth | yea | 11:42:19 |
| magic_rb | If its blake2 cant we sha256 hash the blake2 hash, or is that a horrible idea | 11:42:45 |
| roberth | Probably a symlink to a synthetic .git/annex directory is preferable | 11:42:52 |
| magic_rb | In reply to @roberthensing:matrix.org
Probably a symlink to a synthetic .git/annex directory is preferable Id say that each annexed object should be a separate store path | 11:43:15 |
| magic_rb | You dont want to be copying 40GB every time a single image changes | 11:43:25 |
| roberth | True, but we shouldn't be copying those store paths in the first place | 11:43:50 |
| roberth | And if we do, they should match the non-Nix layouts | 11:44:02 |
| roberth | Currently there's not much of a connection between libfetchers fetchers and the store layer, so putting it in the store is more complicated than putting it in .git/annex, fwiw | 11:44:27 |
| magic_rb | Wait so we just tell nix to include .git/annex during eval? | 11:44:57 |
| roberth | (copying to the store only happens at the end of fetchTree and not within the individual source accessors) | 11:45:02 |
| magic_rb | Okay im lost lol | 11:45:17 |
| magic_rb | We do have to copy for a build still no? | 11:45:50 |
| roberth | So we have the sources accessor objects that behave like very simple virtual file systems, and we plan to use them directly instead of copying everything to the store all the time | 11:46:11 |
| roberth | They can implement operations like readDirectory or readFile as they please, so the git accessor with annex enabled could add a .git and .git/annex to what it returns, and then do whatever is necessary to return the contents of that | 11:47:31 |
| roberth | Yes, but only the things you bring into the derivation. You could use a source filter to avoid some unneeded stuff.
Currently that's all moot because fetchTree copies everything it could return, but we'll change that, and make source filtering a solution for this problem. | 11:49:19 |
| magic_rb | In reply to @roberthensing:matrix.org
Yes, but only the things you bring into the derivation. You could use a source filter to avoid some unneeded stuff.
Currently that's all moot because fetchTree copies everything it could return, but we'll change that, and make source filtering a solution for this problem. Right which is why im saying that i think each annexed file should become its own store path, so that youre not copying about 40GB for each build | 11:50:14 |
| roberth | If it's not clear which is better, we could make this behavior configurable. Making the right parts of .git/annex available to derivations would be a pain. | 11:50:38 |
| magic_rb | Say im working on a game, then to nix build it, i need essentially all the annexed files, so the for every build im copying all the assets which can be arbitrarily huge | 11:51:13 |
| roberth | That's designing for the current Nix, not the Nix we're promising, fwiw | 11:51:33 |
| magic_rb | Even with lazy tree, if im using 40GB, i have to copy 40GB every time no? | 11:52:15 |
| magic_rb | Thats how i understood it | 11:52:25 |
| roberth | So I guess we have three possible behaviors:
- deference it completely and return the file contents
- put it in the store and return store references somehow (more complicated than it seems, but potentially lazy under more circumstances)
- generate a
.git/annex which only has the relevant entries (probably annoying to work with, unless your build makes assumptions about .git/annex)
| 11:53:59 |
