| 27 Jul 2021 |
 andi- | but that only applies to names of derivations not files within a folder, right | 18:30:29 |
| andi- | * but that only applies to names of derivations not files within a folder, right? | 18:30:31 |
 tomberek | correct, i'd presume file names inside a store-path are not so restricted.... time to test it! | 18:31:12 |
| tomberek | [tom@tsurf:~/temp]$ ls /nix/store/mv1iw4cg27fnavhshiwk642s0dwr047a-unicode/
☺
| 18:32:42 |
| andi- | nix-repl> builtins.readDir ./.
{ "👍" = "regular"; }
| 18:33:08 |
 sterni (he/him) | andi-: at least syntactically paths are quite restricted, i. e. only ascii alpha numeric characters, no spaces etc. but you can always get around that by concatenating a string | 18:33:40 |
| sterni (he/him) | andi-: in that sense you can have arbitrary encoding since nix doesn't care about string encoding as long as they don't contain any NUL bytes | 18:34:28 |
| sterni (he/him) | but valid store paths are an issue of course | 18:34:38 |
| andi- | I am concerned about handling of names in e.g. NAR files but since those names always start with a byte size u64 I guess it could go well most of the time? Perhaps utf-16 could be a challenge if there is a 0 byte inbetween chars.. | 18:34:40 |
| andi- | The string functions (e.g. builtins.split) work on bytes instead of characters so I guess that is one of the limitations. | 18:41:19 |
| sterni (he/him) | not sure what split has to do with paths? | 19:18:37 |
| sterni (he/him) | on unix paths are just byte sequences so there shouldn't be an issue wrt that | 19:19:12 |
| sterni (he/him) | I guess at least as your encoding doesn't allow for anything non ASCII to be mistaken as ASCII | 19:19:27 |
| sterni (he/him) | * I guess at least as your encoding doesn't allow for any (part of a) non ASCII char to be mistaken as ASCII | 19:19:42 |
| andi- | In reply to @sternenseemann:systemli.org
not sure what split has to do with paths? imagine a combination of builtins.readDir and postprocessing the paths | 19:22:10 |
| andi- | Yeah, I think thinking of paths and strings in nix in general (?) as byte sequences is probably the best. | 19:23:15 |
| sterni (he/him) | Can an unix path contain a literal '/'? If not, builtins.split should at least always work to separate the path components correctly | 19:29:43 |
| 28 Jul 2021 |
|  stick changed their display name from prusnak to stick. | 15:09:45 |
| 29 Jul 2021 |
| tomberek | can a libstore implementation access or query the local host's store? My intention is query the db for the closest-version available package corresponding to a download request. | 02:55:48 |
 niksnut | A store implementation can do anything in principle. Nix used to have a substituter that computed a shortest sequence of binary deltas to apply to an existing store path, to avoid a full download. | 06:59:02 |
| tomberek | niksnut: i've been playing with a zsync as an extension of the http-binary-cache. What happened with that previous implementation? Not worth it? maintenance cost? bad idea? | 07:00:50 |
| tomberek | * niksnut: i've been playing with a zsync-based extension of the http-binary-cache. What happened with that previous implementation? Not worth it? maintenance cost? bad idea? | 07:01:03 |
| tomberek | the limitation i'm running into right now is that xz doesn't have a "rsyncable" mode the way gzip and zstd does | 07:02:08 |
| niksnut | It depended on the client having a full list of all the NARs / patches available on the server (which used to be how substitution worked, before the binary cache). There's a paper about it: https://edolstra.github.io/pubs/eupfcdm-cbse2005-final.pdf | 07:09:49 |
| tomberek | I was looking at either a zsync or casync mechanic (zsync is less intrusive to the existing cache structure and requires Range requests; casync requires more requests, but also allows for space savings on the server with cross-package dedup). I have a PoC to measure how much this saves and get some metrics. Thanks for the paper, reading.... | 07:15:47 |
 Vladimír Čunát | I'm not sure if the server-side savings will be feasible in practice for cache.nixos.org. I suspect we'll want to keep the full downloads there anyway. | 07:51:30 |
| tomberek | Vladimír Čunát: that's why i'm leaning toward the zsync-style approach where the full-downloads are kept. Though i'm open to thoughts/ideas. | 08:06:51 |
| Vladimír Čunát | Probably. I'd try the schemes on gzip --rsyncable from NAR instead of plain NARs. (Yes, current cache.nixos.org uses .xz which surely won't work well, but perhaps that won't be such a problem to switch.)
It's an earlier idea but I never got to check the effect experimentally. The point is that on typical rebuilds the majority changes in NAR contents is replacement of pairs of nix-store hashes. One might hope that compression schemes like gzip will squash such differences to fewer positions.
| 08:12:12 |
| Vladimír Čunát | * Probably. I'd try the schemes on gzip --rsyncable from NAR instead of plain NARs. (Yes, current cache.nixos.org uses .xz which surely won't work well, but perhaps that won't be such a problem to switch.)
It's an earlier idea but I never got to check the effect experimentally. The point is that on typical rebuilds the majority of changes in NAR contents is replacement of pairs of nix-store hashes. One might hope that compression schemes like gzip will squash such differences to fewer positions.
| 08:12:24 |
| Vladimír Čunát | I also played with ideas around (forward) error correction, but zsync and similar sounded more realistic. | 08:13:56 |
