| 27 Jul 2021 |
 andi- | Yeah, I think thinking of paths and strings in nix in general (?) as byte sequences is probably the best. | 19:23:15 |
 sterni (he/him) | Can an unix path contain a literal '/'? If not, builtins.split should at least always work to separate the path components correctly | 19:29:43 |
| 28 Jul 2021 |
|  stick changed their display name from prusnak to stick. | 15:09:45 |
| 29 Jul 2021 |
 tomberek | can a libstore implementation access or query the local host's store? My intention is query the db for the closest-version available package corresponding to a download request. | 02:55:48 |
 niksnut | A store implementation can do anything in principle. Nix used to have a substituter that computed a shortest sequence of binary deltas to apply to an existing store path, to avoid a full download. | 06:59:02 |
| tomberek | niksnut: i've been playing with a zsync as an extension of the http-binary-cache. What happened with that previous implementation? Not worth it? maintenance cost? bad idea? | 07:00:50 |
| tomberek | * niksnut: i've been playing with a zsync-based extension of the http-binary-cache. What happened with that previous implementation? Not worth it? maintenance cost? bad idea? | 07:01:03 |
| tomberek | the limitation i'm running into right now is that xz doesn't have a "rsyncable" mode the way gzip and zstd does | 07:02:08 |
| niksnut | It depended on the client having a full list of all the NARs / patches available on the server (which used to be how substitution worked, before the binary cache). There's a paper about it: https://edolstra.github.io/pubs/eupfcdm-cbse2005-final.pdf | 07:09:49 |
| tomberek | I was looking at either a zsync or casync mechanic (zsync is less intrusive to the existing cache structure and requires Range requests; casync requires more requests, but also allows for space savings on the server with cross-package dedup). I have a PoC to measure how much this saves and get some metrics. Thanks for the paper, reading.... | 07:15:47 |
 Vladimír Čunát | I'm not sure if the server-side savings will be feasible in practice for cache.nixos.org. I suspect we'll want to keep the full downloads there anyway. | 07:51:30 |
| tomberek | Vladimír Čunát: that's why i'm leaning toward the zsync-style approach where the full-downloads are kept. Though i'm open to thoughts/ideas. | 08:06:51 |
| Vladimír Čunát | Probably. I'd try the schemes on gzip --rsyncable from NAR instead of plain NARs. (Yes, current cache.nixos.org uses .xz which surely won't work well, but perhaps that won't be such a problem to switch.)
It's an earlier idea but I never got to check the effect experimentally. The point is that on typical rebuilds the majority changes in NAR contents is replacement of pairs of nix-store hashes. One might hope that compression schemes like gzip will squash such differences to fewer positions.
| 08:12:12 |
| Vladimír Čunát | * Probably. I'd try the schemes on gzip --rsyncable from NAR instead of plain NARs. (Yes, current cache.nixos.org uses .xz which surely won't work well, but perhaps that won't be such a problem to switch.)
It's an earlier idea but I never got to check the effect experimentally. The point is that on typical rebuilds the majority of changes in NAR contents is replacement of pairs of nix-store hashes. One might hope that compression schemes like gzip will squash such differences to fewer positions.
| 08:12:24 |
| Vladimír Čunát | I also played with ideas around (forward) error correction, but zsync and similar sounded more realistic. | 08:13:56 |
