there's discussion of this over at https://github.com/facebook/zstd/issues/2949 for zstd

zstd is deterministic... given the same version
which for the sake of reproducible builds isn't an issue, you can just use the older version

reproducible across all versions is quite a niche requirement, especially since that means you can't ever make the compression more efficient space-wise

for that i guess you'd just pick a given zstd(or other library) version and fork it, applying changes as needed, but having tests that ensure you never break reproducability

In reply to @pveierland:matrix.org
It was just something I was thinking about since I was doing some stuff with binary build caches. E.g. nix binary caches sets the file name of a compressed NAR based on the file hash of the compressed NAR, meaning that when the compression is not reproducible, you cannot directly infer the location of the data, but need to depend on metadata

If you want to associate a computation with its outcome, but not run the computation yourself, you always need to rely on some metadata for that association.

That's not only true for the kind of compression you describe, but also for the build steps themselves, even with content-addressed derivations in Nix.

In reply to @mschwaig:matrix.org

If you want to associate a computation with its outcome, but not run the computation yourself, you always need to rely on some metadata for that association.

That's not only true for the kind of compression you describe, but also for the build steps themselves, even with content-addressed derivations in Nix.