Hey there,

I am currently investigating nix assertion errors that consistently appear once in a while in our CI. Only MacOS machines are affected by this and the error persists until the nix store is garbage collected. It also only persists on the same user on the machine.

[...]
using cache entry 'fetchToStore:{"fingerprint":"3f0bf132df94785670cb324fbb759e85ff5ad0b6","method":"nar","name":"source","path":"/","store":"/nix/store"}' -> 'null', '/nix/store/72alw9m62226brv4v4m98fqrk31mlp34-source'
store path cache hit for '«github:runtimeverification/k/3f0bf132df94785670cb324fbb759e85ff5ad0b6?narHash=sha256-SYrGoV4PuOWux7d4nYhluWD515%2B9sh0DjAMcEDbREVM%3D»/nix/store/8ck644nfh5mb5dqkzgvqhfbznsk1n4df-source'
custom debug: storePath='72alw9m62226brv4v4m98fqrk31mlp34-source' originalInput.computeStorePath(*state.store)='8ck644nfh5mb5dqkzgvqhfbznsk1n4df-source'
Assertion failed: (!originalInput.getNarHash() || storePath == originalInput.computeStorePath(*state.store)), function copyInputToStore, file ../flake.cc, line 40.

This is the assertion that is violated though the check has since been moved in master. We currently run nix version 2.31.2.

So essentially the two nix store paths differ due to existing and non-existing submodule directories, without their content and also somewhat their parent directory, see:

% diff -qr expected instead | sort
Only in expected/haskell-backend/src/main: native
Only in expected/llvm-backend/src/main: native
Only in expected/web: k-web-theme

Does anybody see what could be wrong here or where to look for bugs in nix? I will probably continue to debug this, but there's a lot going on in nix that I am not aware of yet.

In reply to @juliankuners:matrix.org
So I guess there must be some code that puts invalid entries into the fetcher cache? And this specifically only happens when the repository contains submodules that are not fetched?

Hmm, considering all the past CI issues, I think I spotted one point in CI logs where the issue starts happening, as it logs the additional unpacking [...] log that later is not printed anymore. Though I cannot confirm whether this is the only place where the issue starts to appear. This code snippet in CI executes nix flake metadata . --refresh. It runs in a checkout without submodules, and I suppose without the empty git submodule directories.

Do you think this could populate the cache with an invalid entry?

In reply to @juliankuners:matrix.org

Hmm, considering all the past CI issues, I think I spotted one point in CI logs where the issue starts happening, as it logs the additional unpacking [...] log that later is not printed anymore. Though I cannot confirm whether this is the only place where the issue starts to appear. This code snippet in CI executes nix flake metadata . --refresh. It runs in a checkout without submodules, and I suppose without the empty git submodule directories.

Do you think this could populate the cache with an invalid entry?

In reply to @juliankuners:matrix.org
As I am thinking about ways to work around this, I feel like all workarounds are very hacky. Is this wrong caching behaviour that I observed covered by the above mentioned PR? Or would it be an on-going issue without clear solution, because submodules are involved? Just wondering whether there's a feasible fix in the meantime or whether it's a bigger scale on-going issue.

In reply to @juliankuners:matrix.org
I created a respective issue, highlighting the I suppose actual root cause, all things considered https://github.com/NixOS/nix/issues/14317