| 21 Oct 2025 |
 Julian | Sergei Zimmerman (xokdvium): What specifically do you mean with submodules enabled/disabled? And which cache specifically? | 13:57:38 |
 Sergei Zimmerman (xokdvium) | The fetcher cache. | 13:58:44 |
| Julian | Sergei Zimmerman (xokdvium): Only the revision with no empty submodule directories is in the fetcher cache fetcher-cache-v4.sqlite. For the build to work, the store path should include empty submodule directories (or atleast for hashes to match). | 14:15:12 |
| Julian | So I guess there must be some code that puts invalid entries into the fetcher cache? And this specifically only happens when the repository contains submodules that are not fetched? | 14:16:40 |
| Sergei Zimmerman (xokdvium) | In reply to @juliankuners:matrix.org
So I guess there must be some code that puts invalid entries into the fetcher cache? And this specifically only happens when the repository contains submodules that are not fetched? When submodules are fetched it should not try to use the fetcher cache at all. That’s the issue I think. There’s some logic for it there, but it must have been broken somehow | 14:17:59 |
| Sergei Zimmerman (xokdvium) | (As in, it should not use the fetchToStore cache entry) | 14:18:27 |
| Sergei Zimmerman (xokdvium) | Because submodules are wonky and caching them correctly in a way that the narHash stays the same is hard | 14:19:20 |
| Sergei Zimmerman (xokdvium) | This must have surfaced because fetchToStore caching was broken for a while and only got fixed recently in https://github.com/NixOS/nix/pull/14050 | 14:20:54 |
| Julian | Hmm, considering all the past CI issues, I think I spotted one point in CI logs where the issue starts happening, as it logs the additional unpacking [...] log that later is not printed anymore. Though I cannot confirm whether this is the only place where the issue starts to appear. This code snippet in CI executes nix flake metadata . --refresh. It runs in a checkout without submodules, and I suppose without the empty git submodule directories.
Do you think this could populate the cache with an invalid entry?
| 14:24:42 |
| Julian | Thank you for these thoughts and hints. I guess I at least figured out the root cause now with this. | 14:25:52 |
| Sergei Zimmerman (xokdvium) | In reply to @juliankuners:matrix.org
Hmm, considering all the past CI issues, I think I spotted one point in CI logs where the issue starts happening, as it logs the additional unpacking [...] log that later is not printed anymore. Though I cannot confirm whether this is the only place where the issue starts to appear. This code snippet in CI executes nix flake metadata . --refresh. It runs in a checkout without submodules, and I suppose without the empty git submodule directories.
Do you think this could populate the cache with an invalid entry?
Hm, a local checkout might do that yeah. The git fetcher might consider it to be cacheable and doesn’t take into account empty subdirs | 14:27:23 |
| Julian | As I am thinking about ways to work around this, I feel like all workarounds are very hacky. Is this wrong caching behaviour that I observed covered by the above mentioned PR? Or would it be an on-going issue without clear solution, because submodules are involved? Just wondering whether there's a feasible fix in the meantime or whether it's a bigger scale on-going issue. | 14:37:45 |
| Sergei Zimmerman (xokdvium) | In reply to @juliankuners:matrix.org
As I am thinking about ways to work around this, I feel like all workarounds are very hacky. Is this wrong caching behaviour that I observed covered by the above mentioned PR? Or would it be an on-going issue without clear solution, because submodules are involved? Just wondering whether there's a feasible fix in the meantime or whether it's a bigger scale on-going issue. Reasonably we could just avoid caching the fetchToStore entry if submodules are present in the local checkout. A more clever solution would be to filter out the empty subdirs for submodules when fetching to the store from a local checkout. The first solution seems more immediate and backportable. Submodules + caching is definitely an ongoing issue | 14:45:20 |
| Julian | Re-looking at the CI logs, it's probably not immediately the checkout problem, as it the mis-cached repository is another repository and not the ceckout (that is also not a submodule or anything) :/. Sorry for the confusion. | 14:48:26 |
| Julian | For clarification. The repositories have submodules, but none are used for nix flakes and derivations. | 14:49:13 |
| Sergei Zimmerman (xokdvium) | I think I understand what’s going on. Will poke around soonish and see | 15:13:25 |
| Julian | Yea, me somwhat too. It is actually the checkout issue, but it's caused by a checkout of the mis-cached repository in another CI on the same CI machine, see here. The timestamp of the CI job and the cache entry in the sqlite database match exactly, same as the wrong nix store path 72alw9m62226brv4v4m98fqrk31mlp34. | 15:21:07 |
| Julian | I created a respective issue, highlighting the I suppose actual root cause, all things considered https://github.com/NixOS/nix/issues/14317 | 17:03:51 |
 dramforever | In reply to @juliankuners:matrix.org
I created a respective issue, highlighting the I suppose actual root cause, all things considered https://github.com/NixOS/nix/issues/14317 oh yeah that's a duplicate of https://github.com/NixOS/nix/issues/13698 | 17:13:40 |
| dramforever | guess i was a few hours late to the discussion | 17:15:47 |
| Julian | Thank you, I missed this one during my issue search. I closed it and linked to it in the respective other issue. | 17:19:57 |
|  @echobc:matrix.org joined the room. | 18:09:44 |
|  mjolnir banned @echobc:matrix.org (<no reason supplied>). | 18:09:44 |
 niksnut | learned today that using std::string for large buffers is very inefficient (huge kernel overhead): https://github.com/DeterminateSystems/nix-src/pull/238/commits/edf45d6e1158a059e7ded5460d2a3947dc35fdf8 | 20:44:48 |
| Sergei Zimmerman (xokdvium) | This is sort of more about doing unnecessary construction/destruction of an object. Not special to std::string in any way | 20:55:40 |
| niksnut | it's a result of having a large contiguous allocation (so it would also affect std::vector<char>) | 20:56:21 |
| Sergei Zimmerman (xokdvium) | Yeah, it's much better to allocate once and reuse that allocation as you did there | 20:56:45 |
| niksnut | a data type that consists of a vector of buffers would avoid this problem | 20:57:06 |
| Sergei Zimmerman (xokdvium) | I did a similar cleanup in filetransfer..cc at some point | 20:57:07 |
| Sergei Zimmerman (xokdvium) | You'd still benefit from allocating it only once though | 20:57:23 |
