| 29 Jul 2025 |
 Mic92 | * Surprising because the throughput of our current derivation parser is slower than what I expect a JSON parser to be able to pull off. | 20:48:04 |
 John Ericson | Mic92: oh wait tbh I was doing unoptimized builds | 20:51:13 |
| John Ericson | that doesn't mean much | 20:51:19 |
| John Ericson | I opened up https://github.com/NixOS/nix/issues/13573 regardless | 20:57:19 |
| John Ericson | since we can make GC faster by not parsing the entire derivation | 20:57:28 |
| John Ericson | OK optimized before
BM_ParseRealDerivationFile/hello 6814 ns 6791 ns 104081 bytes_per_second=247.735Mi/s
BM_ParseRealDerivationFile/firefox 31529 ns 31420 ns 22281 bytes_per_second=486.429Mi/s
| 21:18:01 |
| John Ericson | BM_ParseRealDerivationFile/hello 6952 ns 6929 ns 102364 bytes_per_second=242.772Mi/s
BM_ParseRealDerivationFile/firefox 71569 ns 71292 ns 9695 bytes_per_second=214.381Mi
| 21:18:27 |
| John Ericson | optimized after | 21:18:29 |
| John Ericson | 4x to 2x | 21:18:39 |
| John Ericson | not good enough | 21:18:42 |
| Mic92 | In reply to @Ericson2314:matrix.org
I opened up https://github.com/NixOS/nix/issues/13573 regardless Is it not all the store references we care about in those derivations, not just outpaths? | 21:22:57 |
| John Ericson | Mic92: I think it is just looking to keep derivations pointing to live store paths? | 21:23:25 |
| John Ericson | I could be wrong | 21:23:33 |
| Mic92 | https://github.com/NixOS/nix/blob/b062730665d120088ab6c3e154b4545f95d2fa4b/src/libstore/gc.cc#L767 | 21:26:42 |
| Mic92 | Ah but wait this for keepOutputs | 21:31:24 |
| Mic92 | This is not the same as keep derivations, no? | 21:31:55 |
| John Ericson | Mic92: I am talking about the queryOutput stuff in there | 21:35:18 |
| John Ericson | Mic92: oh wow, it's already done via the database in the local store case! | 21:37:58 |
| John Ericson | see queryStaticPartialDerivationOutputMap | 21:38:03 |
| John Ericson | https://github.com/NixOS/nix/issues/13573#issuecomment-3134157257 | 21:40:17 |
| John Ericson | so....why the hell is it so slow? | 21:40:23 |
| John Ericson | SQLite should be fast enough | 21:40:29 |
| John Ericson | (slow based on your said your sysadmining experience, Mic92) | 21:40:42 |
| John Ericson | Eelco: you inadvertently trolled us quite well, because you in 2010 https://github.com/NixOS/nix/commit/c4d388add4942f6f99a8df12f4e49149005047e2 fixed the perf issue you brought up in my PR :) | 21:47:11 |
| Mic92 | In reply to @Ericson2314:matrix.org
(slow based on your said your sysadmining experience, Mic92) It's parsing all the derivation files during GC by default. You have to turn of keep-derivations to disable it | 21:52:45 |
| Mic92 | If it's turned off it's really just sqlite and fast | 21:54:10 |
| John Ericson | Mic92: hmm? because of what? | 21:55:20 |
| Mic92 | In reply to @Ericson2314:matrix.org
Mic92: hmm? because of what? Can't point you to the concrete code but it's very obvious if you strace | 21:56:48 |
| Mic92 | Something else we should consider is doing VACUUM after GC to restore sqlite size. But I have to experiment a bit with that. Can be tricky if the disk is almost full. | 21:57:48 |
| John Ericson | hmmm OK | 21:58:06 |
