| 6 Dec 2025 |
 aloisw | I guess you could point it to a store on tmpfs and it at least wouldn't copy to the disk. | 14:50:39 |
 kuruczgy | (In particular I probably want to avoid copying anything to my store, I have a complicated FOD to put together the source tree, and I want to check that it comes out identical to what I have checked out in my working tree.) | 14:52:08 |
| aloisw | Right, but "100s to 1000s of writes per second" is pretty much exactly what Lix does. | 14:52:12 |
 522 it/its ⛯ΘΔ | in a transaction per write? | 14:55:01 |
| aloisw | No, one transaction per derivation I think. | 14:55:38 |
 KFears (burnt out) | That's on a very high load, no? My desktop closure size is around 17k drvs, and another 17k for FODs | 14:55:44 |
| aloisw | For reference, evaluating my system configuration into a fresh store takes about 85 seconds including downloads and builds for IFD, and creates about 350k pages of WAL (with page_size = 512, haven't checked with 4096 but probably about the same). | 14:56:50 |
 hexa | loving the implications for build farms | 14:57:30 |
| aloisw | Yes, that's exactly why it is so slow and we're having this conversation. | 14:57:32 |
| hexa | thanks for looking int o that | 14:57:36 |
| hexa | * thanks for looking into that | 14:57:39 |
| aloisw | You eval on tmpfs, right? | 14:58:07 |
| hexa | not sure what "eval on tmpfs" means | 14:59:04 |
| aloisw | That the store that the machine performing the evaluation writes to is located on tmpfs. | 14:59:39 |
| hexa | oh, you mean a non-default store then | 15:00:05 |
| hexa | no | 15:00:19 |
| hexa | * that'd be a no | 15:00:24 |
| KFears (burnt out) | In reply to @aloisw:julia0815.de
Yes, that's exactly why it is so slow and we're having this conversation. I must say, I'm very surprised SQLite suffocates under that load | 15:09:32 |
| aloisw | Ah lovely, when you try to do PRAGMA journal_mode = TRUNCATE on a WAL database concurrently it may just deadlock. | 15:23:16 |
| aloisw | Of course, as they open a connection with WAL first, but you obviously can't have a WAL connection while trying to convert the database to truncate mode. In particular once you're in the deadlock state, no amount of retry within the connection (as Lix does) will try to help you. | 15:27:51 |
 raitobezarius | In reply to @hexa:lossy.network
oh, you mean a non-default store then Technically /Nix can be tmpfs | 15:56:42 |
 ShalokShalom | In reply to @aloisw:julia0815.de
Yes, that's exactly why it is so slow and we're having this conversation. Isn't there somebody rewriting SQLite in Rust? | 16:30:32 |
| ShalokShalom | Could that make a difference | 16:30:42 |
 K900 | Rust is not magic | 16:31:40 |
| K900 | And I'm not aware of a Rust API compatible SQLite replacement | 16:31:56 |
| aloisw | Yes, a VC-backed company plastering "AI" all over their website called Turso. It was already suggested above. | 16:37:23 |
| aloisw | In any case Rust is not going to solve write amplification. | 16:37:36 |
| aloisw | diff --git a/overlay.nix b/overlay.nix
--- a/overlay.nix
+++ b/overlay.nix
@@ -26,7 +26,7 @@ meowLib.runOnDirectory (path: final.call
_finalLixes: prevLixes: {
git = prevLixes.git.overrideScope (
_finalLix: prevLix: {
- lix = prevLix.lix.overrideAttrs (old: {
+ lix = (prevLix.lix.override { sqlite = final.sqlite-wal2; }).overrideAttrs (old: {
version = "2.95.0-pre-${builtins.substring 0 12 source.lix.rev}";
src = source.lix;
patches = old.patches or [ ] ++ meowLib.collectPatches ./patches/lix;
diff --git a/patches/lix/wal2.diff b/patches/lix/wal2.diff
new file mode 100644
--- /dev/null
+++ b/patches/lix/wal2.diff
@@ -0,0 +1,13 @@
+diff --git a/lix/libstore/sqlite.cc b/lix/libstore/sqlite.cc
+index fe606bce20..81f71a8b9f 100644
+--- a/lix/libstore/sqlite.cc
++++ b/lix/libstore/sqlite.cc
+@@ -134,7 +134,7 @@
+ // Set the SQLite journal mode.
+ // WAL mode is fastest, so it's the default.
+ if (settings.useSQLiteWAL) {
+- exec("pragma main.journal_mode = wal", always_progresses);
++ exec("pragma main.journal_mode = wal2", always_progresses);
+ /* persist the WAL files when the DB connection is closed.
+ * This allows for read-only connections without any write permissions
+ * on the state directory to succeed on a closed database. Setting the
diff --git a/pkgs/sqlite-wal2/package.nix b/pkgs/sqlite-wal2/package.nix
new file mode 100644
--- /dev/null
+++ b/pkgs/sqlite-wal2/package.nix
@@ -0,0 +1,22 @@
+# SPDX-FileCopyrightText: Alois Wohlschlager <alois1@gmx-topmail.de>
+# SPDX-License-Identifier: MIT
+{
+ stdenv,
+ fetchurl,
+ readline,
+}:
+stdenv.mkDerivation {
+ pname = "sqlite-wal2";
+ version = "3.52.0-unstable-2025-11-28";
+
+ src = fetchurl {
+ url = "https://sqlite.org/src/tarball/sqlite-20251128204824-18d8e307df.tar.gz";
+ hash = "sha256-ELgz1zlIljJWFhunjixDjS2yeYGHjVl3m12v4XFKruU=";
+ };
+
+ strictDeps = true;
+
+ buildInputs = [ readline ];
+
+ configureFlags = [ "--with-readline-header=${readline.dev}/include/readline/readline.h" ];
+}
I wonder if I will regret this… | 19:43:02 |
| raitobezarius | :D | 19:43:36 |
| aloisw | Great, the WAL hook is not called at all… | 20:05:11 |
