| 5 Dec 2025 |
|  adam changed their profile picture. | 16:26:10 |
 raitobezarius | yeah this was the joke here | 16:38:28 |
 just1602 | But did some of the fork of sqlite like libsql or turso could fix the issue we face with sqlite? | 16:40:42 |
| adam changed their profile picture. | 16:43:41 |
|  đ@crystallinefire:chat.solarpunk.moe changed their display name from EVA-01 to đ@crystallinefire:chat.solarpunk.moe. | 17:17:17 |
 aloisw | I would rather trust SQLite than a VC-backed fork plastering "AI" over their website to stay around. | 17:52:46 |
| aloisw | And I think it's helpful to figure out what our actual issues with SQLite are. "The WAL grows too large" is a <10 line fix, but of course I don't know yet if this is the only problem. | 17:54:24 |
| aloisw | Update: RESTART checkpoints with 40000 pages tank the speed even more, since checkpoints happen very often due to nix-eval-jobs growing the WAL so fast. | 18:36:17 |
| raitobezarius | hah | 18:47:21 |
| aloisw | Yeah the WAL just grows so much faster than the database, even with large WAL sizes checkpointing takes quite long. | 19:01:35 |
| aloisw | Page size of 512, WAL size of 1 GiB (lower should also work, only increasing the fsync cost) and RESTART checkpointing looks quite nice with the synthetic test case. There are still stalls, but they are like 7 seconds after 2 minutes. | 19:30:36 |
| 6 Dec 2025 |
 Jassuko | Hmmm. The database itself should not be very big, right? What is the consistency requirement for it? I.e. do we really need to care if the system crashes mid-build somewhere as long as the final state or such gets committed properly? | 10:50:42 |
| Jassuko | So, do we a actually need WAL, or would it better to just use the restore based journaling? Or if the intermediate steps are not that important to keep as we can just run the builds again after a crash, we could just copy the whole database on tempfs and run the restore checkpointed mode on there where latency on additional file operations would be minimal. And after some things or the whole operation is done, only then copy the relevant state to the db store on disk? | 10:53:56 |
 K900 | SQLite can only do multiple concurrent readers in WAL mode | 11:28:30 |
 522 it/its âŻÎÎ | wait, really? i thought WAL provided one writer and multiple readers, as opposed to one writer or multiple readers | 11:41:46 |
| 522 it/its âŻÎÎ | that's what https://sqlite.org/wal.html implies | 11:42:10 |
 Martin Häcker | Hey, we had a discussion in the bugtracker about adding nix develop âprint-phases which was rejected on the grounds that nixpkgs would need to provide the infra for that. I provided a pull request to nixpkgs to provide that which got favorable reviews, but could perhaps use one more - or someone who dares merge it (as it will trigger a rebuild of everything).
I was hoping there are some people here on the channel wo think this useful and would be willing to help?
| 11:42:41 |
| 522 it/its âŻÎÎ | wal is in general faster though yeah | 11:43:17 |
| raitobezarius | In reply to @dont.wanna.tell:matrix.org
Hey, we had a discussion in the bugtracker about adding nix develop âprint-phases which was rejected on the grounds that nixpkgs would need to provide the infra for that. I provided a pull request to nixpkgs to provide that which got favorable reviews, but could perhaps use one more - or someone who dares merge it (as it will trigger a rebuild of everything).
I was hoping there are some people here on the channel wo think this useful and would be willing to help?
If all comments are resolved, I will merge it for you :) | 12:10:49 |
| aloisw | I expect rollback journal to be disastrously slow with the Lix store database, as it does at least one fsync per commit. You can try out yourself with the use-sqlite-wal setting if you want. | 12:14:13 |
| aloisw | We're already running in NORMAL synchronous mode, which sacrifices durability (which is in most cases, including Lix, not a big deal). Turning it off completely can (on most stock filesystems) give you a corrupted database after unclean shutdown which is absolutely not what you want. | 12:16:03 |
| aloisw | Yes, this is correct. Rollback journal allows any number of readers but the writer needs exclusive access. | 12:16:40 |
| aloisw | raitobezarius could you test https://gerrit.lix.systems/c/lix/+/4712 in your real-world setting? Be advised that if you use the daemon, the change will also be needed there, and for whatever reason there may happen a weird deadlock during the generation switch (I need to debug that). | 12:37:02 |
| raitobezarius | will do! | 12:58:29 |
| Martin Häcker | I think they are, are you missing anyhting? | 13:57:18 |
| Martin Häcker | * | 13:57:43 |
| raitobezarius | In reply to @dont.wanna.tell:matrix.org
I think they are, am I missing anything? https://github.com/NixOS/nixpkgs/pull/441897#discussion_r2343911652 ? | 14:08:11 |
| Martin Häcker | ah; I should have marked that as resolved, he did agree that minimal churn (especially in high stakes code like this) is an equally valuable goal. | 14:16:49 |
| Jassuko | In reply to @raitobezarius:matrix.org
It's the metadata layer of the Nix store, the actual source of truth How is this handled currently? And how big of a deal is rebuilding said metadata if everything goes wrong?
Iâm trying to figure out what are the a actual requirements for the consistency of this DB. In some cases it might make sense to provide fast âhappy pathâ usage in expense of having to do considerably more costly repair operation in case of total failure. | 14:34:15 |
| aloisw | This database is extremely important, if things go wrong you can of course still access the store paths that were already there as files so the system is not immediately broken, but from the Lix perspective the entire store might as well be trash. | 14:35:23 |
