| 9 Jul 2025 |
 Grimmauld (any/all) | yep | 17:52:02 |
| Grimmauld (any/all) | does help too. Time to benchmark! | 18:00:05 |
| Grimmauld (any/all) | i do think its now the sqlite write thats blocking, because we are opening a new db connection for each line of nej | 18:00:53 |
| Grimmauld (any/all) | that feels slow, but i'll need to do more testing | 18:01:32 |
 K900 | No we're not? | 18:01:33 |
| Grimmauld (any/all) | at least its not lix 2.92 now | 18:01:38 |
| Grimmauld (any/all) | ah wait i am stupid, oops | 18:02:03 |
| K900 | for_each_init basically creates a connection per worker thread | 18:02:12 |
| K900 | Or rather per batch TECHNICALLY | 18:02:16 |
| K900 | I think if you want more cheap perf win you can also do batch inserts for references | 18:03:36 |
| K900 | There's also probably something with prepared statements | 18:04:25 |
| K900 | And possibly also something to be said for doing everything in a transaction? | 18:06:29 |
| K900 | @Grimmauld (any/all) can you try the thing I just pushed | 18:20:36 |
| K900 | OK that does not work | 18:41:05 |
| Grimmauld (any/all) | with the old changes i got a full run in 9m48s | 18:41:06 |
| K900 | I forgot sqlite is single writer | 18:41:25 |
| Grimmauld (any/all) | 10min is fine, 2.91 (without those changes) was 12min | 18:42:02 |
| Grimmauld (any/all) | * 10min on 2.93 with buffer is fine, 2.91 (without those changes) was 12min | 18:42:19 |
| K900 | But what if duckdb | 18:42:20 |
| Grimmauld (any/all) | will python be able to read it as easily? | 18:42:37 |
| K900 | Yes | 18:42:41 |
| Grimmauld (any/all) | then maybe | 18:42:47 |
| K900 | Let me cook | 18:42:53 |
| Grimmauld (any/all) | 🧑🍳 | 18:43:02 |
| K900 | OK I can get 4:30 | 19:06:12 |
| K900 | Let's see how much better that is compared to sqlite I guess | 19:10:39 |
| K900 | * Let's see how much better duckdb is compared to sqlite I guess | 19:15:54 |
| K900 | The answer, as it turns out, is it's actually just worse | 19:16:08 |
| K900 | lol | 19:16:11 |
| Grimmauld (any/all) | and i am once again reminded just how slow my laptop is, despite having been quite capable at the time of buying it... | 19:16:50 |
