| 12 Feb 2025 |
 hexa | but we have 16 workers right now | 20:43:48 |
| hexa | and I might want to increase that even further 😄 | 20:43:55 |
 K900 | Wait is there no sync on the output | 20:46:24 |
| 13 Feb 2025 |
 John Ericson | Mic92: can we has moar pipe? | 01:01:47 |
| John Ericson | we could do a unix domain socket | 01:01:52 |
| John Ericson | and then every worker connects afresh | 01:02:01 |
 Mic92 | What does a long line look like? | 01:10:50 |
| Mic92 | Don't understand yet what the issue is. Is perl reading not full lines? | 01:13:08 |
| hexa | it looked like multiple workers puking into the same pipe | 01:13:52 |
| hexa | which sometimes lead to json … mixed with other json | 01:14:11 |
| hexa | not sure if John Ericson knows anything else by now | 01:14:34 |
| Mic92 | Okay will check for today | 01:15:01 |
| Mic92 | I have machines with a lot of ram and core count but haven't seen that so far | 01:16:19 |
| Mic92 | But maybe libc needs to flush if it gets to the pipe buffer limit regardless of a missing newline | 01:17:32 |
 emily | I wouldn't rely on multiple processes writing into the same stream pipe. | 01:18:04 |
| emily | that's just the wrong primitive and will never work fully reliably | 01:18:17 |
| Mic92 | I can just add a lock around it | 01:18:27 |
| Mic92 | We are not really limited by the speed to write to stdout | 01:18:48 |
| Mic92 | On nixpkgs it's slow enough that I can almost read it | 01:19:06 |
| John Ericson | Mic92: yeah everything is in the issue | 01:21:22 |
| John Ericson | I know nothing that anyone else doesn't | 01:21:28 |
| John Ericson | I do agree with emily that multiple pipes/sockets feels better than locking | 01:21:52 |
| John Ericson | or are you sending everything back to the main process anyways, so it is in a good position to mux them? | 01:22:11 |
| Mic92 | I would prefer it though since it makes it very easy to use. At least unless we identify this an actual performance bottleneck | 01:24:16 |
| Mic92 | * I would prefer a single output though since it makes it very easy to use. At least unless we identify this an actual performance bottleneck | 01:24:32 |
| Mic92 | I am thinking of all the other ci integrations that people build with it | 01:25:21 |
| Mic92 | In reply to @Ericson2314:matrix.org
or are you sending everything back to the main process anyways, so it is in a good position to mux them? I don't remember the details because this code already existed when I copied it out of hydra | 01:48:03 |
| John Ericson | Mic92: My guess is the non-streaming version made the central process buffer everything and accumulate one big message | 01:48:44 |
| John Ericson | but when we switched to streaming, each worker started writing down the pipe itself | 01:48:54 |
| John Ericson | So I am gonna guess roughtly that means "easy mode" should be the old way, while "fast mode" should be the socket way | 01:49:32 |
