| 13 Oct 2023 |
 Theuni | Or the queue runner is apparently a bottleneck for s3 uploads and then doesn't run any other jobs in the meantime while consuming 300% CPU for packing stuff ... o_O | 05:10:06 |
| Theuni | From my perspective it could make sense to try and fix those in the current code base so they can be used as input for whatever comes next. | 05:10:48 |
| Theuni | I'm noticing that nobody here currently has the reflex to add those issues to the issue tracker as the situation is generally considered "nothing happens anyway". | 05:13:26 |
| Theuni | (here = at the FC) | 05:13:33 |
 vcunat | Yes, hydra.nixos.org often does seem limited primarily by "Copying Results" phase, presumably the compression step. | 08:35:55 |
| vcunat | (network links certainly aren't much utilized when it happens) | 08:36:11 |
 @lotte:chir.rs | yeah, ideally there would be a way to run the copy step in the background | 09:14:29 |
| @lotte:chir.rs | my current hydra instance adds it as a runcommand hook right now, but previously i have used nix post-build hooks | 09:15:48 |
| @lotte:chir.rs | my current hydra config adds it as a runcommand hook right now, but previously i have used nix post-build hooks | 09:16:26 |
| @lotte:chir.rs | my current hydra config adds build results to a queue as a runcommand hook right now, but previously i have used nix post-build hooks | 09:16:45 |
| 15 Oct 2023 |
|  meet changed their display name from meet to meetm. | 07:05:59 |
 woobilicious | In reply to @raitobezarius:matrix.org
Plus, as much I am a type theorist, Rust type system has IMHO the right level to focus on the matter at hand rather than building castles of abstraction (which would make sense for other stuff, but not things like Hydra IMHO) That's a surprising take since rust seems like it was designed to do more low level, manual management of memory safely. While we just want to interact with some high level stuff like S3 and github, website? you'd get a long way with Haskell web framework and a little abstraction, without having to think too much about it, especially since it' has a real runtime with garbage collection & green threads. I do agree however with the pragmatic stuff like C-FFI, code reuse and accessibility. | 09:24:06 |
| woobilicious | Rust's lack of uninitialised, heavy used of optional, careful type design, and borrow checker does solve like 99% of the issues I have with every other language. | 09:33:18 |
| vcunat | hydra-queue-runner is quite performance-critical and I don't think the task would be suitable for Haskell. | 11:51:20 |
| vcunat | (build-step graph with like a million nodes in memory, scheduling that, etc.) | 11:52:19 |
 K900 | Honestly | 11:52:24 |
| K900 | We probably don't need that graph in memory at all | 11:52:29 |
| vcunat | Yes, it sounds avoidable to me. (as you mentioned recently, in #infra:nixos.org probably) | 11:53:31 |
| K900 | We can just shove it all into Redis or something | 11:53:35 |
| K900 | And have a priority queue | 11:53:43 |
| K900 | We can also do fancy work stealing stuff that way | 11:55:28 |
| K900 | So we don't even need a central scheduler | 11:55:55 |
| K900 | Well, kinda | 11:56:12 |
| K900 | But we don't need it on the hot path | 11:56:17 |
 das_j | In reply to @k900:0upti.me
We can just shove it all into Redis or something But who will do that | 16:27:04 |
| K900 | Like, who will actually write the code? | 16:27:28 |
| K900 | Or what part of Hydra will do that thing | 16:27:35 |
| das_j | In reply to @k900:0upti.me
Like, who will actually write the code? Yes | 16:27:53 |
| K900 | Yeah that's kind of the problem with everything | 16:28:07 |
| Theuni | Apart from performance i would (ftom my perspective) focus rather on robustness. I am sure som current issues might just be craft issues. | 17:43:35 |
