| 11 Oct 2024 |
 K900 | But I'd expect roughly similar MT perf | 14:20:18 |
| K900 | With a pretty strong ST lead for the Epyc | 14:20:23 |
 Tristan Ross | Gotcha, and the thermals would probably be similar | 14:20:53 |
| K900 | Thermals, frankly, should not be our problem | 14:21:59 |
| K900 | If Hetzner can't figure out a way to get us hardware that's not thermal throttling, we'll just have to do the math | 14:22:32 |
| Tristan Ross | Yeah | 14:22:57 |
 Mic92 | In reply to @hexa:lossy.network
bottlenecks:
- parallel compress slots (currently limited at 30, which seems reasonable in relation to the compute rhea has)
- eval memory, which we compensate with zram at 150%
- eval time, which is single-threaded and probably not fixable through hw upgrades
Eval is parallel in hydra | 14:27:12 |
 hexa (signing key rotation when) | it can be, but it is not on h.n.o | 14:27:33 |
| Mic92 | Not enabled? | 14:27:52 |
| hexa (signing key rotation when) | evaling trunk-combined exceeds the available memory | 14:27:58 |
| K900 | Single threaded eval nearly OOMs the box | 14:28:08 |
| K900 | And the way parallel eval works just makes it even worse | 14:28:20 |
| Mic92 | Ok. Got it | 14:28:22 |
| Mic92 | Is there a derivation that depends on all other derivations? | 14:30:54 |
| Mic92 | Because this not normal | 14:31:07 |
| Mic92 | I am able to eval arbitrary large package sets with nix-eval-jobs | 14:33:33 |
| Mic92 | It will reclaim memory | 14:33:45 |
| K900 | In reply to@joerg:thalheim.io
Is there a derivation that depends on all other derivations? The tested job depends on A LOT of things | 14:34:31 |
| K900 | I believe it is the primary bottleneck | 14:34:37 |
| K900 | Because one of the things it depends on is like 200 VM tests | 14:34:50 |
| Mic92 | Sure but every of these should be able to eval independently | 14:35:25 |
| Tristan Ross | Is it possible to predict memory usage of different derivations during eval based on the usage from the previous evals and then use that queue parallel evals in a way to not OOM? | 14:35:36 |
| K900 | Theoretically, yes | 14:35:49 |
| K900 | Practically, by the time you have tooling to do that, you can probably use the same tooling to just reduce eval requirements on the nixpkgs side | 14:36:09 |
| Mic92 | This is what hydra-eval-jobs does | 14:36:16 |
| Mic92 | When I evaled nixpkgs I was I/o bound for packages | 14:37:27 |
| Mic92 | Only nixos tests were CPU bound | 14:37:49 |
| K900 | You can try to just eval nixos/release.nix | 14:39:18 |
| K900 | If you want to profile it | 14:39:21 |
