| 26 Jan 2025 |
 emily | might be good to lower cores/max-jobs though I don't know if it's already overloaded from other builds | 21:24:55 |
 Ihar Hrachyshka | emily: ok sorry, I'm obtuse :) you want me to hold off for now? that's fine, I can do something else. | 21:24:59 |
| emily | could you just try with lower cores/max-jobs settings maybe? | 21:25:21 |
| emily | the default is 10 cores + 10 jobs, which can mean up to ~100 threads on a 10-core processor | 21:25:39 |
| emily | the stdenv build will take hours anyway so it's not urgent, but I don't want to throw more jobs at an already overloaded machine | 21:26:08 |
| Ihar Hrachyshka | it's 1 job for me. how do I limit the cores? is there a universal recipe or I patch cmake files? | 21:26:25 |
| emily | it's --option cores and --option max-jobs in Nix (-j for short on the latter) | 21:28:41 |
| emily | that said there are like 6 Python processes using up a huge amount of CPU so if you're only building one package I'm not sure it's the problem :) | 21:29:21 |
| Ihar Hrachyshka | I think I know which package it is lol | 21:31:43 |
| Ihar Hrachyshka | jax pretty sure | 21:31:48 |
| Ihar Hrachyshka | you should probably ask Gaétan Lepage I believe he was looking at this derivation lately and we had some convos around the memory hogging / test timeouts there | 21:32:27 |
| emily | ideally the builder would not be unusably overloaded every couple days :( | 21:33:28 |
| emily | anyway, if you ^C'd your build then it's definitely not that (since the machine is still struggling) | 21:35:17 |
| emily | so feel free to put it back | 21:35:25 |
| Ihar Hrachyshka | yeah I'm not building most of the time. | 21:35:39 |
| Ihar Hrachyshka | for context on jax, I reported the build cpu hogging behavior here: https://github.com/NixOS/nixpkgs/pull/374810#issuecomment-2602217892 and I guess we are still not sure what's going on with the package. but tl;dr it's not even darwin specific, my nix-darwin builder vm also struggles the same way. | 21:36:38 |
| emily | even without memory hogging, >100% of available CPU cores regularly being used by one user on a community resource is just not fun to deal with | 21:37:23 |
| Ihar Hrachyshka | think the suggestion by SuperSandro to disable the test suite for jax until we have a grasp of what's going on may be a good one then https://github.com/NixOS/nixpkgs/pull/374810#discussion_r1928394273 | 21:39:53 |
| emily | I mean, anyone can choose how many threads they use up on the builders, it's just a matter of adjusting the setting | 21:40:48 |
| emily | (except for packages that don't respect $NIX_BUILD_CORES at all, but that's a package bug) | 21:41:01 |
| emily | zowoq: would it be possible to get some jobs killed on the Darwin builder? I had some free time tonight and am trying to test Darwin stdenv build fixes and Rust hash changes for the staging-next cycle in a couple of days, but with load average at 2× cores it can't handle the current load let alone piling on more. | 21:50:19 |
| emily | looks like it's doing a bit better now | 22:09:30 |
 Gaétan Lepage | I ran some jax builds on darwin this afternoon.
I have reduced --max-jobs to 4 for the darwin builders, but when 4 "big" packages are built at the same time the load can grow too much for sure.
Btw, is there a way to set the --cores option from the remote builders configuration ? I couldn't find it.
Final question, how fare are we budget-wise from getting more mac builders ? I can probably manage to contribute to get us there... | 23:39:00 |
| Gaétan Lepage | * I ran some jax builds on darwin this afternoon.
I have reduced --max-jobs to 4 for the darwin builders, but when 4 "big" packages are built at the same time the load can grow too much for sure.
Btw, is there a way to set the --cores option from the remote builders configuration ? I couldn't find it.
Final question, how far are we budget-wise from getting more mac builders ? I can probably manage to contribute to get us there... | 23:43:00 |
| 27 Jan 2025 |
 zowoq |
I have reduced --max-jobs to 4 for the darwin builders, but when 4 "big" packages are built at the same time the load can grow too much for sure.
Please set max-jobs to 1.
is there a way to set the --cores option from the remote builders configuration
No, not supported.
how far are we budget-wise from getting more mac builders
Not planning to get more mac builders and budget-wise we haven't covered the cost of the new aarch64-linux build box yet.
| 00:38:27 |
| zowoq | We could try changing the defaults on the box? | 00:39:34 |
| emily | does the nix.conf cores setting apply to remote builder use? | 00:40:02 |
| emily | seems like a good idea if so | 00:40:07 |
| emily | though picking a balance is always hard, it's the old big-parallel problem. but cores = jobs = nproc is probably not ideal for a shared machine | 00:40:49 |
| zowoq | cores will apply to remote builds but everyone is trusted so it can be overridden if running builds directly on the box. | 00:47:25 |
