| 26 Jan 2025 |
 emily | it's --option cores and --option max-jobs in Nix (-j for short on the latter) | 21:28:41 |
| emily | that said there are like 6 Python processes using up a huge amount of CPU so if you're only building one package I'm not sure it's the problem :) | 21:29:21 |
 Ihar Hrachyshka | I think I know which package it is lol | 21:31:43 |
| Ihar Hrachyshka | jax pretty sure | 21:31:48 |
| Ihar Hrachyshka | you should probably ask Gaétan Lepage I believe he was looking at this derivation lately and we had some convos around the memory hogging / test timeouts there | 21:32:27 |
| emily | ideally the builder would not be unusably overloaded every couple days :( | 21:33:28 |
| emily | anyway, if you ^C'd your build then it's definitely not that (since the machine is still struggling) | 21:35:17 |
| emily | so feel free to put it back | 21:35:25 |
| Ihar Hrachyshka | yeah I'm not building most of the time. | 21:35:39 |
| Ihar Hrachyshka | for context on jax, I reported the build cpu hogging behavior here: https://github.com/NixOS/nixpkgs/pull/374810#issuecomment-2602217892 and I guess we are still not sure what's going on with the package. but tl;dr it's not even darwin specific, my nix-darwin builder vm also struggles the same way. | 21:36:38 |
| emily | even without memory hogging, >100% of available CPU cores regularly being used by one user on a community resource is just not fun to deal with | 21:37:23 |
| Ihar Hrachyshka | think the suggestion by SuperSandro to disable the test suite for jax until we have a grasp of what's going on may be a good one then https://github.com/NixOS/nixpkgs/pull/374810#discussion_r1928394273 | 21:39:53 |
| emily | I mean, anyone can choose how many threads they use up on the builders, it's just a matter of adjusting the setting | 21:40:48 |
| emily | (except for packages that don't respect $NIX_BUILD_CORES at all, but that's a package bug) | 21:41:01 |
| emily | zowoq: would it be possible to get some jobs killed on the Darwin builder? I had some free time tonight and am trying to test Darwin stdenv build fixes and Rust hash changes for the staging-next cycle in a couple of days, but with load average at 2× cores it can't handle the current load let alone piling on more. | 21:50:19 |
| emily | looks like it's doing a bit better now | 22:09:30 |
 Gaétan Lepage | I ran some jax builds on darwin this afternoon.
I have reduced --max-jobs to 4 for the darwin builders, but when 4 "big" packages are built at the same time the load can grow too much for sure.
Btw, is there a way to set the --cores option from the remote builders configuration ? I couldn't find it.
Final question, how fare are we budget-wise from getting more mac builders ? I can probably manage to contribute to get us there... | 23:39:00 |
| Gaétan Lepage | * I ran some jax builds on darwin this afternoon.
I have reduced --max-jobs to 4 for the darwin builders, but when 4 "big" packages are built at the same time the load can grow too much for sure.
Btw, is there a way to set the --cores option from the remote builders configuration ? I couldn't find it.
Final question, how far are we budget-wise from getting more mac builders ? I can probably manage to contribute to get us there... | 23:43:00 |
| 27 Jan 2025 |
 zowoq |
I have reduced --max-jobs to 4 for the darwin builders, but when 4 "big" packages are built at the same time the load can grow too much for sure.
Please set max-jobs to 1.
is there a way to set the --cores option from the remote builders configuration
No, not supported.
how far are we budget-wise from getting more mac builders
Not planning to get more mac builders and budget-wise we haven't covered the cost of the new aarch64-linux build box yet.
| 00:38:27 |
| zowoq | We could try changing the defaults on the box? | 00:39:34 |
| emily | does the nix.conf cores setting apply to remote builder use? | 00:40:02 |
| emily | seems like a good idea if so | 00:40:07 |
| emily | though picking a balance is always hard, it's the old big-parallel problem. but cores = jobs = nproc is probably not ideal for a shared machine | 00:40:49 |
| zowoq | cores will apply to remote builds but everyone is trusted so it can be overridden if running builds directly on the box. | 00:47:25 |
| Gaétan Lepage | In reply to @zowoq:matrix.org
I have reduced --max-jobs to 4 for the darwin builders, but when 4 "big" packages are built at the same time the load can grow too much for sure.
Please set max-jobs to 1.
is there a way to set the --cores option from the remote builders configuration
No, not supported.
how far are we budget-wise from getting more mac builders
Not planning to get more mac builders and budget-wise we haven't covered the cost of the new aarch64-linux build box yet.
Ok thanks for the details.
I have set it to 1 now. | 06:45:44 |
 leona | Gaétan Lepage: pytest of botorch eats all the cores on the aarch64 | 14:52:23 |
| Gaétan Lepage | Killed | 14:57:22 |
| leona | leona@darwin01 ~ % git
xcode-select: error: No developer tools were found and no install could be requested (possibly because there is no active GUI session).
If developer tools are located at a non-default location on disk, use `sudo xcode-select --switch path/to/Xcode.app` to specify the Xcode that you wish to use for command line developer tools.
Use `xcode-select --install` to install the standalone command line developer tools, or visit http://adc.apple.com to download Xcode or the standalone command line tools installation package.
See `man xcode-select` for more details.
uh. how is this intended to work?
| 14:57:37 |
| Gaétan Lepage | Those packages ignore cores this is very annoying | 14:57:48 |
 Roberto Abdelkader Martínez Pérez | Is there a way to issue a warning (e.g., with builtins.warn) only when a package is being built locally and not fetched from the cache? Looking to notify users during the build process without blocking it. Any ideas? | 16:16:28 |
