| 26 Jan 2025 |
 Ihar Hrachyshka | I think I know which package it is lol | 21:31:43 |
| Ihar Hrachyshka | jax pretty sure | 21:31:48 |
| Ihar Hrachyshka | you should probably ask Gaétan Lepage I believe he was looking at this derivation lately and we had some convos around the memory hogging / test timeouts there | 21:32:27 |
 emily | ideally the builder would not be unusably overloaded every couple days :( | 21:33:28 |
| emily | anyway, if you ^C'd your build then it's definitely not that (since the machine is still struggling) | 21:35:17 |
| emily | so feel free to put it back | 21:35:25 |
| Ihar Hrachyshka | yeah I'm not building most of the time. | 21:35:39 |
| Ihar Hrachyshka | for context on jax, I reported the build cpu hogging behavior here: https://github.com/NixOS/nixpkgs/pull/374810#issuecomment-2602217892 and I guess we are still not sure what's going on with the package. but tl;dr it's not even darwin specific, my nix-darwin builder vm also struggles the same way. | 21:36:38 |
| emily | even without memory hogging, >100% of available CPU cores regularly being used by one user on a community resource is just not fun to deal with | 21:37:23 |
| Ihar Hrachyshka | think the suggestion by SuperSandro to disable the test suite for jax until we have a grasp of what's going on may be a good one then https://github.com/NixOS/nixpkgs/pull/374810#discussion_r1928394273 | 21:39:53 |
| emily | I mean, anyone can choose how many threads they use up on the builders, it's just a matter of adjusting the setting | 21:40:48 |
| emily | (except for packages that don't respect $NIX_BUILD_CORES at all, but that's a package bug) | 21:41:01 |
| emily | zowoq: would it be possible to get some jobs killed on the Darwin builder? I had some free time tonight and am trying to test Darwin stdenv build fixes and Rust hash changes for the staging-next cycle in a couple of days, but with load average at 2× cores it can't handle the current load let alone piling on more. | 21:50:19 |
| emily | looks like it's doing a bit better now | 22:09:30 |
 Gaétan Lepage | I ran some jax builds on darwin this afternoon.
I have reduced --max-jobs to 4 for the darwin builders, but when 4 "big" packages are built at the same time the load can grow too much for sure.
Btw, is there a way to set the --cores option from the remote builders configuration ? I couldn't find it.
Final question, how fare are we budget-wise from getting more mac builders ? I can probably manage to contribute to get us there... | 23:39:00 |
| Gaétan Lepage | * I ran some jax builds on darwin this afternoon.
I have reduced --max-jobs to 4 for the darwin builders, but when 4 "big" packages are built at the same time the load can grow too much for sure.
Btw, is there a way to set the --cores option from the remote builders configuration ? I couldn't find it.
Final question, how far are we budget-wise from getting more mac builders ? I can probably manage to contribute to get us there... | 23:43:00 |
| 27 Jan 2025 |
 zowoq |
I have reduced --max-jobs to 4 for the darwin builders, but when 4 "big" packages are built at the same time the load can grow too much for sure.
Please set max-jobs to 1.
is there a way to set the --cores option from the remote builders configuration
No, not supported.
how far are we budget-wise from getting more mac builders
Not planning to get more mac builders and budget-wise we haven't covered the cost of the new aarch64-linux build box yet.
| 00:38:27 |
| zowoq | We could try changing the defaults on the box? | 00:39:34 |
| emily | does the nix.conf cores setting apply to remote builder use? | 00:40:02 |
| emily | seems like a good idea if so | 00:40:07 |
| emily | though picking a balance is always hard, it's the old big-parallel problem. but cores = jobs = nproc is probably not ideal for a shared machine | 00:40:49 |
| zowoq | cores will apply to remote builds but everyone is trusted so it can be overridden if running builds directly on the box. | 00:47:25 |
| Gaétan Lepage | In reply to @zowoq:matrix.org
I have reduced --max-jobs to 4 for the darwin builders, but when 4 "big" packages are built at the same time the load can grow too much for sure.
Please set max-jobs to 1.
is there a way to set the --cores option from the remote builders configuration
No, not supported.
how far are we budget-wise from getting more mac builders
Not planning to get more mac builders and budget-wise we haven't covered the cost of the new aarch64-linux build box yet.
Ok thanks for the details.
I have set it to 1 now. | 06:45:44 |
 leona | Gaétan Lepage: pytest of botorch eats all the cores on the aarch64 | 14:52:23 |
| Gaétan Lepage | Killed | 14:57:22 |
| leona | leona@darwin01 ~ % git
xcode-select: error: No developer tools were found and no install could be requested (possibly because there is no active GUI session).
If developer tools are located at a non-default location on disk, use `sudo xcode-select --switch path/to/Xcode.app` to specify the Xcode that you wish to use for command line developer tools.
Use `xcode-select --install` to install the standalone command line developer tools, or visit http://adc.apple.com to download Xcode or the standalone command line tools installation package.
See `man xcode-select` for more details.
uh. how is this intended to work?
| 14:57:37 |
| Gaétan Lepage | Those packages ignore cores this is very annoying | 14:57:48 |
 Roberto Abdelkader Martínez Pérez | Is there a way to issue a warning (e.g., with builtins.warn) only when a package is being built locally and not fetched from the cache? Looking to notify users during the build process without blocking it. Any ideas? | 16:16:28 |
 bandithedoge | In reply to @nilp0inter2k6:matrix.org
Is there a way to issue a warning (e.g., with builtins.warn) only when a package is being built locally and not fetched from the cache? Looking to notify users during the build process without blocking it. Any ideas? not really, every bit of the derivation is evaluated when building to determine the hash so a warning will show up regardless | 16:20:46 |
| Roberto Abdelkader Martínez Pérez | In reply to @bandithedoge:matrix.org
not really, every bit of the derivation is evaluated when building to determine the hash so a warning will show up regardless Thanks! That clears it up. I'll look into other ways to handle it. | 16:59:54 |
