| 15 Jan 2024 |
 Lily Foster | In reply to @adam:robins.wtf
how can we speed up eval and aarch64-darwin builders? can we throw money at these problems? money can to an extent probably be thrown at the problem. really our eval shouldn't be so expensive, but if we had the money, we could still theoretically just get more evaluators/x86_64-linux builders (they are colocated) and more aarch64-darwin builders for short-term and have it be better (but our eval compute cost has been going up a lot the last year or so, and it's not helped by nix performance regressing in both eval time and memory pretty much every release iirc...) | 13:56:53 |
 @adam:robins.wtf | honestly i think the aarch64-darwin builders is the bigger problem. | 13:58:04 |
| @adam:robins.wtf | 12+ hours just to get time on an aarch64-darwin builder is absurd and can only hurt the support of this platform as people will merge without waiting | 14:00:00 |
| @adam:robins.wtf | especially given that x86_64-darwin, a dying platform, is competing in a much more reasonable amount of time | 14:01:21 |
| @adam:robins.wtf | * especially given that x86_64-darwin, a dying platform, is completing in a much more reasonable amount of time | 14:01:33 |
| Lily Foster | idk how much the macstadium aarch64-darwin builders are now, but tbh we could probably do what we did for hydra and switch our x86_64-darwin builders to aarch64-darwin and just use rosetta | 14:02:01 |
| Lily Foster | In reply to @lily:lily.flowers
idk how much the macstadium aarch64-darwin builders are now, but tbh we could probably do what we did for hydra and switch our x86_64-darwin builders to aarch64-darwin and just use rosetta thoughts on this @[cole-h]? | 14:02:38 |
| @adam:robins.wtf | i'd be willing to contribute some money directly towards better aarch64-darwin resources. maybe not enough to fully fund expansion, but i'm willing to try and raise funds from others too | 14:05:11 |
| Lily Foster | In reply to @adam:robins.wtf
i'd be willing to contribute some money directly towards better aarch64-darwin resources. maybe not enough to fully fund expansion, but i'm willing to try and raise funds from others too well tbh i'm thinking if the cost difference isn't that much, we can probably just have 6 aarch64-darwin/rosetta instead of 4 x86_64 + 2 aarch64. but depending on how the x86_64 builders are performing it could be a bit if a performance hit. however given they are mostly idle and the aarch64 queue is mostly behind, i think it may come out in our favor | 14:08:13 |
| Lily Foster | * well tbh i'm thinking if the cost difference isn't that much, we can probably just have 6 aarch64-darwin/rosetta instead of 4 x86_64 + 2 aarch64. but depending on how the x86_64 builders are performing it could be a bit of a performance hit. however given they are mostly idle and the aarch64 queue is mostly behind, i think it may come out in our favor | 14:08:26 |
| Lily Foster | if it's not enough, we can probably look at funding more builders from there | 14:08:47 |
| @adam:robins.wtf | i've found rosetta performance to be pretty good | 14:08:54 |
| @adam:robins.wtf | granted i haven't run it at scale, but my M1 mini builds both aarch64 and x86_64 at reasonable speed | 14:09:25 |
| Lily Foster | it's good yeah! having run a lot of big builds on both though, it's probably only about as good as the older intel processors but that's not bad at all, even if it is a perf hit | 14:10:01 |
| @adam:robins.wtf | these macstadium prices are crazy. in 8 months on a M2 Ultra 24 Core Mac Studio, you should have just bought the machine yourself | 14:11:08 |
| Lily Foster | ooof really? | 14:11:32 |
| @adam:robins.wtf | https://www.macstadium.com/pricing | 14:11:57 |
| @adam:robins.wtf | maybe 9 months, since they have 2TB drives | 14:13:08 |
| @adam:robins.wtf | https://www.apple.com/shop/buy-mac/mac-studio/24-core-cpu-60-core-gpu-32-core-neural-engine-64gb-memory-1tb | 14:13:10 |
| Lily Foster | damn, i mean we probably could arrange just getting machines. idk if we're paying that price or if we're getting sponsorship from them though | 14:13:32 |
| @adam:robins.wtf | right | 14:13:42 |
| Lily Foster | * damn, i mean we probably could arrange just getting machines if we end up getting a colo for nixos infra. idk if we're paying that price or if we're getting sponsorship from them though | 14:13:48 |
 trofi | Bringing up https://github.com/NixOS/ofborg/pull/669 ofborg/src/maintainers.nix: look up files pointed by "pos" attribute here again in case it slipped through the cracks. | 19:49:06 |
| 16 Jan 2024 |
 cole-h | In reply to @lily:lily.flowers
thoughts on this @[cole-h]? I don't quite know the intricacies of how the macstadium agreement works, but maybe we could. I'll look into it. One problem is then we're running 2 builds on the same machine (1 aarch64, 1 x86_64), which might continue to make the slowness worse. | 15:07:29 |
| 19 Jan 2024 |
 hexa | domen will know | 01:26:39 |
| 20 Jan 2024 |
 ris_ | still getting an ofborg internal error on https://github.com/NixOS/nixpkgs/pull/274089 after a rebase | 12:31:57 |
| Lily Foster | In reply to @r_i_s:matrix.org
still getting an ofborg internal error on https://github.com/NixOS/nixpkgs/pull/274089 after a rebase can you try running the command that it's getting the internal error at? nix-build --arg pkgs 'import ./. {}' ./lib/tests/release.nix? | 14:28:24 |
| ris_ | -> huge rebuild | 14:46:22 |
| Lily Foster | In reply to @r_i_s:matrix.org
-> huge rebuild but not some weird failure? | 14:46:40 |
| Lily Foster | hmmm, @[cole-h] can access the logs to see what exactly happened on the evaluator. otherwise i'll try to replicate myself when i get a chance (my, uh, when-i-have-a-chance list is getting a bit long though....) | 14:47:40 |
