| 1 Mar 2025 |
 Vladimír Čunát | * As it is designed now, everyone will be waiting for this huge thing to finish. | 16:29:11 |
 emily | so… has hydraPlatforms = [ ]; been considered? | 16:30:05 |
| emily | how important is the composable_kernel thing? | 16:30:07 |
| emily | I know ROCm has a bunch of stuff and some of it nobody uses, right? | 16:30:19 |
 Lun | It's needed for MIOpen which is needed for pytorch for kernels for some ops, it will also be used directly by pytorch in 2.6 | 16:32:47 |
| Lun | https://github.com/NixOS/nixpkgs/blob/bda6bcbbacbd8f48e69b228b91b5541c03f7ab35/pkgs/development/rocm-modules/6/composable_kernel/default.nix
If anyone happens to know cmake wizardry and can work out how to split this into multiple derivations and get a working end result without needing a massive patch to the CMakeFiles that might be the best option? | 16:36:38 |
| emily | okay that sounds pretty important then | 16:37:02 |
| emily | is the issue that there's only one install target and it depends on everything? | 16:37:28 |
| emily | I assume you can build individual libraries with ninja <target> at least | 16:37:43 |
| Lun | The kernels under tensor_operation_instance https://github.com/ROCm/composable_kernel/blob/1bf29478cdada3c7f56fbedc5542b275b0c107b3/library/src/tensor_operation_instance/gpu/CMakeLists.txt are approximately all the build time. | 16:40:02 |
| emily | what if you have derivations that do the same CMake setup dance, use ninjaTargets (or whatever that variable is called) to build one (set of) kernel(s), and the installPhase just directly copies it over? then the main derivation can aggregate all those sub-derivations and copy them into place in the build directory before running the rest of the build | 16:44:18 |
| emily | that way you can avoid patching the CMake build and still split it up | 16:44:28 |
| Lun | I'll give that a try | 16:46:28 |
| emily | the advantage is that it can be split across multiple builders and maybe not even need big-parallel | 16:47:11 |
| Vladimír Čunát | Especially if there are cases that someone needs to depend only on some of the kernels? | 16:48:40 |
| emily | I assume the install phase still expects all of it, I wasn't thinking the overall packaging would change since that sounds more involved | 16:49:16 |
| emily | I imagine PyTorch wants to be able to use whatever | 16:49:25 |
| Vladimír Čunát | So PyTorch will be a monster of several gigabytes, needing a big computer to build it? Anyway, I guess I'm too verbose for this channel. | 16:52:44 |
| emily | aren't all the ML libraries already that? | 16:59:12 |
| emily | (but I guess this is what rocmSupport or whatever is for?) | 16:59:26 |
| 2 Mar 2025 |
| Vladimír Čunát | Huh, I wonder what this is:
Aborted: [31;1merror:[0m deleting cgroup '[35;1m/sys/fs/cgroup/system.slice/nix-daemon.service/nix-build-uid-30014[0m': [35;1mDevice or resource busy[0m
| 13:34:04 |
 hexa | wild | 17:42:45 |
 raitobezarius | joining non leaf node cgroups is illegal in Linux | 17:44:12 |
| raitobezarius | this is probably the reason for the EBUSY | 17:44:18 |
| raitobezarius | is it running lix or nix? | 17:45:19 |
| hexa | the builder is running lix | 17:45:32 |
| hexa | still that version from when I set the node up | 17:45:41 |
| raitobezarius | hm | 17:47:13 |
| raitobezarius | are you running with the cgroup xp feature? | 17:47:18 |
| hexa | yep | 17:47:24 |
