| 18 Mar 2023 |
 SomeoneSerge (back on matrix) | There's absolutely zero reason not to have a separate build step before install here =\ | 22:39:32 |
 hexa | can imagine this to get quite messy | 22:42:33 |
| 19 Mar 2023 |
 connor (he/him) | Is there guidance on when to use preConfigure to set environment variables vs. setting it in the derivation via all-caps attributes? For example, when building on the PyTorch derivation, should I add attributes for environment variables like https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/torch/default.nix#L208-L222, or set them in the preConfigure, like https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/torch/default.nix#L196? | 21:24:32 |
| 20 Mar 2023 |
| SomeoneSerge (back on matrix) | crt/host_defines.h and such are shipped in cuda_nvcc -> currently we can't simply drop cuda_nvcc in nativeBuildInputs, but have to sometimes add it to buildInputs as well
We should split the outputs
| 20:15:15 |
| connor (he/him) | We should also definitely split the outputs for CUDNN given that the static libraries nearly, what, double the size? 1.2GB -> 2.4GB? | 20:34:13 |
| SomeoneSerge (back on matrix) | In reply to @connorbaker:matrix.org
We should also definitely split the outputs for CUDNN given that the static libraries nearly, what, double the size? 1.2GB -> 2.4GB? Yes, we should split out all static libs, not only cudnn, but cuda redist | 20:59:37 |
| 21 Mar 2023 |
|  freesig joined the room. | 00:59:54 |
| SomeoneSerge (back on matrix) | Let's see https://github.com/NixOS/nixpkgs/pull/222273 | 01:36:36 |
| SomeoneSerge (back on matrix) | Not that I know, but we probably want to minimize the size of imperative scripts. One valid reason to use preConfigure could be to explicitly extent an existing bash variable or something. In the linked examples we could probably just as well use env = optionalAttrs ...? | 01:39:49 |
 mjlbach | connor (he/him): correct me if I'm wrong, but I believe if you only build for compute capability 9.0 (sm_90) that will only support the NVIDIA H100 because its not backwards compatible (PTX ensures forward compatibility with newer GPUS) | 02:30:07 |
| mjlbach | Also Tyler (tbenst) doesn't really use nix anymore, I saw some people pinging him for approval on magma and such. I just texted him to confirm (we are friends IRL) | 02:30:40 |
| connor (he/him) | That's correct, it's only forward not backward compatible
Come to think of it, should we even have a default set of cuda architectures we build for? If we're looking at making the default just the single test architecture, why bother having a default? Why not require the user to specify what they want? | 03:21:08 |
| connor (he/him) | In reply to @atrius:matrix.org
Also Tyler (tbenst) doesn't really use nix anymore, I saw some people pinging him for approval on magma and such. I just texted him to confirm (we are friends IRL) Would he be okay if I removed him from stuff like Magma? What's the recommended way to handle that? | 03:24:57 |
| mjlbach | I just texted him, will let you know :) | 03:25:33 |
| mjlbach | In reply to @connorbaker:matrix.org
That's correct, it's only forward not backward compatible
Come to think of it, should we even have a default set of cuda architectures we build for? If we're looking at making the default just the single test architecture, why bother having a default? Why not require the user to specify what they want? Is there a reason for targeting 90 instead of 86 + PTX | 03:26:27 |
| mjlbach | The only issue with having users specify the default is its not exactly clear | 03:27:10 |
| mjlbach | I guess you can just put a link to https://developer.nvidia.com/cuda-gpus for people to to figure out the needed versions | 03:28:20 |
| connor (he/him) | In reply to @atrius:matrix.org
Is there a reason for targeting 90 instead of 86 + PTX Not really; if this is in reference https://github.com/NixOS/nixpkgs/issues/221564#issuecomment-1477191551 that was just because it's the latest. 86+PTX would give the best performance with the largest compatibility with Nixpkgs as it is currently, I think (latest version Torch 1.13 supports for example). 50+PTX would give the broadest HW support with (possibly) the worst performance. | 03:30:54 |
| mjlbach | Ah, sorry for the phrasing, was suggesting 86 :) | 03:31:37 |
| mjlbach | * Ah, sorry for the phrasing, was suggesting 86, 89, 90 + PTX :) | 03:31:56 |
| mjlbach | I also finally got pytorch building with poetry2nix... | 03:32:25 |
| connor (he/him) | In reply to @atrius:matrix.org
The only issue with having users specify the default is its not exactly clear that's definitely an issue! Personally I'd be a fan of an assert like cudaSupport -> cudaCapabilities != [ ] in cudaPackages which points the user to docs about how to build for a specific target/how to find out the cuda compute capability of their device | 03:33:32 |
| mjlbach | Maybe there should be a default cuda flake with instructions that shows how to override it | 03:34:31 |
| mjlbach | * Maybe there should be a default cuda flake with instructions that shows how to override it/ with the doc link embedded | 03:34:41 |
| mjlbach | https://github.com/cpcloud/torch-p2nix/blob/81f318026f19e1a2a41cf6126f8e1cd5a7fab8be/flake.nix#L24-L61 | 03:35:03 |
| mjlbach | The issue with torch is that I can't see how one derivation is going to provide support for building all versions of torch that are currently in circulation | 03:35:41 |
| SomeoneSerge (back on matrix) | In reply to @atrius:matrix.org
The issue with torch is that I can't see how one derivation is going to provide support for building all versions of torch that are currently in circulation
all versions of torch that are currently in circulation
You mean semvers?
| 11:10:56 |
| SomeoneSerge (back on matrix) | LoL, I think supplying a global cudaSupport overrides USE_CUDA in the ROCM version of torch and spoils nixpkgs-review 🙃 | 11:26:53 |
| SomeoneSerge (back on matrix) | Also, we ought to do sometihng about these pytestCheckPhases, they really don't go well in parallel | 11:27:55 |
| connor (he/him) | Does anyone else set max-jobs = auto in their nix configuration? I've found that I need to limit it to 2 or similar to prevent nixpkgs-review from building several copies of torch and Jax in parallel lol | 14:27:15 |
