Here's the stack trace when I re-enable the failing test:

______________________ ERROR collecting test_nvjitlink.py ______________________
tests/test_nvjitlink.py:62: in <module>
    not check_nvjitlink_usable(), reason="nvJitLink not usable, maybe not installed or too old (<12.3)"
        ^^^^^^^^^^^^^^^^^^^^^^^^
tests/test_nvjitlink.py:58: in check_nvjitlink_usable
    return inner_nvjitlink._inspect_function_pointer("__nvJitLinkVersion") != 0
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cuda/bindings/_internal/nvjitlink.pyx:254: in cuda.bindings._internal.nvjitlink._inspect_function_pointer
    ???
cuda/bindings/_internal/nvjitlink.pyx:257: in cuda.bindings._internal.nvjitlink._inspect_function_pointer
    ???
cuda/bindings/_internal/nvjitlink.pyx:205: in cuda.bindings._internal.nvjitlink._inspect_function_pointers
    ???
cuda/bindings/_internal/nvjitlink.pyx:99: in cuda.bindings._internal.nvjitlink._check_or_init_nvjitlink
    ???
cuda/bindings/_internal/nvjitlink.pyx:57: in cuda.bindings._internal.nvjitlink.load_library
    ???
cuda/bindings/_internal/utils.pyx:137: in cuda.bindings._internal.utils.get_nvjitlink_dso_version_suffix
    ???
E   cuda.bindings._internal.utils.NotSupportedError: only CUDA 12 driver is supported
------------------------------- Captured stdout --------------------------------
+++++++++++++++ 13010

hm i'm getting an infinite hang on pytestCheckPhase for python3.12-keras3.13.1 since the 3.13.1 update 🤔 https://github.com/NixOS/nixpkgs/commit/73806a397d9f9360f331ed98202e5cecc1230548
building python3.12-keras-3.13.1 (pytestCheckPhase): keras/src/trainers/trainer_test.py::TestTrainer::test_fit_with_data_adapter_py_dataset_infinite_cw PASSED [ 94%]
hang happens every time

nix-top shows

  UID     PID    PPID STIME     TIME COMMAND
30001 3422767 3422639 15:44 00:00:00 bash -e /nix/store/l622p70vy8k5sh7y5wizi5f2mic6ynpg-source-stdenv.sh /nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh
30001 3425989 3422767 15:45 00:17:09 /nix/store/iizfzmqjdfvbyx82gqw92glb5zl6m4my-python3-3.12.12/bin/python3.12 -m pytest --ignore-glob=keras/src/layers/preprocessing/data_layer_test.py --ignore-glob=keras/src/layers/preprocessing/image_preprocessing/resizing_test.py --ignore-glob=keras/src/layers/preprocessing/rescaling_test.py --ignore-glob=keras/src/trainers/data_adapters/grain_dataset_adapter_test.py --ignore-glob=keras/src/export/onnx_test.py --ignore-glob=integration_tests/dataset_tests --ignore-glob=keras/src/applications/applications_test.py --ignore-glob=integration_tests/jax_custom_fit_test.py --ignore-glob=integration_tests/tf_distribute_training_test.py --ignore-glob=integration_tests/torch_custom_fit_test.py --ignore-glob=integration_tests/torch_workflow_test.py --ignore-glob=keras/src/backend/tensorflow/saved_model_test.py -k not (test_fit_with_data_adapter_grain_dataloader) and not (test_fit_with_data_adapter_grain_datast) and not (test_fit_with_data_adapter_grain_datast_with_len) and not (test_image_dataset_from_directory_binary_grain) and not (test_image_dataset_from_directory_color_modes_grain) and not (test_image_dataset_from_directory_crop_to_aspect_ratio_grain) and not (test_image_dataset_from_directory_follow_links_grain) and not (test_image_dataset_from_directory_manual_labels_grain) and not (test_image_dataset_from_directory_multiclass_grain) and not (test_image_dataset_from_directory_no_labels_grain) and not (test_image_dataset_from_directory_not_batched_grain) and not (test_image_dataset_from_directory_pad_to_aspect_ratio_grain) and not (test_image_dataset_from_directory_shuffle_grain) and not (test_image_dataset_from_directory_validation_split_grain) and not (test_sample_count_grain) and not (test_text_dataset_from_directory_binary_grain) and not (test_text_dataset_from_directory_follow_links_grain) and not (test_text_dataset_from_directory_manual_labels_grain) and not (test_text_dataset_from_directory_multiclass_grain) and not (test_text_dataset_from_directory_not_batched_grain) and not (test_text_dataset_from_directory_standalone_grain) and not (test_text_dataset_from_directory_validation_split_grain) and not (test_keras_imports) and not (test_reloading_default_saved_model)
30001 3797431 3425989 15:59 00:00:00 /nix/store/iizfzmqjdfvbyx82gqw92glb5zl6m4my-python3-3.12.12/bin/python3.12 -m pytest --ignore-glob=keras/src/layers/preprocessing/data_layer_test.py --ignore-glob=keras/src/layers/preprocessing/image_preprocessing/resizing_test.py --ignore-glob=keras/src/layers/preprocessing/rescaling_test.py --ignore-glob=keras/src/trainers/data_adapters/grain_dataset_adapter_test.py --ignore-glob=keras/src/export/onnx_test.py --ignore-glob=integration_tests/dataset_tests --ignore-glob=keras/src/applications/applications_test.py --ignore-glob=integration_tests/jax_custom_fit_test.py --ignore-glob=integration_tests/tf_distribute_training_test.py --ignore-glob=integration_tests/torch_custom_fit_test.py --ignore-glob=integration_tests/torch_workflow_test.py --ignore-glob=keras/src/backend/tensorflow/saved_model_test.py -k not (test_fit_with_data_adapter_grain_dataloader) and not (test_fit_with_data_adapter_grain_datast) and not (test_fit_with_data_adapter_grain_datast_with_len) and not (test_image_dataset_from_directory_binary_grain) and not (test_image_dataset_from_directory_color_modes_grain) and not (test_image_dataset_from_directory_crop_to_aspect_ratio_grain) and not (test_image_dataset_from_directory_follow_links_grain) and not (test_image_dataset_from_directory_manual_labels_grain) and not (test_image_dataset_from_directory_multiclass_grain) and not (test_image_dataset_from_directory_no_labels_grain) and not (test_image_dataset_from_directory_not_batched_grain) and not (test_image_dataset_from_directory_pad_to_aspect_ratio_grain) and not (test_image_dataset_from_directory_shuffle_grain) and not (test_image_dataset_from_directory_validation_split_grain) and not (test_sample_count_grain) and not (test_text_dataset_from_directory_binary_grain) and not (test_text_dataset_from_directory_follow_links_grain) and not (test_text_dataset_from_directory_manual_labels_grain) and not (test_text_dataset_from_directory_multiclass_grain) and not (test_text_dataset_from_directory_not_batched_grain) and not (test_text_dataset_from_directory_standalone_grain) and not (test_text_dataset_from_directory_validation_split_grain) and not (test_keras_imports) and not (test_reloading_default_saved_model)
30001 3797433 3425989 15:59 00:00:00 /nix/store/iizfzmqjdfvbyx82gqw92glb5zl6m4my-python3-3.12.12/bin/python3.12 -m pytest --ignore-glob=keras/src/layers/preprocessing/data_layer_test.py --ignore-glob=keras/src/layers/preprocessing/image_preprocessing/resizing_test.py --ignore-glob=keras/src/layers/preprocessing/rescaling_test.py --ignore-glob=keras/src/trainers/data_adapters/grain_dataset_adapter_test.py --ignore-glob=keras/src/export/onnx_test.py --ignore-glob=integration_tests/dataset_tests --ignore-glob=keras/src/applications/applications_test.py --ignore-glob=integration_tests/jax_custom_fit_test.py --ignore-glob=integration_tests/tf_distribute_training_test.py --ignore-glob=integration_tests/torch_custom_fit_test.py --ignore-glob=integration_tests/torch_workflow_test.py --ignore-glob=keras/src/backend/tensorflow/saved_model_test.py -k not (test_fit_with_data_adapter_grain_dataloader) and not (test_fit_with_data_adapter_grain_datast) and not (test_fit_with_data_adapter_grain_datast_with_len) and not (test_image_dataset_from_directory_binary_grain) and not (test_image_dataset_from_directory_color_modes_grain) and not (test_image_dataset_from_directory_crop_to_aspect_ratio_grain) and not (test_image_dataset_from_directory_follow_links_grain) and not (test_image_dataset_from_directory_manual_labels_grain) and not (test_image_dataset_from_directory_multiclass_grain) and not (test_image_dataset_from_directory_no_labels_grain) and not (test_image_dataset_from_directory_not_batched_grain) and not (test_image_dataset_from_directory_pad_to_aspect_ratio_grain) and not (test_image_dataset_from_directory_shuffle_grain) and not (test_image_dataset_from_directory_validation_split_grain) and not (test_sample_count_grain) and not (test_text_dataset_from_directory_binary_grain) and not (test_text_dataset_from_directory_follow_links_grain) and not (test_text_dataset_from_directory_manual_labels_grain) and not (test_text_dataset_from_directory_multiclass_grain) and not (test_text_dataset_from_directory_not_batched_grain) and not (test_text_dataset_from_directory_standalone_grain) and not (test_text_dataset_from_directory_validation_split_grain) and not (test_keras_imports) and not (test_reloading_default_saved_model)

So, it seems that the latest glibc (2.42) brought by the latest staging-next iteration does not play well with cuda 12.8.
This at least breaks bitsandbytes.

See more details here: https://github.com/NixOS/nixpkgs/pull/483507

Thanks for the reviews connor (burnt/out) (UTC-8).

Btw, if you have a minute for the cuda-bindings PR, that would help me unblocking torch.
https://github.com/NixOS/nixpkgs/pull/482638

Hello everyone!

I want to share some news regarding the progress yorik.sar and I made on the infra. Some were already communicated, but it doesn't work to summarize everything.

• We have been building and caching all cudaSupport-sensitive nixpkgs packages on both stable and unstable nixpkgs channels on our Hydra instance for a few weeks. You can check the cuda-packages-unstable and cuda-packages-stable jobsets.
• All gpuCheck instances across nixpkgs are now automatically and exhaustively collected and built in Hydra as well. See the cuda-gpu-checks-unstable and cuda-gpu-checks-stable jobsets. For a reminder, gpuChecks are derivations that run some package tests that need access to a physical GPU.
• Regarding package maintenance and updates, recent times were hectic as nixpkgs received several breaking changes since the beginning of 2026 (recursing into python314Packages, update to GCC 15 and more staging-next treats). Besides, here are notable ones:
  • [MERGED] onnxruntime update to 1.23.1
  • [MERGED] triton-bin 3.6.0 + torch-bin 2.10.0
  • [WIP] triton 3.6.0 + torch 2.10.0
  • [WIP] opencv 4.13.0. This one should allow us to migrate to CUDA 13 treewide.
  • [WIP] jax 0.9.0
  • [WIP] vllm 0.14.1
  • [MERGED] mistral-rs 0.7.0
• This is still an idea, but my plan is to create two cuda-specific nix channels: nixos-unstable-cuda and nixos-stable-cuda where we could ensure that a curated set of package builds and tests are succesful (release blockers). I would be glad to hearing your feedback on this idea.

We're trying our best to move things forward as fast as possible. Unfortunately, time and compute resources are limited, so it's never fast enough 😅

Have a nice day!

Hello everyone!

I want to share some news regarding the progress yorik.sar, connor (burnt/out) (UTC-8), SomeoneSerge (back on matrix) and I made recently. Some were already communicated, but it doesn't work to summarize everything.

• We have been building and caching all cudaSupport-sensitive nixpkgs packages on both stable and unstable nixpkgs channels on our Hydra instance for a few weeks. You can check the cuda-packages-unstable and cuda-packages-stable jobsets.
• All gpuCheck instances across nixpkgs are now automatically and exhaustively collected and built in Hydra as well. See the cuda-gpu-checks-unstable and cuda-gpu-checks-stable jobsets. For a reminder, gpuChecks are derivations that run some package tests that need access to a physical GPU.
• Regarding package maintenance and updates, recent times were hectic as nixpkgs received several breaking changes since the beginning of 2026 (recursing into python314Packages, update to GCC 15 and more staging-next treats). Besides, here are notable ones:
  • [MERGED] onnxruntime update to 1.23.1
  • [MERGED] triton-bin 3.6.0 + torch-bin 2.10.0
  • [WIP] triton 3.6.0 + torch 2.10.0
  • [WIP] opencv 4.13.0. This one should allow us to migrate to CUDA 13 treewide.
  • [WIP] jax 0.9.0
  • [WIP] vllm 0.14.1
  • [MERGED] mistral-rs 0.7.0
• This is still an idea, but my plan is to create two cuda-specific nix channels: nixos-unstable-cuda and nixos-stable-cuda where we could ensure that a curated set of package builds and tests are succesful (release blockers). I would be glad to hearing your feedback on this idea.

We're trying our best to move things forward as fast as possible. Unfortunately, time and compute resources are limited, so it's never fast enough 😅

Have a nice day!

Hello everyone!

I want to share some news regarding the progress yorik.sar, connor (burnt/out) (UTC-8), SomeoneSerge (back on matrix) and I made recently. Some were already communicated, but it doesn't work to summarize everything.

• We have been building and caching all cudaSupport-sensitive nixpkgs packages on both stable and unstable nixpkgs channels on our Hydra instance for a few weeks. You can check the cuda-packages-unstable and cuda-packages-stable jobsets.
• All gpuCheck instances across nixpkgs are now automatically and exhaustively collected and built in Hydra as well. See the cuda-gpu-checks-unstable and cuda-gpu-checks-stable jobsets. For a reminder, gpuChecks are derivations that run some package tests that need access to a physical GPU.
• Regarding package maintenance and updates, recent times were hectic as nixpkgs received several breaking changes since the beginning of 2026 (recursing into python314Packages, update to GCC 15 and more staging-next treats). Besides, here are notable ones:
  • [MERGED] onnxruntime update to 1.23.1
  • [MERGED] triton-bin 3.6.0 + torch-bin 2.10.0
  • [WIP] triton 3.6.0 + torch 2.10.0
  • [WIP] opencv 4.13.0. This one should allow us to migrate to CUDA 13 treewide.
  • [WIP] jax 0.9.0
  • [WIP] vllm 0.14.1
  • [MERGED] mistral-rs 0.7.0
• This is still an idea, but my plan is to create two cuda-specific nix channels: nixos-unstable-cuda and nixos-25.11-cuda where we could ensure that a curated set of package builds and tests are succesful (release blockers). I would be glad to hearing your feedback on this idea.

We're trying our best to move things forward as fast as possible. Unfortunately, time and compute resources are limited, so it's never fast enough 😅

Have a nice day!