!VRULIdgoKmKPzJZzjj:nixos.org

Nix Hackers

779 Members
For people hacking on the Nix package manager itself168 Servers

Load older messages


SenderMessageTime
25 Dec 2024
@trofi:matrix.orgtrofi * Attempted to improve an error message as https://github.com/NixOS/nix/pull/12105. Looks like nix lotk man output when built a goal with man output. I wonder if outputChecks does not specify some kind of dependency for string references luke plain "man". 22:07:19
26 Dec 2024
@10leej:matrix.org@10leej:matrix.org left the room.01:41:45
@gasu:matrix.orggasu joined the room.05:33:52
@trofi:matrix.orgtrofi Fun fact: -D_GLIBCXX_DEBUG crashes nix-env as std::stable_sort predicate does not maintain !(a < a) property: https://github.com/NixOS/nix/issues/12106 07:46:10
@xokdvium:matrix.orgxokdvium

What should builtins.sort (a: b: true) [ 1 2 3 ]even do? Should this be a nix language-level UB? Currently manual does not provide any preconditions https://nix.dev/manual/nix/2.25/language/builtins.html?highlight=builtins.sort#builtins-sort.

This would be a tricky thing to fix. Failing gracefully would require validating the strict weak ordering or using sorting algorithms that do not explode when the comparator is broken. std::sort can easily explode since it's something like qsort. stable_sort is actually a combination of insertion sort with merge sort on top. https://gcc.gnu.org/onlinedocs/libstdc++/libstdc++-html-USERS-4.4/a01347.html#l03409

I think using a custom merge sort would be safe even for broken predicates and won't crash nix evaluator, but this would still need to be nix UB (such calls to sort are ill-formed and diagnosing it is prohibitively expensive).

Either way this calls for a custom sorting implementation that does not assume that the provided predicate is strict weak ordering.

10:17:33
@xokdvium:matrix.orgxokdviumWhat's more worrying is the fact that somewhere in nixpkgs there are broken comparator predicates.10:27:59
@aloisw:julia0815.dealoisw
In reply to @xokdvium:matrix.org

What should builtins.sort (a: b: true) [ 1 2 3 ]even do? Should this be a nix language-level UB? Currently manual does not provide any preconditions https://nix.dev/manual/nix/2.25/language/builtins.html?highlight=builtins.sort#builtins-sort.

This would be a tricky thing to fix. Failing gracefully would require validating the strict weak ordering or using sorting algorithms that do not explode when the comparator is broken. std::sort can easily explode since it's something like qsort. stable_sort is actually a combination of insertion sort with merge sort on top. https://gcc.gnu.org/onlinedocs/libstdc++/libstdc++-html-USERS-4.4/a01347.html#l03409

I think using a custom merge sort would be safe even for broken predicates and won't crash nix evaluator, but this would still need to be nix UB (such calls to sort are ill-formed and diagnosing it is prohibitively expensive).

Either way this calls for a custom sorting implementation that does not assume that the provided predicate is strict weak ordering.

It currently triggers C++ UB.
10:29:30
@xokdvium:matrix.orgxokdvium

Yup, that's clearly a huge problem and this should be fixed by not using standard sorting algorithms:

Either way this calls for a custom sorting implementation that does not assume that the provided predicate is strict weak ordering.

What I'm trying to say is that this should still be UB in the nix language, but not crash and burn the evaluator.

10:31:14
@aloisw:julia0815.dealoiswI don't think there should be UB in the Nix language, unspecified result should be good enough.10:32:15
@xokdvium:matrix.orgxokdvium Yeah, that makes more sense. Though I hope there could be a better way to find the broken usages in nixpkgs. Unspecified result makes the eval non-deterministic by nature. Does this mean that if this bug gets fixed and replaced with unspecified order instead of C++ UB nixpkgs would suddenly eval differently? 10:37:58
@aloisw:julia0815.dealoiswI don't think it is possible to determine in general whether a given comparator is broken. And I think literally depending on C++ UB is one of the cases where breaking evaluation compatibility is acceptable.10:40:06
@xokdvium:matrix.orgxokdviumAt the very least we could first trace the problematic usages with a custom cppnix build and evaluate the scope of the disaster. Checking strict weak order properties by hand + random shuffling before the sort should surface at least some problems.10:43:29
@xokdvium:matrix.orgxokdvium Though random shuffling would have to preserve the order, since the sort is stable. 10:44:30
@aloisw:julia0815.dealoiswHow do you know what the problematic usages are in the first place?10:44:43
@xokdvium:matrix.orgxokdvium

See the original https://github.com/NixOS/nix/issues/12106#issue-2759417825 issue. I just reproduced the problem on my end on nixpkgs master (9a90c6b6218860ab29f79ecd6213c3a547c15c7e):

../nixpkgs-check-strict-weak-order/result/bin/nix-env -qa --json --arg config 'import pkgs/top-level/packages-config.nix'
/nix/store/4KRAB2H0HD4WVXXMSCXRW21PL77J4I7J-gcc-13.3.0/include/c++/13.3.0/bits/stl_algo.h:5117:
In function:
    void std::stable_sort(_RAIter, _RAIter, _Compare) [with _RAIter =
    nix::Value**; _Compare = nix::prim_sort(EvalState&, PosIdx, Value**,
    Value&)::<lambda(nix::Value*, nix::Value*)>]

Error: comparison doesn't meet irreflexive requirements, assert(!(a < a)).

Objects involved in the operation:
    instance "functor" @ 0x7ffff3cb8c90 {
      type = nix::prim_sort(nix::EvalState&, nix::PosIdx, nix::Value**, nix::Value&)::{lambda(nix::Value*, nix::Value*)#1};
    }
    iterator::value_type "ordered type"  {
      type = nix::Value*;
    }
[1]    2675868 abort (core dumped)  ../nixpkgs-check-strict-weak-order/result/bin/nix-env -qa --json --arg config
11:14:58
@xokdvium:matrix.orgxokdviumI'm not sure that if that actually explodes with normal usage though11:26:03
@xokdvium:matrix.orgxokdvium

A quick glance also shows that this gets triggered in tests.coq, so this is not a one-off problem:

nix-instantiate -A tests.coq

/nix/store/4KRAB2H0HD4WVXXMSCXRW21PL77J4I7J-gcc-13.3.0/include/c++/13.3.0/bits/stl_algo.h:5117:
In function:
    void std::stable_sort(_RAIter, _RAIter, _Compare) [with _RAIter =
    nix::Value**; _Compare = nix::prim_sort(EvalState&, PosIdx, Value**,
    Value&)::<lambda(nix::Value*, nix::Value*)>]

Error: comparison doesn't meet irreflexive requirements, assert(!(a < a)).

Objects involved in the operation:
    instance "functor" @ 0x7ffc7a9bef10 {
      type = nix::prim_sort(nix::EvalState&, nix::PosIdx, nix::Value**, nix::Value&)::{lambda(nix::Value*, nix::Value*)#1};
    }
    iterator::value_type "ordered type"  {
      type = nix::Value*;
    }
[1]    2713634 abort (core dumped)  nix-instantiate -A tests.coq
11:44:28
@trofi:matrix.orgtrofihttps://bpa.st/raw/JNGQ allows a bit better exception reporting. like https://bpa.st/raw/NBNQ11:52:31
@xokdvium:matrix.orgxokdvium Thanks! I guess it's just up to someone to rain nixmas gifts of bugfixes on nixpkgs. 11:59:34
@elikoga:matrix.orgelikoga changed their display name from elikoga to elikoga (@38c3 📞448{0,1}.15:21:56
@elikoga:matrix.orgelikoga changed their display name from elikoga (@38c3 📞448{0,1} to elikoga (@38c3 📞448{0,1}).15:26:08
@elikoga:matrix.orgelikoga changed their display name from elikoga (@38c3 📞448{0,1}) to elikoga (@38c3 📞488{0,1}).15:26:50
@trofi:matrix.orgtrofiExample fix: https://github.com/NixOS/nixpkgs/pull/36836615:49:49
@interacsion:matrix.orgMai joined the room.18:59:31
@czl92783719:mozilla.orgCarter Zhang[M] joined the room.22:55:39
@czl92783719:mozilla.orgCarter Zhang[M] Hello everyone, I just sudo apt install nix-bin and followed the steps mentioned in cat /usr/share/doc/nix-bin/README.Debian. What's next? 22:56:56
@trofi:matrix.orgtrofi* Example fix: https://github.com/NixOS/nixpkgs/pull/368366 (and also https://github.com/NixOS/nixpkgs/pull/368429 and https://github.com/NixOS/nixpkgs/pull/368433).23:15:07
27 Dec 2024
@perchun:matrix.orgPerchun Pak
In reply to @czl92783719:mozilla.org
Hello everyone, I just sudo apt install nix-bin and followed the steps mentioned in cat /usr/share/doc/nix-bin/README.Debian. What's next?
Nix pills are a good way to learn nix
https://nixos.org/guides/nix-pills/
06:36:02
@czl92783719:mozilla.orgCarter Zhang[M]
In reply to @perchun:matrix.org
Nix pills are a good way to learn nix
https://nixos.org/guides/nix-pills/
Thanks
06:36:39
@raitobezarius:matrix.orgraitobezarius changed their display name from raitobezarius to raitobezarius (DECT: 3538 / EPVPN 2681).07:30:29

Show newer messages


Back to Room ListRoom Version: 6