| 8 Apr 2025 |
 Leonardo Santiago | And I think yours is much more careful when handling thread locking and GC. I don't really understand the constraints there so maybe there's something wrong I did that caused this. | 12:26:45 |
 Robert Hensing (roberth) | If everything is on the main thread you're fine, and nix_value registers/deregisters itself with the GC just fine (if all is well), but the GC may not be happy if it needs to operate from a thread it doesn't know about. That includes allocation, so for all intents and purposes that's the whole of nix-expr that should be called from registered threads (or the main thread) only | 12:29:59 |
| Robert Hensing (roberth) | You'd get an error message along the lines of "trying to GC from unknown thread". I don't think it can cause corruption necessarily | 12:30:37 |
| Robert Hensing (roberth) | but maybe my assumption about GC roots is wrong, and my code does rely on stack scanning regardless | 12:31:07 |
| Leonardo Santiago | I think the problem may lie there, pyo3 requires that all your structs be freely movable between threads, as python is not really a single threaded interpreter, it just heavily relies on the GIL. I'm not doing anything multithreaded from the python side, much to the contrary, I'm just state.eval_file('path').get(attr) but I wouldn't say it isn't moving it to another thread either | 12:32:48 |
| Robert Hensing (roberth) | Thing is, you might get away with coincidentally not triggering GC in your other threads / stacks | 12:32:58 |
| Leonardo Santiago | One more weird detail is that the segfault is really consisten, happens everytime I run the program, but it doesn't happen always at the same place. | 12:33:59 |
| Leonardo Santiago | Sometimes it happens at nix::ExprSelect::eval, sometimes at nix::ExprAttr::eval, in the original message I think it happened at nix::ExprVar::eval | 12:34:32 |
| Leonardo Santiago | So indeed this may be related to something that is shared/passed to all of them, most likely the EvalState itself, as I may be doing something incorrectly with it. | 12:35:21 |
| Leonardo Santiago | But I don't understand what it is yet, I'll dig further | 12:35:50 |
| Leonardo Santiago | => if this was a race condition I don't think it would be this reproducible, it would sometimes fail and sometimes not. | 12:36:47 |
| Leonardo Santiago | * One more weird detail is that the segfault is really consistent, happens everytime I run the program, but it doesn't happen always at the same place. | 12:36:55 |
| Robert Hensing (roberth) | yeah | 12:37:53 |
| Leonardo Santiago | Guess what? Setting ulimit -s unlimited made it work. | 16:29:35 |
| Leonardo Santiago | It was an uncaught stack overflow. | 16:29:45 |
| Leonardo Santiago | Didn't even occur to me until now. | 16:29:57 |
| 10 Apr 2025 |
| Leonardo Santiago | @roberth how does nix circunvent this issue in their main binary? I see I can try leveraging ld's -z stack_size=X but it only seems to work if you set it in the entry point elf binary, which I can't do as it's python! I didn't want to bleed this problem elsewhere, like force people to set ulimit -s unlimited, but I don't see many other ways around it, and surely nix has had to deal with this; though it is the elf entry point. Any tips or hints? | 13:21:26 |
| Leonardo Santiago | There's the possiblity of spawning a new thread with an increased stack size, but that adds the context switching overhead to every nix evaluation, which is something I'd like to avoid if possible. | 13:22:44 |
| Leonardo Santiago | * @roberth how does nix circunvent this issue in their main binary? I see I can try leveraging ld's -z stack_size=X but it only seems to work if you set it in the entry point elf binary, which I can't do as it's python and I'm merely offering a .so extension library. I didn't want to bleed this problem elsewhere, like force people to set ulimit -s unlimited, but I don't see many other ways around it, and surely nix has had to deal with this; though it is the elf entry point. Any tips or hints? | 13:23:25 |
| Leonardo Santiago | There's also the possibility of using setrlimit to increase the process's own stack size, that seems to me like the most graceful solution | 14:09:58 |
| Leonardo Santiago | And indeed, that seems the solution that nix's binary uses, src/nix/main.cc calls nix::setStackSize(64MB) which internally calls setrlimit with that value. Awesome to know, most likely I'll try going down this route. | 14:12:10 |
| 15 Apr 2025 |
| Leonardo Santiago | I notice I may be bringing all the stupid problems and ideas to the chat, but would it be possible to static link against the nix C libraries? | 15:55:05 |
| Leonardo Santiago | * I notice I may be the onebringing all the stupid problems and ideas to the chat, but would it be possible to static link against the nix C libraries? | 15:56:12 |
| Leonardo Santiago | * I notice I may be the one bringing all the stupid problems and ideas to the chat, but would it be possible to static link against the nix C libraries? | 15:56:14 |
| Leonardo Santiago | I'm trying to optimize the performance of a custom eval cache using the C API I wrote and the cache hit is at around 20ms, which seems very good, but knowing what it does I know it's not really that impressive, since it's hashing some 20 files and querying on a sqlite file, 20ms is actually very poor performance if you consider it. | 15:57:49 |
| Leonardo Santiago | I tried perf recording and most of the time (15ms~ish) seems to be spent on do_lookup_x, which seems to be a libc function related to finding the dynamic libraries. | 15:59:04 |
| Leonardo Santiago | * I tried perf recording and most of the time (15ms~ish) seems to be spent on do_lookup_x, which seems to be a libc function related to finding the dynamic libraries, and there are ~66 linked libraries, most related to nix stuff like libaws-c-sdkutils.so.1.0.0 which shouldn't even be used in this case but are loaded anyway before the program starts. | 16:01:47 |
| Leonardo Santiago | If I remove most of the C API usage and just compile a simple binary to query the sqlite file it reduces down to 4 linked libraries and returns in ~6ms~ in --release mode, which seems to hint that indeed most of the time is spent finding the libraries | 16:03:47 |
| Leonardo Santiago | I'm using pkg-config to find and link against the C libraries, and it seems to have a .statik(true) option to static link each of them, but the moment I try to set it, it complains that it can't find them:
The file `nix-expr-c.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory.
PKG_CONFIG_PATH_FOR_TARGET contains the following:
- /nix/store/f2q12pxzicc2739nsxg35yfzqqja1jbg-python3-3.12.8-env/lib/pkgconfig
- /nix/store/7qxaap76irik1jqmqzgjwkrw3qs7mpcw-nix-store-c-2.27.0-dev/lib/pkgconfig
- /nix/store/wizqfb99xs3f4fh856bgp5qzvs76scbr-nix-util-c-2.27.0-dev/lib/pkgconfig
- /nix/store/9c9h0zbqa9xfk6pk4387l1h9pj28r692-nix-util-2.27.0-dev/lib/pkgconfig
- /nix/store/b5fg5mqb5y1r1yvzf9paql1cr1iqy4ma-boost-1.81.0-dev/lib/pkgconfig
- /nix/store/47zbszclyyy55nb5z7dvpcki4x5g99v4-libarchive-3.7.7-dev/lib/pkgconfig
- /nix/store/rwrb0afm7z79720kb2kb84vyzcbslc5i-attr-2.5.2-dev/lib/pkgconfig
- /nix/store/gsddsc9hs8l3j26r5958xcc5hhdf6hlf-acl-2.3.2-dev/lib/pkgconfig
- /nix/store/c303g1m646jv9rir4zv49q1ggphsh9b5-nlohmann_json-3.11.3/share/pkgconfig
- /nix/store/n29xgyyb4ig1sc275j8d9h7blx3g3jam-nix-store-2.27.0-dev/lib/pkgconfig
- /nix/store/v8paqfgzs6gyznsxwmgwa798nr10yk11-nix-util-2.27.0-dev/lib/pkgconfig
- /nix/store/xlw57h1x6r9l1kj8f25ij0n4j22i094n-nix-expr-c-2.27.0-dev/lib/pkgconfig
- /nix/store/x980i1fsgak6392wj9jxnr7d1yzcmpsd-nix-store-c-2.27.0-dev/lib/pkgconfig
- /nix/store/1fxkf2bvj1mcvrydx053bw7kyqnnplkg-nix-store-2.27.0-dev/lib/pkgconfig
- /nix/store/0iv0kkf8hyzjv7rnxhbkvg8s0gqnk3h7-nix-expr-2.27.0-dev/lib/pkgconfig
- /nix/store/sdx89ykxak1rdzxlg6qpb70n5bar354m-nix-fetchers-2.27.0-dev/lib/pkgconfig
- /nix/store/6mn1mdcvv6rgyj8q2wh5q3v0riv3z3z1-boehm-gc-8.2.8-dev/lib/pkgconfig
- /nix/store/ah7ylnwnsaibh71dnkxranzqahypy714-nix-util-c-2.27.0-dev/lib/pkgconfig
| 16:06:07 |
| Leonardo Santiago | * I'm using pkg-config to find and link against the C libraries inside the build.rs, and it seems to have a .statik(true) option to static link each of them, but the moment I try to set it, it complains that it can't find them:
The file `nix-expr-c.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory.
PKG_CONFIG_PATH_FOR_TARGET contains the following:
- /nix/store/f2q12pxzicc2739nsxg35yfzqqja1jbg-python3-3.12.8-env/lib/pkgconfig
- /nix/store/7qxaap76irik1jqmqzgjwkrw3qs7mpcw-nix-store-c-2.27.0-dev/lib/pkgconfig
- /nix/store/wizqfb99xs3f4fh856bgp5qzvs76scbr-nix-util-c-2.27.0-dev/lib/pkgconfig
- /nix/store/9c9h0zbqa9xfk6pk4387l1h9pj28r692-nix-util-2.27.0-dev/lib/pkgconfig
- /nix/store/b5fg5mqb5y1r1yvzf9paql1cr1iqy4ma-boost-1.81.0-dev/lib/pkgconfig
- /nix/store/47zbszclyyy55nb5z7dvpcki4x5g99v4-libarchive-3.7.7-dev/lib/pkgconfig
- /nix/store/rwrb0afm7z79720kb2kb84vyzcbslc5i-attr-2.5.2-dev/lib/pkgconfig
- /nix/store/gsddsc9hs8l3j26r5958xcc5hhdf6hlf-acl-2.3.2-dev/lib/pkgconfig
- /nix/store/c303g1m646jv9rir4zv49q1ggphsh9b5-nlohmann_json-3.11.3/share/pkgconfig
- /nix/store/n29xgyyb4ig1sc275j8d9h7blx3g3jam-nix-store-2.27.0-dev/lib/pkgconfig
- /nix/store/v8paqfgzs6gyznsxwmgwa798nr10yk11-nix-util-2.27.0-dev/lib/pkgconfig
- /nix/store/xlw57h1x6r9l1kj8f25ij0n4j22i094n-nix-expr-c-2.27.0-dev/lib/pkgconfig
- /nix/store/x980i1fsgak6392wj9jxnr7d1yzcmpsd-nix-store-c-2.27.0-dev/lib/pkgconfig
- /nix/store/1fxkf2bvj1mcvrydx053bw7kyqnnplkg-nix-store-2.27.0-dev/lib/pkgconfig
- /nix/store/0iv0kkf8hyzjv7rnxhbkvg8s0gqnk3h7-nix-expr-2.27.0-dev/lib/pkgconfig
- /nix/store/sdx89ykxak1rdzxlg6qpb70n5bar354m-nix-fetchers-2.27.0-dev/lib/pkgconfig
- /nix/store/6mn1mdcvv6rgyj8q2wh5q3v0riv3z3z1-boehm-gc-8.2.8-dev/lib/pkgconfig
- /nix/store/ah7ylnwnsaibh71dnkxranzqahypy714-nix-util-c-2.27.0-dev/lib/pkgconfig
| 16:06:18 |
