 | Nix Hackers | 893 Members |
| For people hacking on the Nix package manager itself | 191 Servers |
| Nix Hackers | 893 Members |
| For people hacking on the Nix package manager itself | 191 Servers |
| Sender | Message | Time |
|---|---|---|
| 19 Sep 2021 | ||
 sterni | as long as the referenced paths come from derivations | 18:58:30 |
 Robert Hensing (roberth) | or from sources (path values, filtersource, that kind of thing) | 18:59:05 |
 Kha | Boost coroutines have led to some GC issues (all fixed now though?) and are an issue on Cygwin. Has anyone thought about replacing them with C++20 coroutines? It might be too early to mandate C++20 support, but perhaps it would be worth supporting both impls for some time... | 21:16:06 |
| Robert Hensing (roberth) | Kha: getting boehmgc and any kind of coroutines to play nice together is non-trivial. At least the boost coroutines expose some implementation details that allow us to inform boehmgc properly (although the way we do it isn't optimal yet). Any coroutine implementation will confuse boehmgc because it doesn't handle on its own the case where the stack pointer isn't in any known thread stacks. C++20 coroutines do not seem to have an allocator parameter or anything similar that would allow us to pass this info to the gc. I haven't delved deeply into this, but from the information I could find, no such interface exists | 21:52:00 |
| Robert Hensing (roberth) | any coroutine integration will have to solve two problems: 1. making boehmgc accept the 'weird' stack pointers; 2. making sure the coroutine stack gets scanned by the gc. The current integration uses independent solutions for both, which is not optimal. A C++20 implementation has to solve both problems too | 21:54:33 |
| Robert Hensing (roberth) | there might be a way in which Nix could be refactored so that we only need to scan the evaluator's stack, so that we can configure boehm to scan only that stack. This would require a significant amount of communication between threads and an entirely new interface for working with this isolated evaluator. Such an interface would actually be useful for tracking Value provenance in a few locations where this is currently lacking, which improves error messages, but that's another topic I guess. Point is, with this much thread synchronization, it's easier and probably faster to just use threads instead of coroutines in the first place | 22:07:02 |
| Robert Hensing (roberth) | coroutines are only used for working with streams in a synchronous style. In some cases this can be replaced by code that written in an async style. I'm not sure if we even have interactions with synchronous style libraries. | 22:09:41 |
| 20 Sep 2021 | ||
 Rev. CornWallace III (sun/tzu) changed their display name from CornWallace to Rev. CornWallace III. | 06:45:07 | |
 tomberek | While interacting directly with the nix-daemon I noticed that the response to exceptions/errors are sent to the client twice. Is this expected? This is a wopIsValidPath. daemon.cc looks like it sends STDERR_ERROR for the operation, and then sends another in the try-catch for the connection. | 06:54:33 |
 NinjaTrappeur joined the room. | 09:25:17 | |
 Eelco 😴 changed their display name from Eelco Dolstra to Eelco 😴. | 11:26:41 | |
| Kha | In reply to @roberthensing:matrix.org Ah, but that's the thing: C++20 coroutines are specified to be stackless: "they suspend execution by returning to the caller and the data that is required to resume execution is stored separately from the stack" Custom allocation can be done via a custom https://en.cppreference.com/w/cpp/language/coroutines | 12:19:39 |
| Robert Hensing (roberth) | does that guarantee that the stack pointer remains in the thread stack region? | 12:22:32 |
| Robert Hensing (roberth) | and if it does, it doesn't seem like this could call into arbitrary code, which does seem to be something we need to make the whole sink/source translation business work | 12:23:38 |
| Robert Hensing (roberth) | more specifically, that arbitrary code would not be capable of knowing how to yield | 12:24:48 |
| Robert Hensing (roberth) | (I mean any C++ code we wrote, not the security almost-term "arbitrary code") | 12:26:39 |
| Kha | Does Nix even use more than that single coroutine in serialise.cc, which does not have a nested yield? In any case, afaiui you cannot have directly yield from inside a nested call with these coroutines, but you can yield to the surrounding coroutine, which then yields to the caller/next coroutine, etc. | 12:34:20 |
| Kha | It's like Python before yield from | 12:34:31 |
| Kha | * Does Nix even use more than that single coroutine in serialise.cc, which does not have a nested yield? In any case, afaiui you cannot directly yield from inside a nested call with these coroutines, but you can yield to the surrounding coroutine, which then yields to the caller/next coroutine, etc. | 12:35:06 |
| Kha | There is absolutely no stack pointer manipulation with this approach | 12:35:38 |
| Rev. CornWallace III (sun/tzu) changed their display name from Rev. CornWallace III to Rev. CornWallace III (tzu/tzi). | 15:22:23 | |
| Rev. CornWallace III (sun/tzu) changed their display name from Rev. CornWallace III (tzu/tzi) to Rev. CornWallace III (sun/tzu). | 15:24:44 | |
| Robert Hensing (roberth) | Kha: I'm going to trust you on that | 15:29:53 |
| Robert Hensing (roberth) | A possible issue is that these coroutines are "viral" in the sense that any stack frame between coroutine start and yield must become part of the coroutine. Some of these are virtual method invocations and some might be calls into libraries we don't control (not sure) | 15:31:42 |
| Robert Hensing (roberth) | So those are possible challenges. I would really like for this to work and I suspect that C++20 may be a nicer requirement than boost_context, although I'm blissfully unaware of C++20 maturity | 15:33:08 |
| Robert Hensing (roberth) | You could give it a try | 15:34:26 |
| Kha | It does sound fun. But to be honest, if that one tiny function really remains the only use of coroutines since it was introduced 3 years ago, I feel like the only reasonable approach is to replace it with a hand-written state machine :) . I might try that one first. | 15:46:05 |
| Robert Hensing (roberth) | That includes such a code transformation on the nar parser and the Store interface for adding directories to the store with an arbitrary filter function, among potentially other things | 16:47:28 |
| Robert Hensing (roberth) | an async or state machine style nar parser will improve memory consumption in one case iirc, so that alone may be worth it | 16:48:16 |
 trofi |
Does nix have a knob (or could it be tweaked?) to start first required build sooner to get final result faster? It's a bit odd to see 3 hours if prerequisite download and 2 hours of actual build time. Example current state: [0/2091 built, 16/898/1354 copied (9886.6/14818.7 MiB), 4775.8/6830.3 MiB DL] fetching source from https://cache.nixos.org | 17:50:31 |