| 20 Sep 2021 |
 Robert Hensing (roberth) | A possible issue is that these coroutines are "viral" in the sense that any stack frame between coroutine start and yield must become part of the coroutine. Some of these are virtual method invocations and some might be calls into libraries we don't control (not sure) | 15:31:42 |
| Robert Hensing (roberth) | So those are possible challenges. I would really like for this to work and I suspect that C++20 may be a nicer requirement than boost_context, although I'm blissfully unaware of C++20 maturity | 15:33:08 |
| Robert Hensing (roberth) | You could give it a try | 15:34:26 |
 Kha | It does sound fun. But to be honest, if that one tiny function really remains the only use of coroutines since it was introduced 3 years ago, I feel like the only reasonable approach is to replace it with a hand-written state machine :) . I might try that one first. | 15:46:05 |
| Robert Hensing (roberth) | That includes such a code transformation on the nar parser and the Store interface for adding directories to the store with an arbitrary filter function, among potentially other things | 16:47:28 |
| Robert Hensing (roberth) | an async or state machine style nar parser will improve memory consumption in one case iirc, so that alone may be worth it | 16:48:16 |
 trofi | nix build parallelism question: it looks like currently nix tries very hard to download everything from internet before attempting the build for things that certainly don't have binary substitutions.
Does nix have a knob (or could it be tweaked?) to start first required build sooner to get final result faster? It's a bit odd to see 3 hours if prerequisite download and 2 hours of actual build time. Example current state:
[0/2091 built, 16/898/1354 copied (9886.6/14818.7 MiB), 4775.8/6830.3 MiB DL] fetching source from https://cache.nixos.org
| 17:50:31 |
| 21 Sep 2021 |
|  nrdxp joined the room. | 14:49:28 |
 Las | I'm probably missing something obvious, but I can't use recursive nix even with recursive-nix in my experimental-features in /etc/nix/nix.conf. It only works if I use single-user mode by specifying a store with --store. | 19:14:22 |
| Las | Nevermind, I can't get it to work at all. I get /nix/var: Permission denied when running nix build. | 19:32:26 |
| Las | I fixed my issue: It has to be in system-features too... | 20:05:48 |
| 22 Sep 2021 |
| Kha | In reply to @trofi:matrix.org
nix build parallelism question: it looks like currently nix tries very hard to download everything from internet before attempting the build for things that certainly don't have binary substitutions.
Does nix have a knob (or could it be tweaked?) to start first required build sooner to get final result faster? It's a bit odd to see 3 hours if prerequisite download and 2 hours of actual build time. Example current state:
[0/2091 built, 16/898/1354 copied (9886.6/14818.7 MiB), 4775.8/6830.3 MiB DL] fetching source from https://cache.nixos.org
AFAIR this is because substitution and building uses a common work queue constrained by max-jobs. If they used independent (and ideally differently-sized) pools, building might be able to start sooner. | 09:19:56 |
| Kha | In reply to @trofi:matrix.org
nix build parallelism question: it looks like currently nix tries very hard to download everything from internet before attempting the build for things that certainly don't have binary substitutions.
Does nix have a knob (or could it be tweaked?) to start first required build sooner to get final result faster? It's a bit odd to see 3 hours if prerequisite download and 2 hours of actual build time. Example current state:
[0/2091 built, 16/898/1354 copied (9886.6/14818.7 MiB), 4775.8/6830.3 MiB DL] fetching source from https://cache.nixos.org
* AFAIR this is because substitution and building use a common work queue constrained by max-jobs. If they used independent (and ideally differently-sized) pools, building might be able to start sooner. | 09:20:06 |
| 23 Sep 2021 |
|  tazjin joined the room. | 14:51:43 |
| 24 Sep 2021 |
|  Rev. CornWallace III (novus ordo seclorum) changed their display name from Rev. CornWallace III (sun/tzu) to Rev. CornWallace III (novus ordo seclorum). | 01:01:26 |
|  enterprisey joined the room. | 04:58:06 |
| 25 Sep 2021 |
| Kha | Robert Hensing (roberth): Ah, now I get it. The single place where coroutines are used in Nix is in decompression via libarchive, to convert between the non-blocking Sink "push" model used everywhere else and the Source "pull" model because libarchive does not have a non-blocking API. Yikes. | 09:20:32 |
| Robert Hensing (roberth) | we do have some internal use as well, but this kind of forces us to use stackful coroutines | 09:24:51 |
| Robert Hensing (roberth) | threads could be an alternative. On disk buffering probably not desirable | 09:26:57 |
| Kha | Ah right, I missed that the other direction, sinkToSource, uses it as well. Which is used, err, everyhwere. | 09:29:27 |
| Kha | * Ah right, I missed that the other direction, sinkToSource, uses coroutines as well. Which is used, err, everyhwere. | 09:29:36 |
| Robert Hensing (roberth) | mostly because nar parsing is pull based I believe | 09:31:18 |
| Kha | So libarchive needs pull. If all the parts using the push model are internal to Nix, they could be rewritten to stackless Sources using C++20 coroutines. But that would be a major refactoring. | 09:42:09 |
| Robert Hensing (roberth) | that would include the evaluator, when a filtersource predicate causes paths to be added to the store as a side effect | 10:03:31 |
| Robert Hensing (roberth) | using a library for source filtering is actually very reasonable and can cause this to happen; fetching the library source (eg hercules-ci/gitignore.nix) or doing ifd (eg nix-gitignore) | 10:05:11 |
| Robert Hensing (roberth) | (partly solved by flakes; another use case could be a library that filter sources based on a haskel cabal file or other kind of project file) | 10:05:51 |
| Robert Hensing (roberth) | a stackless evaluator would be nice btw, but I'm not sure if C++20 coroutines are the best vehicle for such a transformation | 10:07:43 |
| Robert Hensing (roberth) | * a stackless evaluator would be nice btw, but I don't know if C++20 coroutines are the best vehicle for such a transformation | 10:09:34 |
| trofi | Weekly complain about broken DNS in nix builders. Today's report: https://github.com/NixOS/nixpkgs/issues/139470. Would be nice to get a fix landed to nix in some form. | 21:57:46 |
| 26 Sep 2021 |
|  Tian Gao joined the room. | 02:42:28 |
