| 20 Sep 2021 |
 Robert Hensing (roberth) | So those are possible challenges. I would really like for this to work and I suspect that C++20 may be a nicer requirement than boost_context, although I'm blissfully unaware of C++20 maturity | 15:33:08 |
| Robert Hensing (roberth) | You could give it a try | 15:34:26 |
 Kha | It does sound fun. But to be honest, if that one tiny function really remains the only use of coroutines since it was introduced 3 years ago, I feel like the only reasonable approach is to replace it with a hand-written state machine :) . I might try that one first. | 15:46:05 |
| Robert Hensing (roberth) | That includes such a code transformation on the nar parser and the Store interface for adding directories to the store with an arbitrary filter function, among potentially other things | 16:47:28 |
| Robert Hensing (roberth) | an async or state machine style nar parser will improve memory consumption in one case iirc, so that alone may be worth it | 16:48:16 |
 trofi | nix build parallelism question: it looks like currently nix tries very hard to download everything from internet before attempting the build for things that certainly don't have binary substitutions.
Does nix have a knob (or could it be tweaked?) to start first required build sooner to get final result faster? It's a bit odd to see 3 hours if prerequisite download and 2 hours of actual build time. Example current state:
[0/2091 built, 16/898/1354 copied (9886.6/14818.7 MiB), 4775.8/6830.3 MiB DL] fetching source from https://cache.nixos.org
| 17:50:31 |
| 21 Sep 2021 |
|  nrdxp joined the room. | 14:49:28 |
 Las | I'm probably missing something obvious, but I can't use recursive nix even with recursive-nix in my experimental-features in /etc/nix/nix.conf. It only works if I use single-user mode by specifying a store with --store. | 19:14:22 |
| Las | Nevermind, I can't get it to work at all. I get /nix/var: Permission denied when running nix build. | 19:32:26 |
| Las | I fixed my issue: It has to be in system-features too... | 20:05:48 |
| 22 Sep 2021 |
| Kha | In reply to @trofi:matrix.org
nix build parallelism question: it looks like currently nix tries very hard to download everything from internet before attempting the build for things that certainly don't have binary substitutions.
Does nix have a knob (or could it be tweaked?) to start first required build sooner to get final result faster? It's a bit odd to see 3 hours if prerequisite download and 2 hours of actual build time. Example current state:
[0/2091 built, 16/898/1354 copied (9886.6/14818.7 MiB), 4775.8/6830.3 MiB DL] fetching source from https://cache.nixos.org
AFAIR this is because substitution and building uses a common work queue constrained by max-jobs. If they used independent (and ideally differently-sized) pools, building might be able to start sooner. | 09:19:56 |
| Kha | In reply to @trofi:matrix.org
nix build parallelism question: it looks like currently nix tries very hard to download everything from internet before attempting the build for things that certainly don't have binary substitutions.
Does nix have a knob (or could it be tweaked?) to start first required build sooner to get final result faster? It's a bit odd to see 3 hours if prerequisite download and 2 hours of actual build time. Example current state:
[0/2091 built, 16/898/1354 copied (9886.6/14818.7 MiB), 4775.8/6830.3 MiB DL] fetching source from https://cache.nixos.org
* AFAIR this is because substitution and building use a common work queue constrained by max-jobs. If they used independent (and ideally differently-sized) pools, building might be able to start sooner. | 09:20:06 |
| 23 Sep 2021 |
|  tazjin joined the room. | 14:51:43 |
| 24 Sep 2021 |
|  Rev. CornWallace III (novus ordo seclorum) changed their display name from Rev. CornWallace III (sun/tzu) to Rev. CornWallace III (novus ordo seclorum). | 01:01:26 |
|  enterprisey joined the room. | 04:58:06 |
| 25 Sep 2021 |
| Kha | Robert Hensing (roberth): Ah, now I get it. The single place where coroutines are used in Nix is in decompression via libarchive, to convert between the non-blocking Sink "push" model used everywhere else and the Source "pull" model because libarchive does not have a non-blocking API. Yikes. | 09:20:32 |
| Robert Hensing (roberth) | we do have some internal use as well, but this kind of forces us to use stackful coroutines | 09:24:51 |
| Robert Hensing (roberth) | threads could be an alternative. On disk buffering probably not desirable | 09:26:57 |
| Kha | Ah right, I missed that the other direction, sinkToSource, uses it as well. Which is used, err, everyhwere. | 09:29:27 |
| Kha | * Ah right, I missed that the other direction, sinkToSource, uses coroutines as well. Which is used, err, everyhwere. | 09:29:36 |
| Robert Hensing (roberth) | mostly because nar parsing is pull based I believe | 09:31:18 |
| Kha | So libarchive needs pull. If all the parts using the push model are internal to Nix, they could be rewritten to stackless Sources using C++20 coroutines. But that would be a major refactoring. | 09:42:09 |
| Robert Hensing (roberth) | that would include the evaluator, when a filtersource predicate causes paths to be added to the store as a side effect | 10:03:31 |
| Robert Hensing (roberth) | using a library for source filtering is actually very reasonable and can cause this to happen; fetching the library source (eg hercules-ci/gitignore.nix) or doing ifd (eg nix-gitignore) | 10:05:11 |
| Robert Hensing (roberth) | (partly solved by flakes; another use case could be a library that filter sources based on a haskel cabal file or other kind of project file) | 10:05:51 |
| Robert Hensing (roberth) | a stackless evaluator would be nice btw, but I'm not sure if C++20 coroutines are the best vehicle for such a transformation | 10:07:43 |
| Robert Hensing (roberth) | * a stackless evaluator would be nice btw, but I don't know if C++20 coroutines are the best vehicle for such a transformation | 10:09:34 |
| trofi | Weekly complain about broken DNS in nix builders. Today's report: https://github.com/NixOS/nixpkgs/issues/139470. Would be nice to get a fix landed to nix in some form. | 21:57:46 |
| 26 Sep 2021 |
|  Tian Gao joined the room. | 02:42:28 |
| trofi | nix error message question: https://dpaste.com/56YEHBN65.txt. Do I read it correctly that meta is not allowed as an attribute in {...} function? | 08:45:34 |
