| 22 Nov 2022 |
 profpatsch | if you call -A foo | 20:36:27 |
| 23 Nov 2022 |
 growpotkin1 | multiple small files definitely matters in my experience.
I had a huge performance issue with Node stuff if I read too many.
In <=2.10 you'll hit max open file descriptor limit on most VMs is you don't use deepSeq, which has the expected performance impact. | 00:13:14 |
| growpotkin1 | There's a happy medium to be found between all-packages.nix and multiple composed overlays though. | 00:14:38 |
 sterni | In reply to @profpatsch:augsburg.one
newfstatat(ATFDCWD, "/home/philip/.nix-defexpr/channels/nix/fetchurl.nix", 0x7fffac509c30, ATSYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) That's it resolving <nix/fetchurl.nix> | 02:00:42 |
| growpotkin1 | I thought that nlohmann_json would improve performance as well, and I encoded thousands of packages as JSON files.
From what I could tell the problem was that the parser can't push/pop positions with JSON - you have to process the entire file as values rather than symbols. It consumed a massive amount of memory.
I'm not an expert on the evaluator but if you wanted to benchmark JSON vs Nix I have a perfect playground of a ~8,000 packages at ~800,000 versions that you can test with.
It's possible I botched something so I wouldn't treat this as "evidence" as much as an anecdote. | 03:00:09 |
| growpotkin1 | Having said that I agree with the premise of "flattening all thunks" in most package definitions | 03:02:16 |
| profpatsch | sternenseemann: oh, maybe we have to put that into its own file and call toString on it | 09:28:45 |
| profpatsch | so that nix stops doing that | 09:28:56 |
| sterni | check first why it’s doing that lol | 11:02:36 |
 Robert Hensing (roberth) | if you want to be really memory efficient, you could make the Nix C++ do a streaming parse that ignores all values, and then return an attrset with thunks that re-parse the relevant slices. Saves a ton of unused attrset AST or JSON object allocations. The unused file bits do remain in memory, but are very compact, as opposed to some pointer heavy data structure | 13:52:54 |
| 25 Nov 2022 |
|  raphi changed their profile picture. | 19:41:46 |
| 26 Nov 2022 |
| growpotkin1 | Robert Hensing (roberth): I'll be quiet about path types now. For all the footguns, this won me over /foo/bar/../baz -> /foo/baz:
nix-repl> /. + "/foo/bar/../baz"
/. + "/foo/bar/../baz"
/foo/baz
Feel like sort of a dingus for doing that with strings. For whatever reason I was convinced that the paths had to actually exist for that resolution to work, but they don't need to ( at least not in 2.10.x ).
| 18:32:29 |
| 28 Nov 2022 |
 infinisil | @room: The next meeting will take place soon, we will discuss this RFC draft - Meeting link - live stream - meeting notes | 15:20:18 |
| infinisil | John Ericson: tomberek: Joining? | 15:33:14 |
| growpotkin1 | Robert Hensing (roberth): this was my little set of wrappers for detecting and essentially tryEval catching reads to store paths. It's short.
https://github.com/aakropotkin/ak-nix/blob/main/lib/safe-reads.nix | 16:59:59 |
| growpotkin1 | The use case is scraping in impure mode to create derivations that can be rerun in pure mode or no-IFD mode in a single run. | 17:01:30 |
| growpotkin1 | It's different from Nixpkgs' approach that works with static package declarations. Really the purpose is to leverage Nix's eval and store caches for derivation generation | 17:02:53 |
| 29 Nov 2022 |
 tomberek | In reply to @infinisil:matrix.org
John Ericson: tomberek: Joining? Sorry, i missed it. Just came back from a nice Thanksgiving vacation and had a lot to do. | 00:21:40 |
| 30 Nov 2022 |
| infinisil | Aw yeah: https://github.com/nixpkgs-architecture/simple-package-paths/pull/22 | 21:34:08 |
| 1 Dec 2022 |
|  hexa changed their display name from hexa to hexa (22.11 now). | 13:08:47 |
| hexa changed their display name from hexa (22.11 now) to hexa. | 14:38:35 |
| infinisil | I'm not sure what to do about https://github.com/NixOS/nixpkgs/pull/157056. It's about allowing nixpkgs lib to be changeable, e.g. import <nixpkgs> { lib = lib.extend ...; }. I am the one who originally added support lib.extend which enables this, but I'm split about what to do here | 20:37:45 |
| infinisil | Also I'm not even sure if we should be distracted by other things right now, focusing on https://github.com/nixpkgs-architecture/simple-package-paths has worked pretty well. But there are issues like these popping up every now and then. We could also encourage people spending time on such issues to instead help with the NAT efforts. | 20:46:50 |
| infinisil | ncfavier: Oh you're the author of that PR, hello! So yeah, your PR is totally in scope for this team to address, but we are trying to focus and make progress on just a single issue right now which is the above.
And in fact these issues are distantly related in that we're working towards a standardized format for repositories to declare packages, maybe to be combined under pkgs, and this idea could be extended to lib as well. | 20:55:50 |
| Robert Hensing (roberth) | In reply to @infinisil:matrix.org
Also I'm not even sure if we should be distracted by other things right now, focusing on https://github.com/nixpkgs-architecture/simple-package-paths has worked pretty well. But there are issues like these popping up every now and then. We could also encourage people spending time on such issues to instead help with the NAT efforts. As long as we don't discourage people solving their problems. 👍️ | 21:05:11 |
| Robert Hensing (roberth) |
working towards a standardized format for repositories
I feel like to a large degree, nixpkgs will remain unique, with the possible exception of a few private, huge monorepos, or perhaps forks / distros. For instance, sharding is a pain and you'd need a seriously huge org to run into the problem that it solves. Also many repos come with the actual sources, which changes the dynamics.
| 21:09:16 |
 ncfavier | In reply to @infinisil:matrix.org
I'm not sure what to do about https://github.com/NixOS/nixpkgs/pull/157056. It's about allowing nixpkgs lib to be changeable, e.g. import <nixpkgs> { lib = lib.extend ...; }. I am the one who originally added support lib.extend which enables this, but I'm split about what to do here FWIW i don't have strong opinions about this PR anymore | 21:09:26 |
| ncfavier | i still think https://github.com/NixOS/nixpkgs/pull/160459 (extracted from the other one) should be merged though | 21:10:30 |
| infinisil | In reply to @roberthensing:matrix.org
working towards a standardized format for repositories
I feel like to a large degree, nixpkgs will remain unique, with the possible exception of a few private, huge monorepos, or perhaps forks / distros. For instance, sharding is a pain and you'd need a seriously huge org to run into the problem that it solves. Also many repos come with the actual sources, which changes the dynamics.
It could be a setting where the repository could configure whether sharding should be on or not, depending on size | 21:10:51 |
| growpotkin1 | yo I definitely want to follow the lib.extend changes | 21:11:27 |
