| 31 Jul 2024 |
 adisbladis | https://github.com/nix-community/emacs-overlay/issues/391 could be addressed by registering the unpacked elpa hashes instead | 04:07:31 |
 linj | In reply to @adis:blad.is
Thinking out loud on generators:
It might be a good idea to embed melpa recipes in our codegen.
I don't know how representative these sizes are so take them with a grain of salt:
len("""(request :repo "tkf/emacs-request" :fetcher github :files ("request.el"))""") -> 73
len("0h4jqg98px9dqqvjp08vi2z1lhmk0ca59lnrcl96bi7gkkj3jiji") -> 52
For the low cost of 21 bytes extra storage we can avoid fetching the recipe, and can also save a bit on memory on the FOD that's created to fetch it.
It might be that some recipes are too big, but I think we're probably better of embedding them by default.
The benefit of embedding recipes seems small to me. About the disadvantage, replacing a hash with the encoded content of a recipe increases the size of Nixpkgs. Recently, there are discussion about the size of Nixpkgs. Another thing is that it makes it harder to override the recipe. There is only one of these overrides in the current Nixpkgs so this is not a big issue I guess. | 05:44:55 |
| linj | In reply to @adis:blad.is
Thinking out loud on generators:
It might be a good idea to embed melpa recipes in our codegen.
I don't know how representative these sizes are so take them with a grain of salt:
len("""(request :repo "tkf/emacs-request" :fetcher github :files ("request.el"))""") -> 73
len("0h4jqg98px9dqqvjp08vi2z1lhmk0ca59lnrcl96bi7gkkj3jiji") -> 52
For the low cost of 21 bytes extra storage we can avoid fetching the recipe, and can also save a bit on memory on the FOD that's created to fetch it.
It might be that some recipes are too big, but I think we're probably better of embedding them by default.
* The benefit of embedding recipes seems small to me. About the disadvantage, replacing a hash with the encoded content of a recipe increases the size of Nixpkgs. Recently, there are discussion about the size of Nixpkgs. Another thing is that now melpaBuild is used in almost all manual packages, so that use case should be supported, too. | 05:46:44 |
| linj | * The benefit of embedding recipes seems small to me. About the disadvantage, replacing a hash with the content of a recipe increases the size of Nixpkgs. Recently, there are discussion about the size of Nixpkgs. Another thing is that now melpaBuild is used in almost all manual packages, so that use case should be supported, too. | 05:48:20 |
| linj | * The benefit of embedding recipes seems small to me. About the disadvantage, replacing a hash with the content of a recipe increases the size of Nixpkgs. Recently, there are discussions about the size of Nixpkgs. Another thing is that now melpaBuild is used in almost all manual packages, so that use case should be supported, too. | 05:48:37 |
| linj | turns out the increased size should not be a big issue: 0.43MB - 0.30MB = 0.13MB | 05:59:39 |
| linj | In reply to @adis:blad.is
https://github.com/nix-community/emacs-overlay/issues/391 could be addressed by registering the unpacked elpa hashes instead Yeah, we can use fetchzip for tarballs to get a unchanged hash. However, then we have to tar them again because elpaBuild wants tarballs. | 06:06:29 |
| linj | I think generating nix code instead of JSON adds unnecessary complexity. The first motivation of code generation is about performance. That was 2015 though. Things probably have changed. Are there any benchmarks about this? The second motivation is to do override, which the current code of updating MELPA shows that it can be done without code generation. https://github.com/nix-community/emacs2nix/issues/4 | 06:17:24 |
| adisbladis | In reply to @me:linj.tech
I think generating nix code instead of JSON adds unnecessary complexity. The first motivation of code generation is about performance. That was 2015 though. Things probably have changed. Are there any benchmarks about this? The second motivation is to do override, which the current code of updating MELPA shows that it can be done without code generation. https://github.com/nix-community/emacs2nix/issues/4 Agreed. Parsing/evaluating Nix code is slower than the equivalent json/toml export. | 07:18:59 |
| adisbladis | I haven't benchmarked this difference in ages and don't have numbers on it. But it's by quite a margin. | 07:19:52 |
 magic_rb | So a JSON file will parse and load quicker than the exact same thing in Nix syntax? | 07:22:38 |
| linj | I guesst usually JSON needs to be loaded and transformed to be the exact thing to generated nix code | 07:25:27 |
| linj | * I guesst the case where JSON needs to be loaded and transformed to be the exact thing to generated nix code is more interesting | 07:26:07 |
| linj | In reply to @me:linj.tech
I think generating nix code instead of JSON adds unnecessary complexity. The first motivation of code generation is about performance. That was 2015 though. Things probably have changed. Are there any benchmarks about this? The second motivation is to do override, which the current code of updating MELPA shows that it can be done without code generation. https://github.com/nix-community/emacs2nix/issues/4 There is a pr for R to switch from code gen to JSON: https://github.com/NixOS/nixpkgs/pull/328272 . The performance overhead is very small. | 07:39:48 |
| adisbladis | In reply to @magic_rb:matrix.redalder.org
So a JSON file will parse and load quicker than the exact same thing in Nix syntax? Yes. Some of that is faster parsing, some of that is strictness (json/toml values aren't thunked afaik) | 07:46:55 |
| adisbladis | I'll make a micro benchmark, gimme a minute | 07:47:38 |
| adisbladis | Loading & adding 500k numbers from a json & nix file respectively (env NIX_SHOW_STATS=1 nix-instantiate --eval --strict ./bench.nix):
let
# d = import ./out.nix;
d = builtins.fromJSON (builtins.readFile ./out.json);
in
builtins.foldl' (acc: v: acc + v) 0 (builtins.attrValues d)
{
"cpuTime": 1.6208760738372803,
"envs": {
"bytes": 24000024,
"elements": 1000001,
"number": 1000001
},
"gc": {
"heapSize": 402915328,
"totalBytes": 60013088
},
"list": {
"bytes": 4000008,
"concats": 0,
"elements": 500001
},
"nrAvoided": 500003,
"nrFunctionCalls": 1000000,
"nrLookups": 2,
"nrOpUpdateValuesCopied": 0,
"nrOpUpdates": 0,
"nrPrimOpCalls": 3,
"nrThunks": 3,
"sets": {
"bytes": 8002128,
"elements": 500130,
"number": 3
},
"sizes": {
"Attr": 16,
"Bindings": 16,
"Env": 16,
"Value": 24
},
"symbols": {
"bytes": 8502311,
"number": 500252
},
"values": {
"bytes": 12002880,
"number": 500120
}
}
{
"cpuTime": 1.1166590452194214,
"envs": {
"bytes": 24000024,
"elements": 1000001,
"number": 1000001
},
"gc": {
"heapSize": 402915328,
"totalBytes": 131455856
},
"list": {
"bytes": 4000008,
"concats": 0,
"elements": 500001
},
"nrAvoided": 3,
"nrFunctionCalls": 1000000,
"nrLookups": 4,
"nrOpUpdateValuesCopied": 0,
"nrOpUpdates": 0,
"nrPrimOpCalls": 4,
"nrThunks": 4,
"sets": {
"bytes": 8002128,
"elements": 500130,
"number": 3
},
"sizes": {
"Attr": 16,
"Bindings": 16,
"Env": 16,
"Value": 24
},
"symbols": {
"bytes": 8002311,
"number": 500252
},
"values": {
"bytes": 24002904,
"number": 1000121
}
}
| 07:51:14 |
| magic_rb | Now i wonder if a subset of nix, unsafeImportStrict could be created. Essentially treating nix code like json and erroring on anything non json like, functions and let bindings | 07:56:35 |
| magic_rb | Because json is really hard to read | 07:56:48 |
| adisbladis | Ideally TOML should be readable too, but it's a lot slower than either nix or json for the same task | 08:31:21 |
| adisbladis | Conversely I also think we should have a builtins.importJSON/builtins.importTOML that doesn't need to readFile the whole contents | 08:32:31 |
| adisbladis | In reply to @magic_rb:matrix.redalder.org
Now i wonder if a subset of nix, unsafeImportStrict could be created. Essentially treating nix code like json and erroring on anything non json like, functions and let bindings Not sure why that would be any more unsafe than the non-strict import equivalent? | 08:33:18 |
| magic_rb | In reply to @adis:blad.is
Not sure why that would be any more unsafe than the non-strict import equivalent? Because it expects a .nix file but cant fully read it, not sure, maybe a separate nixjson format would be better | 08:42:31 |
| adisbladis | Jesus christ | 08:44:00 |
| adisbladis | Why is the Nix TOML decoding so slow | 08:44:10 |
| adisbladis | I never actually knew how slow it is | 08:44:24 |
 Philip Taron (UTC-8) | In reply to @adis:blad.is
I never actually knew how slow it is I'd love to see the perf result | 19:01:05 |
| 1 Aug 2024 |
| adisbladis | In reply to @philiptaron:matrix.org
I'd love to see the perf result {
"cpuTime": 8.42892837524414,
"envs": {
"bytes": 24000024,
"elements": 1000001,
"number": 1000001
},
"gc": {
"heapSize": 402915328,
"totalBytes": 89455824
},
"list": {
"bytes": 4000008,
"concats": 0,
"elements": 500001
},
"nrAvoided": 3,
"nrFunctionCalls": 1000000,
"nrLookups": 4,
"nrOpUpdateValuesCopied": 0,
"nrOpUpdates": 0,
"nrPrimOpCalls": 4,
"nrThunks": 4,
"sets": {
"bytes": 8002128,
"elements": 500130,
"number": 3
},
"sizes": {
"Attr": 16,
"Bindings": 16,
"Env": 16,
"Value": 24
},
"symbols": {
"bytes": 8002311,
"number": 500252
},
"values": {
"bytes": 24002904,
"number": 1000121
}
}
| 00:06:11 |
| Philip Taron (UTC-8) | Yikes! | 00:15:00 |
| adisbladis | Indeed! Tbf though these files are much larger than your typical ones. | 00:17:19 |
