!VRULIdgoKmKPzJZzjj:nixos.org

Nix Hackers

676 Members
For people hacking on the Nix package manager itself154 Servers

Load older messages


SenderMessageTime
24 Aug 2021
@andi:kack.itandi- Running nix-build as a regular user right now does eval + drives the build in a single process, correct? I was thinking of doing the eval in a forked child such that we can (safely?) discard all the memory from the eval that we don't have to hold on to anymore. Are there any obvious roadblocks that I'd run into? 15:13:51
@pamplemouss_:matrix.orgpamplemousse

does eval + drives the build in a single process, correct?

AFAICT, yes; And it uses threads for parallelism.

18:51:57
@pamplemouss_:matrix.orgpamplemousse andi-: FYI, I considered doing something similar to be able to fuzz the parsing / evaluation part of nix (because it's not reentrant - i.e. mutates a global state)
I don't remember exactly why I abandoned this, but speed was definitely a problem (fork/exec is much slower - but it might not be problematic in your use case).
18:55:03
@pamplemouss_:matrix.orgpamplemousse(it kinda defeated the purposed of having an "in-memory" fuzzer)18:55:31
@andi:kack.itandi-Yeah for fuzzing you probably want something much faster even thought you could fuzz multiple cases in one process (as long as you keep the history of invocations to minimize the case afterwards)18:56:05
@pamplemouss_:matrix.orgpamplemousse
In reply to @andi:kack.it
Yeah for fuzzing you probably want something much faster even thought you could fuzz multiple cases in one process (as long as you keep the history of invocations to minimize the case afterwards)
The problem is that it raises a lot of false positives this way in practice; Every "next" run is corrupted by dangling global state left by the previous...
19:02:40
@pamplemouss_:matrix.orgpamplemousse (I am really talking about nix specifically here) 19:03:01
@andi:kack.itandi-Yeah, I'd only run like 3-5 or so per fork. That is still a 5x speedup?21:08:42
@andi:kack.itandi- Another thing that we talked about earlier in #infrastructure:nixos.org : Could we tell the kernel / emulate the load in the sandbox so it only covers the load produced from the contained jobs? That topic came up as running many builds with a small number of cores might always be limited by load if the build system is load aware (e.g. ninja & gnumake with -l). lxc apparently does something like that. The open question was if the kernel is able to do that or else if we could/should do that in userspace (in the nix-daemon). 22:11:22
25 Aug 2021
@vcunat:matrix.orgVladimír ČunátI'm actually not sure it's a good idea. If the CPU is shared, strictly partitioning it doesn't sound like a good approach for overall efficiency... though it depends how exactly we'd use it.18:55:00
@trofi:matrix.orgtrofi joined the room.21:42:44
@trofi:matrix.orgtrofi set a profile picture.22:19:15
26 Aug 2021
@trofi:matrix.orgtrofi changed their display name from Sergei Trofimovich to trofi.07:41:05
@Las:matrix.orgLasDoes anyone know why Recursive Nix is experimental?08:58:54
@gytis-ivaskevicius:matrix.orgGytis Ivaskeviciusprob due to RFC 9217:06:45
@gytis-ivaskevicius:matrix.orgGytis Ivaskeviciusit implements similar feature but it all is approached differently17:07:04
@tomberek:matrix.orgtomberek(i could be wrong here) I think there is an issue about being in a build, and then starting another one which blocks the original build until the internal one completes.17:07:56
@tomberek:matrix.orgtomberekIf you have overuse of recursive nix, this means you will have long chains of blocked builds. This is compared to the normal situation where a dependency can complete, tear down the sandbox, store results into cache, etc, and then start the next build whenever you want.17:09:29
@tomberek:matrix.orgtomberekMy mental model for this is similar to "tail-recursion optimization". That if you know that the recursion is a tail call, you can do avoid blowing up the stack by turning a function call into a jump. Recursive-nix allows for arbitrary recursion, RFC92 tries to enforce the "tail-call" style.17:11:56
@tomberek:matrix.orgtomberek * My mental model for this is similar to "tail call optimization". That if you know that the recursion is a tail call, you can do avoid blowing up the stack by turning a function call into a jump. Recursive-nix allows for arbitrary recursion, RFC92 tries to enforce the "tail-call" style.17:12:23
@gytis-ivaskevicius:matrix.orgGytis IvaskeviciusNot sure if thats how i'd call it. I'd say its more of a build scheduling17:13:51
@gytis-ivaskevicius:matrix.orgGytis Ivaskeviciusmay need to read it again17:13:57
@sternenseemann:systemli.orgsterni
In reply to @gytis-ivaskevicius:matrix.org
prob due to RFC 92
RFC 92 depends on recursive nix (albeit a more limited variant of it)
17:48:20
@gytis-ivaskevicius:matrix.orgGytis Ivaskevicius
In reply to @sternenseemann:systemli.org
RFC 92 depends on recursive nix (albeit a more limited variant of it)
yeah, John Ericson idea is "STOP WRITING SHIT CODE!!!!" :D
17:51:32
@gytis-ivaskevicius:matrix.orgGytis Ivaskevicius(sorry for the ping)17:51:42
@gytis-ivaskevicius:matrix.orgGytis Ivaskevicius(Actually I'd love to hear some update on all that 😉 )17:52:32
@Ericson2314:matrix.orgJohn Ericson:)17:52:47
@Ericson2314:matrix.orgJohn EricsonOnly news is we need more people to sign up to be shepherds17:54:22
@gytis-ivaskevicius:matrix.orgGytis IvaskeviciusI'm afraid its too big brain for me17:55:10
@tomberek:matrix.orgtomberekbtw: i took a brief stab at updating the eelco commit into the new approach of DerivationTypes, but i started wading in areas of code I think John would be 10x faster than me on.17:55:25

There are no newer messages yet.


Back to Room ListRoom Version: 6