!VRULIdgoKmKPzJZzjj:nixos.org

Nix Hackers

895 Members
For people hacking on the Nix package manager itself187 Servers

Load older messages


SenderMessageTime
4 Nov 2021
@tomberek:matrix.orgtomberekokay... i guess the idea to to update + specialize the install-matrix, maybe a bit of automation to post the results somewhere03:39:57
@abathur:matrix.orgabathur(not saying no; basically, I find it infinitely infuriating that we just collectively hemorrhage good-will with people who run into dumb install/uninstall/reinstall trouble, but I also lost nearly all of my ~hobby programming time from october-february to fixing up the macOS store volume flow and crawling over glass to demonstrate that it worked well enough to get it merged, at the expense of greatly delaying a lot of work on resholve)03:45:30
@abathur:matrix.orgabathurso I'm interested, I just need to defend myself against my own inclination to over-commit in this area, lest I chown the whole installer03:48:36
@baloo_:matrix.orgbaloois there a nix-build --check equivalent with nix2.4?04:59:02
@tomberek:matrix.orgtomberek--rebuild04:59:45
@baloo_:matrix.orgbaloo
nix build --rebuild -f ./foo.nix pkgs.linux.collectd
error: unrecognised flag '--rebuild'
05:00:59
@baloo_:matrix.orgbaloomeh, that's a 2.305:02:39
@baloo_:matrix.orgbaloomy bad05:02:40
@pamplemouss_:matrix.orgpamplemousse
In reply to @fzakaria:matrix.org
Couldn't find the repository they contributed too.
I essentially had things going on my fork of the nix repo
06:12:09
@pamplemouss_:matrix.orgpamplemousseI wouldn't say the project is "over", but I had indeed completed the fellowship, and had a lot going on personally (too much to keep going for now).06:12:54
@pamplemouss_:matrix.orgpamplemousseRight now, I still writing a write-up of what has been done, and hopefully laying some ground for the future of "fuzzing nix".06:13:45
@moritz.hedtke:matrix.orgMoritz Hedtke
In reply to @moritz.hedtke:matrix.org
"SUMMARY: AddressSanitizer: 6032 byte(s) leaked in 141 allocation(s)." - it's going down down down (just using GC for everything though)
Ohh I'm also (just starting) to try to fuzz this. I already saw pamplemousse's discourse threads but I wanted to go a different approach to see whether it works better
11:20:30
@pamplemouss_:matrix.orgpamplemousse I would say that if you want to fuzz the parsing and evaluation logic, you should not use an in-process fuzzer. 12:10:59
@misterio:matrix.org@misterio:matrix.org joined the room.14:18:41
@moritz.hedtke:matrix.orgMoritz Hedtke
In reply to @pamplemouss_:matrix.org
I would say that if you want to fuzz the parsing and evaluation logic, you should not use an in-process fuzzer.

I know that is the easy path but why take the easy path if you can take the hard path?

I assume you say that because it's extremely hard to do this in process because of the current memory management situation?

I think it's quite likely that I will soon go that route it's probably just way slower.

Also LLVM doesn't support out of process fuzzing does it? And the AFL ++ version is old in nixpkgs and a PITA to update

16:04:12
@roberthensing:matrix.orgRobert Hensing (roberth) niksnut: the Nix 2.4 release in Nixpkgs seems to be stuck on a regression https://github.com/NixOS/nix/issues/5495 16:07:56
5 Nov 2021
@fzakaria:matrix.orgfzakariaI was thinking about trying out the different sanitizers on Nix for my compiler project (ASan, TSan etc..) -- and writing a report on the findings. Has this already been done in Nix ?03:12:42
@pamplemouss_:matrix.orgpamplemousse

If by memory management, you mean the GC, then no, it's not because of that:
You can deactivate LeakSanitizer to stop complaining, and also manage the memory somewhat manually with an arena and deactivating the GC.

It's because some functions in nix manipulate a global state that cannot (i.e. that I could not) be reset to its initial state before the next run.
In process fuzzers run the harness in a loop, so the each run will "pollute" the subsequent one by mutating the state.

07:36:01
@pamplemouss_:matrix.orgpamplemousse moritz.hedtke: ^ 07:36:34
@pamplemouss_:matrix.orgpamplemousseAlso, LLVM support in-process fuzzing, and that's what I have been trying to use during the fellowship. https://www.llvm.org/docs/LibFuzzer.html#introduction07:37:18
@pamplemouss_:matrix.orgpamplemousse
In reply to @fzakaria:matrix.org
I was thinking about trying out the different sanitizers on Nix for my compiler project (ASan, TSan etc..) -- and writing a report on the findings.
Has this already been done in Nix ?
Kinda, as part of fuzzing. Also, I think there is an issue on the nix repo mentioning that idea.
07:38:50
@pamplemouss_:matrix.orgpamplemousse
In reply to @pamplemouss_:matrix.org
Also, LLVM support in-process fuzzing, and that's what I have been trying to use during the fellowship.
https://www.llvm.org/docs/LibFuzzer.html#introduction
moritz.hedtke:
with https://github.com/NixOS/nix/pull/4996, you can use nix develop .#.clang11StdenvPackages to get libFuzzer (distributed with LLVM), and all the nix dependencies
07:43:15
@pamplemouss_:matrix.orgpamplemousse
In reply to @pamplemouss_:matrix.org
Kinda, as part of fuzzing. Also, I think there is an issue on the nix repo mentioning that idea.
Mmmh, I can't find the issue... Maybe that was a post on discourse, or a comment somewhere.
07:45:23
@pamplemouss_:matrix.orgpamplemousse fzakaria: Anyway, to my knowledge, it hasn't been done. If you need help to figure out how to integrate that to the build system (using the meson PR could make it much simpler - https://github.com/NixOS/nix/pull/3160), don't hesitate to ask me :) 07:47:37
@moritz.hedtke:matrix.orgMoritz Hedtke
In reply to @pamplemouss_:matrix.org

If by memory management, you mean the GC, then no, it's not because of that:
You can deactivate LeakSanitizer to stop complaining, and also manage the memory somewhat manually with an arena and deactivating the GC.

It's because some functions in nix manipulate a global state that cannot (i.e. that I could not) be reset to its initial state before the next run.
In process fuzzers run the harness in a loop, so the each run will "pollute" the subsequent one by mutating the state.

I meant because some things are allocated by GC, some not and Nix is intentionally leaking data for efficiency. I want to first use the GC for everything also because I want to know whether this reduces memory usage in real use cases (maybe with some more code changes). But if I don't make progress I also thought about using an arena allocator as I read that in your post. That still wouldn't help with resetting globals would it? How did you solve that? On the other hand I think it's feasible to reset them
10:08:48
@moritz.hedtke:matrix.orgMoritz HedtkeYour arena allocator probably just hooked malloc and free and discarded the memory afterwards? Because for the GC that's more complicated and I thought maybe I can use some previous art10:10:13
@moritz.hedtke:matrix.orgMoritz HedtkeOn that note thanks for your help10:12:30
@moritz.hedtke:matrix.orgMoritz Hedtke
In reply to @pamplemouss_:matrix.org
Also, LLVM support in-process fuzzing, and that's what I have been trying to use during the fellowship. https://www.llvm.org/docs/LibFuzzer.html#introduction
Also started with it but quickly realized there is some preparation to do before
10:13:07
@pamplemouss_:matrix.orgpamplemousse
In reply to @moritz.hedtke:matrix.org
I meant because some things are allocated by GC, some not and Nix is intentionally leaking data for efficiency. I want to first use the GC for everything also because I want to know whether this reduces memory usage in real use cases (maybe with some more code changes). But if I don't make progress I also thought about using an arena allocator as I read that in your post. That still wouldn't help with resetting globals would it? How did you solve that? On the other hand I think it's feasible to reset them
I did not manage to solve the problem of resetting the state properly between each input.
That's why I said that fuzzing the parsing / evaluation can't be done with an in-process fuzzer ATM.
10:23:44
@mkg20001:mkg20001.iomkg20001 joined the room.18:08:05

Show newer messages


Back to Room ListRoom Version: 6