| 21 Jan 2025 |
 Leonardo Santiago | ...all of them? As I've stated before, I'm trying to reproduce the nix develop behavior utilizing a different caching mechanism for evaluation | 03:00:54 |
| Leonardo Santiago | the problem is that I realised that source $stdenv/setup during shell entering is very slow and nix develop can only be so fast because it also caches that behind a derivation | 03:01:35 |
| Leonardo Santiago | I'd like to do what it does (cache source $stdenv/setup by making it an output of a derivation) but in order to do it I'd need to fiddle with the actual derivation | 03:02:19 |
| Leonardo Santiago | specifically, I'd like to be able to write it back to the store, so that it can be built, but if I just do it the dumbest way possible (copy the parsed env vars and pass it to derivation primop) then inputSrc and inputDrvs get empty | 03:04:49 |
| Leonardo Santiago | I know that it's not strictly necessary to get it working in this specific case, but it's expected from the library I'm writing that parse_drv(parse_drv(path).serialize()) == parse_drv(path) | 03:11:58 |
| Leonardo Santiago | * specifically, I'd like to be able to write it back to the store, so that it can be built, but if I just do it the dumbest way possible (copy the parsed env vars and pass it to derivation primop) then inputSrc and inputDrvs are generated empty | 12:33:37 |
| 24 Jan 2025 |
|  @idioticself55:matrix.org joined the room. | 14:19:47 |
| Leonardo Santiago | about memory management | 16:45:11 |
| Leonardo Santiago | let's say I have a Value, how am I supposed to free it? there's nix_gc_decref and nix_value_decref | 16:45:50 |
| Leonardo Santiago | * let's say I have a Value, how am I supposed to free it? there's nix_gc_decref and nix_value_decref and there's not a clear explanation on which should be used when. | 16:46:30 |
| Leonardo Santiago | after calling nix_alloc_value, if I don't cal nix_value_incref nor nix_gc_incref, which one should I use to decrement the gc? and am I expected to run anything else in order to force it to be freed? | 16:47:33 |
| Leonardo Santiago | * after calling nix_alloc_value, if I don't cal nix_value_incref nor nix_gc_incref, which one should I use to decrement the gc (or both?)? and am I expected to run anything else in order to force it to be freed? | 16:47:43 |
| Leonardo Santiago | * after calling nix_alloc_value, if I don't cal nix_value_incref nor nix_gc_incref, which one should I use to decrement the gc? should I use both? and am I expected to run anything else in order to force it to be freed? | 16:48:13 |
| Leonardo Santiago | the documentation in nix_alloc_value says that one should use nix_gc_decref but there's no explanation regarding when (if?) to use nix_value_decref | 16:48:43 |
| Leonardo Santiago | okay, looking at the code it's clear that value_decref is just a gc_decref behind the curtains so it shouldn't make a difference, but the API sure could use some explanation/refinement here | 16:52:16 |
| Leonardo Santiago | one of my concerns is that nix flake check takes way too much memory (32GB~) as we have a ton of hosts to check (200~), and surely that is because it's not freeing the previous hosts while iterating, and so my idea was to try to explicitly evaluate and deallocate each attribute | 17:02:02 |
| Leonardo Santiago | ...but the memory is not being freed? I don't understand what is going on, there should only be one ref as the Value's themselves are not being incref'd, and on rust Drop it's calling decref to hopefully free it once and forall | 17:03:33 |
| Leonardo Santiago | but that doesn't seem to be happening? or at least, the memory footprint isn't getting any smaller | 17:03:49 |
 Robert Hensing (roberth) | nix_alloc_value should be released with nix_value_decref. The main benefit for now is that it has a more specific type, but future evaluators might hypothetically require this | 17:58:13 |
| Robert Hensing (roberth) | as for the garbage collection, neither bdwgc (a conservative gc) nor the interpreter data model are particularly conducive to gc. See https://github.com/NixOS/nix/issues/8621, and for the representation of closures: https://github.com/NixOS/nix/issues/8285 | 18:01:13 |
| Robert Hensing (roberth) | Leonardo Santiago: ^ | 18:01:23 |
| Leonardo Santiago | not sure I 100% understand the issue, I'm not holding a reference to a closure | 18:08:51 |
| Leonardo Santiago | even when droping the whole EvalState it still doesn't free the memory | 18:12:14 |
| Leonardo Santiago | the specific program I'm using is akin to
nix = EvalState()
# 8MB ~ ish footprint
attribute_set = nix.eval_file(...)
# 8MB
value = attribute_set.get("key") # wrapper around `get_attr_byname`
# 500 MB
drop(value)
# 500 MB
drop(attribute_set)
# 500 MB
drop(nix)
# 500 MB
| 18:15:30 |
| Leonardo Santiago | even if I'm not freeing memory when dropping individual values (I can see some issues of why it might not decrease), I don't understand why dropping the actual EvalState doesn't work | 18:18:50 |
| Leonardo Santiago | and yes, I'm using eval_state_free | 18:18:56 |
| Leonardo Santiago | albeit, this is a little bit clunky to assert as I'm testing this through the pyo3 library created from the rust library | 18:29:05 |
| Leonardo Santiago | * albeit, this is a little bit clunky to assert as I'm testing this through the pyo3 library created from the rust library, through the python repl | 18:29:31 |
| Leonardo Santiago | but I'm certain that the drops are executing correctly by virtue of println!("dropping val") inside the impl Drop statement | 18:30:07 |
| Leonardo Santiago | * but I'm certain that the drops are executing correctly (ie. after del nix_state by virtue of println!("dropping val") inside the impl Drop statement | 18:30:46 |
