| 24 Jan 2025 |
 Leonardo Santiago | * after calling nix_alloc_value, if I don't cal nix_value_incref nor nix_gc_incref, which one should I use to decrement the gc (or both?)? and am I expected to run anything else in order to force it to be freed? | 16:47:43 |
| Leonardo Santiago | * after calling nix_alloc_value, if I don't cal nix_value_incref nor nix_gc_incref, which one should I use to decrement the gc? should I use both? and am I expected to run anything else in order to force it to be freed? | 16:48:13 |
| Leonardo Santiago | the documentation in nix_alloc_value says that one should use nix_gc_decref but there's no explanation regarding when (if?) to use nix_value_decref | 16:48:43 |
| Leonardo Santiago | okay, looking at the code it's clear that value_decref is just a gc_decref behind the curtains so it shouldn't make a difference, but the API sure could use some explanation/refinement here | 16:52:16 |
| Leonardo Santiago | one of my concerns is that nix flake check takes way too much memory (32GB~) as we have a ton of hosts to check (200~), and surely that is because it's not freeing the previous hosts while iterating, and so my idea was to try to explicitly evaluate and deallocate each attribute | 17:02:02 |
| Leonardo Santiago | ...but the memory is not being freed? I don't understand what is going on, there should only be one ref as the Value's themselves are not being incref'd, and on rust Drop it's calling decref to hopefully free it once and forall | 17:03:33 |
| Leonardo Santiago | but that doesn't seem to be happening? or at least, the memory footprint isn't getting any smaller | 17:03:49 |
 Robert Hensing (roberth) | nix_alloc_value should be released with nix_value_decref. The main benefit for now is that it has a more specific type, but future evaluators might hypothetically require this | 17:58:13 |
| Robert Hensing (roberth) | as for the garbage collection, neither bdwgc (a conservative gc) nor the interpreter data model are particularly conducive to gc. See https://github.com/NixOS/nix/issues/8621, and for the representation of closures: https://github.com/NixOS/nix/issues/8285 | 18:01:13 |
| Robert Hensing (roberth) | Leonardo Santiago: ^ | 18:01:23 |
| Leonardo Santiago | not sure I 100% understand the issue, I'm not holding a reference to a closure | 18:08:51 |
| Leonardo Santiago | even when droping the whole EvalState it still doesn't free the memory | 18:12:14 |
| Leonardo Santiago | the specific program I'm using is akin to
nix = EvalState()
# 8MB ~ ish footprint
attribute_set = nix.eval_file(...)
# 8MB
value = attribute_set.get("key") # wrapper around `get_attr_byname`
# 500 MB
drop(value)
# 500 MB
drop(attribute_set)
# 500 MB
drop(nix)
# 500 MB
| 18:15:30 |
| Leonardo Santiago | even if I'm not freeing memory when dropping individual values (I can see some issues of why it might not decrease), I don't understand why dropping the actual EvalState doesn't work | 18:18:50 |
| Leonardo Santiago | and yes, I'm using eval_state_free | 18:18:56 |
| Leonardo Santiago | albeit, this is a little bit clunky to assert as I'm testing this through the pyo3 library created from the rust library | 18:29:05 |
| Leonardo Santiago | * albeit, this is a little bit clunky to assert as I'm testing this through the pyo3 library created from the rust library, through the python repl | 18:29:31 |
| Leonardo Santiago | but I'm certain that the drops are executing correctly by virtue of println!("dropping val") inside the impl Drop statement | 18:30:07 |
| Leonardo Santiago | * but I'm certain that the drops are executing correctly (ie. after del nix_state by virtue of println!("dropping val") inside the impl Drop statement | 18:30:46 |
| Leonardo Santiago | * but I'm certain that the drops are executing correctly (ie. after del nix_state) by virtue of println!("dropping val") inside the impl Drop statement | 18:30:49 |
| Leonardo Santiago | and even after running eval_state_free the memory footprint does not diminish even in the slighest. | 18:31:36 |
| Leonardo Santiago | and yes, I'm calling del on every intermediate object along the way | 18:32:14 |
| Leonardo Santiago | * but I'm certain that the drops are executing correctly (ie. after del nix_state) by virtue of println!("dropping eval_state") inside the impl Drop statement | 18:33:18 |
| Leonardo Santiago | even if I add a gc_now() after state_free it does not make a dent in the memory footprint, not sure what could be happening | 18:33:51 |
| 27 Jan 2025 |
| Leonardo Santiago | just so I can understand, am I on a dead end here? shouldn' I expect that at least some memory is freed during the runtime? it seems to me that the memory isn't being free at all, not even when the whole state/store is freed. | 13:38:54 |
| Leonardo Santiago | also, some misunderstandings of mine about the state of gc: there are 2 possible implementations of gc? if so, is the bdwgc included by default? do I need to include any dependencies and enable flags? | 13:43:57 |
 Benedikt | In reply to @o-santi:matrix.org
just so I can understand, am I on a dead end here? shouldn' I expect that at least some memory is freed during the runtime? it seems to me that the memory isn't being free at all, not even when the whole state/store is freed. Are you expecting that the memory is returned to the system? To my knowledge free simply tells the allocator, that it can reuse this memory | 13:44:31 |
| Benedikt | So allocating a new eval stat should not increase your total memory | 13:44:56 |
| Benedikt | * | 13:45:53 |
| Leonardo Santiago | tbh not 100% sure what I'm expecting, just trying to clear my confusion. though, it is of my expectation that after clearing all nix related objects from my program, the resident memory winds down to the state it was before, even if during evaluation it's not that | 13:46:14 |
