| 24 Jan 2025 |
 Leonardo Santiago | even when droping the whole EvalState it still doesn't free the memory | 18:12:14 |
| Leonardo Santiago | the specific program I'm using is akin to
nix = EvalState()
# 8MB ~ ish footprint
attribute_set = nix.eval_file(...)
# 8MB
value = attribute_set.get("key") # wrapper around `get_attr_byname`
# 500 MB
drop(value)
# 500 MB
drop(attribute_set)
# 500 MB
drop(nix)
# 500 MB
| 18:15:30 |
| Leonardo Santiago | even if I'm not freeing memory when dropping individual values (I can see some issues of why it might not decrease), I don't understand why dropping the actual EvalState doesn't work | 18:18:50 |
| Leonardo Santiago | and yes, I'm using eval_state_free | 18:18:56 |
| Leonardo Santiago | albeit, this is a little bit clunky to assert as I'm testing this through the pyo3 library created from the rust library | 18:29:05 |
| Leonardo Santiago | * albeit, this is a little bit clunky to assert as I'm testing this through the pyo3 library created from the rust library, through the python repl | 18:29:31 |
| Leonardo Santiago | but I'm certain that the drops are executing correctly by virtue of println!("dropping val") inside the impl Drop statement | 18:30:07 |
| Leonardo Santiago | * but I'm certain that the drops are executing correctly (ie. after del nix_state by virtue of println!("dropping val") inside the impl Drop statement | 18:30:46 |
| Leonardo Santiago | * but I'm certain that the drops are executing correctly (ie. after del nix_state) by virtue of println!("dropping val") inside the impl Drop statement | 18:30:49 |
| Leonardo Santiago | and even after running eval_state_free the memory footprint does not diminish even in the slighest. | 18:31:36 |
| Leonardo Santiago | and yes, I'm calling del on every intermediate object along the way | 18:32:14 |
| Leonardo Santiago | * but I'm certain that the drops are executing correctly (ie. after del nix_state) by virtue of println!("dropping eval_state") inside the impl Drop statement | 18:33:18 |
| Leonardo Santiago | even if I add a gc_now() after state_free it does not make a dent in the memory footprint, not sure what could be happening | 18:33:51 |
| 27 Jan 2025 |
| Leonardo Santiago | just so I can understand, am I on a dead end here? shouldn' I expect that at least some memory is freed during the runtime? it seems to me that the memory isn't being free at all, not even when the whole state/store is freed. | 13:38:54 |
| Leonardo Santiago | also, some misunderstandings of mine about the state of gc: there are 2 possible implementations of gc? if so, is the bdwgc included by default? do I need to include any dependencies and enable flags? | 13:43:57 |
 Benedikt | In reply to @o-santi:matrix.org
just so I can understand, am I on a dead end here? shouldn' I expect that at least some memory is freed during the runtime? it seems to me that the memory isn't being free at all, not even when the whole state/store is freed. Are you expecting that the memory is returned to the system? To my knowledge free simply tells the allocator, that it can reuse this memory | 13:44:31 |
| Benedikt | So allocating a new eval stat should not increase your total memory | 13:44:56 |
| Benedikt | * | 13:45:53 |
| Leonardo Santiago | tbh not 100% sure what I'm expecting, just trying to clear my confusion. though, it is of my expectation that after clearing all nix related objects from my program, the resident memory winds down to the state it was before, even if during evaluation it's not that | 13:46:14 |
| Leonardo Santiago | I'm mostly envisioning a use case where I'd like to evaluate a quite complex expression that takes multiple GB, that builds a package that I'd like to use, and those GB are forever entrapped as "nix evaluation memory" even though I'm not going to be evaluating anything else. | 13:47:21 |
| Leonardo Santiago | also, this is not the case, if I do the whole evaluate file -> evaluate attribute of attribute set -> free all of them twice the memory roughly doubles | 13:49:38 |
| Leonardo Santiago | goes from 470MB in the first cycle to 700MB in the next cycle, even if I free the evaluation state | 13:50:08 |
| Leonardo Santiago | and if I do it a third time, it jumps to 1,0GB | 13:51:41 |
| Leonardo Santiago | * tbh not 100% sure what I'm expecting, just trying to clear my confusion. though, it is of my expectation that after clearing all nix related objects from my program, the resident memory winds down to the state it was before, even if during evaluation it's not exactly perfect | 13:52:00 |
| Leonardo Santiago | I don't know how to interpret this other than nix gc isn't actually gc'ing anything:
- maybe because of something very dumb I'm doing
- maybe because of an implementation problem
- maybe I did not compile with the bdwgc (as I'm not doing anything explicit, and don't know if that's necessary)
| 13:54:01 |
| Leonardo Santiago | * I don't know how to interpret this other than nix gc isn't actually gc'ing anything:
- maybe because of something very dumb I'm doing
- maybe because of an implementation issue in nix itself
- maybe I did not compile with the bdwgc (as I'm not doing anything explicit, and don't know if that's necessary)
| 13:54:22 |
| Leonardo Santiago | * also, this is not the case, if I do the whole evaluate file -> evaluate attribute of attribute set -> free all of them twice the memory roughly doubles | 13:55:14 |
| Leonardo Santiago | maybe this is related to the GC_register_my_thread stuff, but I don't quite understand how that works | 14:07:21 |
| Leonardo Santiago | my only reference is $nixops4/rust/nix-expr/src/eval_state.rs stuff, which I will try to copy and see if it changes anything | 14:07:49 |
| Leonardo Santiago | * maybe this is related to the GC_register_my_thread stuff (that I'm doing 0 of), but I don't quite understand how that works | 14:16:35 |
