Question:
In a submodule with a freeformType, how can one access the freeform definition (and option metadata) separately from any sub-options?

I.e. get definitions, metadata and final merged value for the freeform definitions separately from option definitions.

Alternative:
Is it possible to configure a submodule evaluation to only merge "defined" options into the final config value?

Motivation:
In nixvim, we have several freeform "settings" submodules, and we directly pass the resulting config to a toLua generator.

Currently any sub-options defined will always show up in the resulting config, and we work-around this by having them default to null.

This has two main downsides:

1. When filtering out nulls, we also filter out any explicitly defined nulls.
2. When we have options at nested attr paths, we have to filter out the empty attr after we filter the null

Instead, I would like to have sub-options without a default and have them not show up in the final config if they are not defined.

I don't mind doing this manually, by merging the freeform value with any options.*.value where options.*.isDefined, but idk how to access the freeform value separately.

Follow up:
If I have to do this within the submodule, idk the best way to access the result from outside the submodule? I could declare an internal option definedConfig, but I'm not sure if I could avoid infinite recursion when merging the other option values into it?

Apologies for the long-form question!

Maybe the correct implementation is to define any "optional" options outside the freeform submodule, and then copy them in if defined?

# Set settings.foo to foo, if defined
optionalAttrs options.something.foo.isDefined {
  # Use mkDerivedConfig to also copy definition priority
  something.settings.foo = lib.mkDerivedConfig options.something.foo (v: v);
}

Perhaps submodule sub-options are only intended to be used when the value should always be in the generated value?

Maybe the correct implementation is to define any "optional" options outside the freeform submodule, and then copy them in if defined?

# Set settings.foo to foo, if defined
lib.optionalAttrs options.something.foo.isDefined {
  # Use mkDerivedConfig to also copy definition priority
  something.settings.foo = lib.mkDerivedConfig options.something.foo (v: v);
}

Perhaps submodule sub-options are only intended to be used when the value should always be in the generated value?

Thanks, those links are insightful!

I think 158598 is also related, but 158594 sounds like exactly what I want:

We could add a mkOption argument that has the effect of producing no default value and no config attribute when no definitions are present for the option.

But it looks like the latest comment is backtracking or downscaling that idea...

Playing around in the repl, I can get _module.freeformType and I can get any _freeformOptions for freeform submodules (freeformType = attrsOf submodule), but I can't seem to find the freeform value or definitions anywhere separate from the fully merged option value/defs. Can you confirm I'm not missing anything here?

I guess to do this I would have to write my own types.submodule equivalent (e.g. submoduleWithoutUndefs)? With a custom type I should be able to merge using evalModules but filter out any undefined options 🤔

Seems similar in principle to the types.record proposal in #257511, although I'd need optional and nested fields which I think is out of scope for that one.

Thanks, those links are insightful!

158594 sounds like exactly what I want:

We could add a mkOption argument that has the effect of producing no default value and no config attribute when no definitions are present for the option.

But it looks like the latest comment is backtracking or downscaling that idea...

Playing around in the repl, I can get _module.freeformType and I can get any _freeformOptions for freeform submodules (freeformType = attrsOf submodule), but I can't seem to find the freeform value or definitions anywhere separate from the fully merged option value/defs. Can you confirm I'm not missing anything here?

I guess to do this I would have to write my own types.submodule equivalent (e.g. submoduleWithoutUndefs)? With a custom type I should be able to merge using evalModules but filter out any undefined options 🤔

Seems similar in principle to the types.record proposal in #257511, although I'd need optional and nested fields which I think is out of scope for that one.

I guess to do this I would have to write my own types.submodule equivalent (e.g. submoduleWithoutUndefs)? With a custom type I should be able to merge using evalModules but filter out any undefined options 🤔

Or maybe this wouldn't work, if evalModules itself is what merges the option+freeform defs together

Seems it is: freeformConfig is evaluated here, but isn't exposed separately from config.

Would a PR be welcome to have evalModules include declaredConfig, freeformConfig and config separately in the evaluated result?

@mattsturg:matrix.org Immediate thoughts are that it might be a bit of a leaky abstraction, but maybe also not. Feel free to make a PR, but I can't promise to get to review it myself soon, but perhaps @roberthensing:matrix.org can :)

I've been playing around with this in this branch (current commit), but what I have now is infinitely recursive, because configs shape is changing based on options._module.definedOptionsOnly, but options itself depends on config.

Is this approach fundamentally flawed? Do you have any suggestions for working around the inf-recursion?

The immediate alternative that springs to mind is to pass both a config and a definedConfig to the modules and include definedConfig in the result, alongside config. This is more inline with my original proposal, but feels like a worse abstraction than having a _module.definedOptionsOnly option.

I definedOptionsOnly could also be an evalModules argument instead of an option, but that isn't really ideal either.

I've been playing around with this in this branch (current commit), but what I have now is infinitely recursive, because configs shape is changing based on options._module.definedOptionsOnly, but options itself recursively depends on config.

Is this approach fundamentally flawed? Do you have any suggestions for working around the inf-recursion?

The immediate alternative that springs to mind is to pass both a config and a definedConfig to the modules and include definedConfig in the result, alongside config. This is more inline with my original proposal, but feels like a worse abstraction than having a _module.definedOptionsOnly option.

I definedOptionsOnly could also be an evalModules argument instead of an option, but that isn't really ideal either.

I've been playing around with this in this branch (current commit), but what I have now is infinitely recursive, because configs shape is changing based on options._module.definedOptionsOnly, but options itself recursively depends on config.

Is this approach fundamentally flawed? Do you have any suggestions for working around the inf-recursion?

The immediate alternative that springs to mind is to pass both a config and a definedConfig to the modules and include definedConfig in the result, alongside config. This is more inline with my original proposal, but feels like a worse abstraction than having a _module.definedOptionsOnly option.

definedOptionsOnly could also be an evalModules argument instead of an option, but that isn't really ideal either.

infinisil Robert Hensing (roberth) Thanks for your support so far!

I've been playing around with the concept in this branch (current commit), but my current implementation is infinitely recursive, because configs shape is changing based on options._module.definedOptionsOnly, but options itself recursively depends on config.

Is this approach fundamentally flawed? Do you have any suggestions for working around the inf-recursion?

The immediate alternative that springs to mind is to pass both a config and a definedConfig to the modules and include definedConfig in the result, alongside config. This is more inline with my original proposal, but feels like a worse abstraction than having a _module.definedOptionsOnly option.

definedOptionsOnly could also be an evalModules argument instead of an option, but that isn't really ideal either.

I've opened a draft PR that does it as an evalModules arg, so it actually works without inf-recursion. I detailed this limitation in the PR description. Any feedback would be greatly appreciated!

https://github.com/NixOS/nixpkgs/pull/333799