Cue suffers from the same slowness problems as the nixos module system afaik

On the performance matter, it's not really related to modules per se but in general I personally have high hopes in the incremental evaluation approach. It's less of an issue that your evaluation from scratch is slow, under the hypothesis that each new generation is just a small diff from the previous one, and that your interpreter can reuse most of the previous results incrementally. Lazy evaluation turns out to play quite well with incremental evaluation (basically, you can more or less swap the evaluator with a one performing various caching strategies without changing the semantics of the language). But hard to tell before trying this at scale.

The problem is unscoped merging, aka if you want to figure out whether something exists, you have to potentially search through every module definition

I might be misunderstanding, but I feel like it's more of a NixOS module system design decision (put everything in one big fixpoint soup), rather than something inherent to merging as defined e.g. in CUE ? Couldn't you build a more scoped/hierarchical system based on CUE for example (or even in pure Nix)?

In reply to @yannham:matrix.org
CUE is a configuration language, non Turing-complete, with a very interesting approach (https://cuelang.org/). I would suspect it's not expressive enough for the Nix use-case, but they have a "meet" operator that is close to the fixpoint-merging done by the NixOS module system and Nickel's merging (and, for that matter, Jsonnet's inheritance '+' operator)

To be frank, one of my hidden pleasure was that the module system is turing complete, and that people do not have to learn about functional programming to use it in NixOS.

On that I agree - recursive merging is just equivalent to functions in expressivity, but it's more natural to write configurations this way (also makes things more inspectable and easier to override if done ).

and Cue has a lot of shortcomings when it comes to property checking I believe

Ah right, I believe it's quite hard to encode custom validator beyond the builtin combinators. I think you can do something like write validation functions in Go and then glue things together at the scripting layer but it's suddenly not that simple anymore

yannham: fwiw, dhall has import-boundary caching of the reduced AST

Yeah, it's already something. But by incremental I really mean, in one file, you have a big configuration with recursive fields (might not be actually recursive, just a DAG, but fields that depend on other fields at least). You change enable to true to enable = false or greetings to "Hello, Plutus", and you want to recompute the least amount of stuff. This is on another level.

And it turns out it’s a pretty expensive operation in many practical use-cases, because partially applied functions tend to blow up a lot

From what I recall, I think this is a problem quite specific to dhall: basically their normal form representation which is cached isn't able to represent sharing, so you have to substitute everything and you end up with sometimes exponential blow up.