Are there any good writeups on using lib.types?

I've gone through the manual, the nix.dev deep dive, and a bunch of other Nix module related posts, but nothing seems to cover the more advanced use cases of lib.types and submodules.

I have an option whose default depends on a value defined in the config.

options.shb.restic.databases = mkOption {
  type = attrsOf (submodule ({ name, config, ... }: {
    result = mkOption { type = submodule { options = {
      default = {
        restoreScript = config.settings.repository;
      };
    };};};
  };
};

(I'm skipping a few details in the type here, not sure what is important or not).
Everything works well AFAICT (at least the NixOS tests work fine) until I try to build the documentation of my project. And then I get:

The option `shb.restic.databases.<name>.settings' was accessed but has no value defined. Try setting the option.

Here is where the documentation code is evaluating the modules.

So I tried changing the default to use options instead of config like so: restoreScript = options.settings.value.repository;. The NixOS tests still work fine but I get the same error as above.

I also tried adding a dummy module inside the individualModuleOptionsDocs call that does something like so:

{
  config = {
    shb.restic.databases."<name>".settings = {
      repository = "...";
    };
  };
}

But that's not working either 😅 What's working is if I use a hardcoded string for the default value of the options.

Here is the PR introducing the changes leading to that error. This Github action shows the error. More specifically, here's the type definition and default setting that causes an issue.
Any idea what I should do to keep this computed default value while being able to generate the documentation?

On a totally different topic, this PR introduces 2 contracts in the form of structural typing for backing up files and backing up databases. They are both implemented by Restic. The correct implementation of both contracts is enforced by 2 generic NixOS tests (here and here) and then the Restic implementation is verified here and here.

I have an option whose default depends on a value defined in the config.

options.shb.restic.databases = mkOption {
  type = attrsOf (submodule ({ name, config, ... }: {
    result = mkOption { type = submodule { options = {
      default = {
        restoreScript = config.settings.repository;
      };
    };};};
  };
};

(I'm skipping a few details in the type here, not sure what is important or not. There's a link to the PR with the full code further down).
Everything works well AFAICT (at least the NixOS tests work fine) until I try to build the documentation of my project. And then I get:

The option `shb.restic.databases.<name>.settings' was accessed but has no value defined. Try setting the option.

Here is where the documentation code is evaluating the modules.

So I tried changing the default to use options instead of config like so: restoreScript = options.settings.value.repository;. The NixOS tests still work fine but I get the same error as above.

I also tried adding a dummy module inside the individualModuleOptionsDocs call that does something like so:

{
  config = {
    shb.restic.databases."<name>".settings = {
      repository = "...";
    };
  };
}

But that's not working either 😅 What's working is if I use a hardcoded string for the default value of the options.

Here is the PR introducing the changes leading to that error. This Github action shows the error. More specifically, here's the type definition and default setting that causes an issue.
Any idea what I should do to keep this computed default value while being able to generate the documentation?

On a totally different topic, this PR introduces 2 contracts in the form of structural typing for backing up files and backing up databases. They are both implemented by Restic. The correct implementation of both contracts is enforced by 2 generic NixOS tests (here and here) and then the Restic implementation is verified here and here.

I have an option whose default depends on a value defined in the config.

options.shb.restic.databases = mkOption {
  type = attrsOf (submodule ({ name, config, ... }: {
    result = mkOption { type = submodule { options = {
      default = {
        restoreScript = config.settings.repository;
      };
    };};};
  };
};

(I'm skipping a few details in the type here, not sure what is important or not. There's a link to the PR with the full code further down).
Everything works well AFAICT (at least the NixOS tests work fine) until I try to build the documentation of my project. And then I get:

The option `shb.restic.databases.<name>.settings' was accessed but has no value defined. Try setting the option.

Here is where the documentation code is evaluating the modules.

So I tried changing the default to use options instead of config like so: restoreScript = options.settings.value.repository;. The NixOS tests still work fine but I get the same error as above.

I also tried adding a dummy module inside the individualModuleOptionsDocs call that does something like so:

{
  config = {
    shb.restic.databases."<name>".settings = {
      repository = "...";
    };
  };
}

But that's not working either 😅 What's working is if I use a hardcoded string for the default value of the options.

Here is the PR introducing the changes leading to that error. This Github action shows the error. More specifically, here's the type definition and default setting that causes an issue.
Any idea what I should do to keep this computed default value while being able to generate the documentation?

On a totally different topic, this PR introduces 2 contracts in the form of structural typing for backing up files and backing up databases. They are both implemented by Restic. The correct implementation of both contracts is enforced by 2 generic NixOS tests (here and here) and then the Restic implementation is verified here and here.

ibizaman: I definitely agree on the fact that we need contracts, at the very least for cases where we have multiple implementations. However, I do not know what you attempted to do with mkOption and I want to suggest a very simple idea instead: “options are already one form of contracts”, there is no need to add an additional layer on top of mkOption, and most people should never look into mkOption.

On the other hand imagine the following:

config.contracts.reverse_proxy.<name> = { … };

This could be what is targetted by requesters which are looking for having a reverse proxy.

But how does these contract get honored? We could add an enum type, where each implementation provide a new entry in the enumerated type of the contract as a provider. If the provider is selected for the given contract, then we can forward the configuration to the provider option.

nbp: thanks for the feedback! First I want to say I'm convinced these contracts will really help in nixpkgs but I'm still experimenting on the UX side. So don't take my mkOption experience as something set in stone. I made other contracts without those and they work better I find, so I'll probably remove that experiment at some point.

I considered enums at the start, but the problem I have with those is it's not easily extendable by the end user without modifying nixpkgs directly. At least I don't think so?

What I have currently is the following. My "best" and most recent contracts are currently the backup and database backup contracts. The former is for backing up files, the latter for backing up the output of a command (like pg_dump) without going through an intermediate file. (writing this down, I should probably find a better name than database backup)

This is the backup contract and the database backup one I have now. They're both just attrsets with some pre-defined schema: { requestType = submodule { ... }; resultType = submodule { ... }; }.
The backup contract essentially takes in a list of directories (in the request attribute) and produces (in the result attribute) a backup systemd service and a restore script. This is quite generic and describes IMO the essence of backing up something. For the database backup, it takes in a backup command and a restore command and again produces the backup service and the restore script.

A requester of the backup contract, for example Nextcloud which wants to be backed up, would then provide a shb.nextcloud.backup attribute which uses the requestType. The trick here is that the Nextcloud module fills out the default values of the type. You can see it sets the attributes expected by the contract - user, sourceDirectories and excludePatterns - to something that suits Nextcloud.

A provider of the backup contract, for example Restic, would provide an attribute that uses the requestType and the resultType, here the shb.restic.instances attribute. Here again I use the default values trick to let Restic set the correct name for the backup script and restore script in the result attribute. You can also see that Restic added a settings attribute next to the request and result ones from the contract. This is where each provider can add specific options that are essential for the provider (like for Restic, you must configure the repository where to backup) but are not mandated by the contract.
It would be cool to actually have an enforced schema with this request, result and settings trio but I didn't think yet if it's a good idea nor how it would work.

Putting both sides of the contract together is the end user's responsibility. Here, for backing up Nextcloud files with Restic, it looks like so:

shb.restic.instances."nextcloud" = {
  request = config.shb.nextcloud.backup;
  settings.enable = true;
};

Assuming settings.enable defaults to true, you get a one-liner:

shb.restic.instances."nextcloud".request = config.shb.nextcloud.backup;

The beauty here is that if you had a Vaultwarden instance you want to backup, you can just add the following snippet without knowing anything about how to backup Vaultwarden:

shb.restic.instances."vaultwarden".request = config.shb.vaultwarden.backup;

Similarly, if you wanted to also use BorgBackup to backup Nextcloud and Vaultwarden, you can also just copy paste the snippets and just s/restic/borgbackup:

shb.restic.instances."nextcloud".request = config.shb.nextcloud.backup;
shb.restic.instances."vaultwarden".request = config.shb.vaultwarden.backup;
shb.borgbackup.instances."nextcloud".request = config.shb.nextcloud.backup;
shb.borgbackup.instances."vaultwarden".request = config.shb.vaultwarden.backup;

If you're curious, backing up Vaultwarden is not necessarily complicated. But still, hiding it from the end user is nice.

But now comes the even better part IMO. How do you make sure the Restic and BorgBackup are equivalent (from the contract perspective)? With generic tests! Here's the one for backup contract and here's the one for database backup contract. They are generic because they're a function taking in a location to a module so you plug it in a module that provides the contract and the test will make sure it has the same behavior. Of course you actually need to call that function to test something and that is done here and here respectively for the backup and database backup contract.

These generic tests have a lot of value IMO.

So yeah, that's where I'm at with contracts. If you spelunk in the code of the project, you'll see I have a contract for secrets and ssl. They predate this request, result and settings trio so I'll update them at some point.

One last thing, you'll note that the restore script and backup service in the result part of either backup contract is not really used. That is true. I mostly needed this to make the generic test actually work. But they're still nice for documentation if anything. One result that is actually used is for the secrets contract. Everything I explained above stays the same for this contract, if you want to see how the result is used (which for the secrets contract is the location of the secret on the filesystem), you can look at this doc I wrote.

nbp: thanks for the feedback! First I want to say I'm convinced these contracts will really help in nixpkgs but I'm still experimenting on the UX side. So don't take my mkOption experience as something set in stone. I made other contracts without those and they work better I find, so I'll probably remove that experiment at some point.

I considered enums at the start, but the problem I have with those is it's not easily extendable by the end user without modifying nixpkgs directly. At least I don't think so?

What I have currently is the following. My "best" and most recent contracts are currently the backup and database backup contracts. The former is for backing up files, the latter for backing up the output of a command (like pg_dump) without going through an intermediate file. (writing this down, I should probably find a better name than database backup)

This is the backup contract and the database backup one I have now. They're both just attrsets with some pre-defined schema: { request = submodule { ... }; result = submodule { ... }; }. (I use requestType and resultType for the database backup contract, but I like it less).
The backup contract essentially takes in a list of directories (in the request attribute) and produces (in the result attribute) a backup systemd service and a restore script. This is quite generic and describes IMO the essence of backing up something. For the database backup, it takes in a backup command and a restore command and again produces the backup service and the restore script.

A requester of the backup contract, for example Nextcloud which wants to be backed up, would then provide a shb.nextcloud.backup attribute which uses the requestType. The trick here is that the Nextcloud module fills out the default values of the type. You can see it sets the attributes expected by the contract - user, sourceDirectories and excludePatterns - to something that suits Nextcloud.

A provider of the backup contract, for example Restic, would provide an attribute that uses the requestType and the resultType, here the shb.restic.instances attribute. Here again I use the default values trick to let Restic set the correct name for the backup script and restore script in the result attribute. You can also see that Restic added a settings attribute next to the request and result ones from the contract. This is where each provider can add specific options that are essential for the provider (like for Restic, you must configure the repository where to backup) but are not mandated by the contract.
It would be cool to actually have an enforced schema with this request, result and settings trio but I didn't think yet if it's a good idea nor how it would work.

Putting both sides of the contract together is the end user's responsibility. Here, for backing up Nextcloud files with Restic, it looks like so:

shb.restic.instances."nextcloud" = {
  request = config.shb.nextcloud.backup;
  settings.enable = true;
};

Assuming settings.enable defaults to true, you get a one-liner:

shb.restic.instances."nextcloud".request = config.shb.nextcloud.backup;

The beauty here is that if you had a Vaultwarden instance you want to backup, you can just add the following snippet without knowing anything about how to backup Vaultwarden:

shb.restic.instances."vaultwarden".request = config.shb.vaultwarden.backup;

Similarly, if you wanted to also use BorgBackup to backup Nextcloud and Vaultwarden, you can also just copy paste the snippets and just s/restic/borgbackup:

shb.restic.instances."nextcloud".request = config.shb.nextcloud.backup;
shb.restic.instances."vaultwarden".request = config.shb.vaultwarden.backup;
shb.borgbackup.instances."nextcloud".request = config.shb.nextcloud.backup;
shb.borgbackup.instances."vaultwarden".request = config.shb.vaultwarden.backup;

If you're curious, backing up Vaultwarden is not necessarily complicated. But still, hiding it from the end user is nice.

But now comes the even better part IMO. How do you make sure the Restic and BorgBackup are equivalent (from the contract perspective)? With generic tests! Here's the one for backup contract and here's the one for database backup contract. They are generic because they're a function taking in a location to a module so you plug it in a module that provides the contract and the test will make sure it has the same behavior. Of course you actually need to call that function to test something and that is done here and here respectively for the backup and database backup contract.

These generic tests have a lot of value IMO.

So yeah, that's where I'm at with contracts. If you spelunk in the code of the project, you'll see I have a contract for secrets and ssl. They predate this request, result and settings trio so I'll update them at some point.

One last thing, you'll note that the restore script and backup service in the result part of either backup contract is not really used. That is true. I mostly needed this to make the generic test actually work. But they're still nice for documentation if anything. One result that is actually used is for the secrets contract. Everything I explained above stays the same for this contract, if you want to see how the result is used (which for the secrets contract is the location of the secret on the filesystem), you can look at this doc I wrote.

nbp: thanks for the feedback! First I want to say I'm convinced these contracts will really help in nixpkgs but I'm still experimenting on the UX side. So don't take my mkOption experience as something set in stone. I made other contracts without those and they work better I find, so I'll probably remove that experiment at some point.

I considered enums at the start, but the problem I have with those is it's not easily extendable by the end user without modifying nixpkgs directly. At least I don't think so?

What I have currently is the following. My "best" and most recent contracts are currently the backup and database backup contracts. The former is for backing up files, the latter for backing up the output of a command (like pg_dump) without going through an intermediate file. (writing this down, I should probably find a better name than database backup)

This is the backup contract and the database backup one I have now. They're both just attrsets with some pre-defined schema: { request = submodule { ... }; result = submodule { ... }; }. (I use requestType and resultType for the database backup contract, but I like it less).
The backup contract essentially takes in a list of directories (in the request attribute) and produces (in the result attribute) a backup systemd service and a restore script. This is quite generic and describes IMO the essence of backing up something. For the database backup, it takes in a backup command and a restore command and again produces the backup service and the restore script.

A requester of the backup contract, for example Nextcloud which wants to be backed up, would then provide a shb.nextcloud.backup attribute which uses the requestType. The trick here is that the Nextcloud module fills out the default values of the type. You can see it sets the attributes expected by the contract - user, sourceDirectories and excludePatterns - to something that suits Nextcloud.

A provider of the backup contract, for example Restic, would provide an attribute that uses the contracts.backup.request type and the contracts.backup.result type, here the shb.restic.instances attribute. Here again I use the default values trick to let Restic set the correct name for the backup script and restore script in the result attribute. You can also see that Restic added a settings attribute next to the request and result ones from the contract. This is where each provider can add specific options that are essential for the provider (like for Restic, you must configure the repository where to backup) but are not mandated by the contract.
It would be cool to actually have an enforced schema with this request, result and settings trio but I didn't think yet if it's a good idea nor how it would work.

Putting both sides of the contract together is the end user's responsibility. Here, for backing up Nextcloud files with Restic, it looks like so:

shb.restic.instances."nextcloud" = {
  request = config.shb.nextcloud.backup;
  settings.enable = true;
};

Assuming settings.enable defaults to true, you get a one-liner:

shb.restic.instances."nextcloud".request = config.shb.nextcloud.backup;

The beauty here is that if you had a Vaultwarden instance you want to backup, you can just add the following snippet without knowing anything about how to backup Vaultwarden:

shb.restic.instances."vaultwarden".request = config.shb.vaultwarden.backup;

Similarly, if you wanted to also use BorgBackup to backup Nextcloud and Vaultwarden, you can also just copy paste the snippets and just s/restic/borgbackup:

shb.restic.instances."nextcloud".request = config.shb.nextcloud.backup;
shb.restic.instances."vaultwarden".request = config.shb.vaultwarden.backup;
shb.borgbackup.instances."nextcloud".request = config.shb.nextcloud.backup;
shb.borgbackup.instances."vaultwarden".request = config.shb.vaultwarden.backup;

If you're curious, backing up Vaultwarden is not necessarily complicated. But still, hiding it from the end user is nice.

But now comes the even better part IMO. How do you make sure the Restic and BorgBackup are equivalent (from the contract perspective)? With generic tests! Here's the one for backup contract and here's the one for database backup contract. They are generic because they're a function taking in a location to a module so you plug it in a module that provides the contract and the test will make sure it has the same behavior. Of course you actually need to call that function to test something and that is done here and here respectively for the backup and database backup contract.

These generic tests have a lot of value IMO.

So yeah, that's where I'm at with contracts. If you spelunk in the code of the project, you'll see I have a contract for secrets and ssl. They predate this request, result and settings trio so I'll update them at some point.

One last thing, you'll note that the restore script and backup service in the result part of either backup contract is not really used. That is true. I mostly needed this to make the generic test actually work. But they're still nice for documentation if anything. One result that is actually used is for the secrets contract. Everything I explained above stays the same for this contract, if you want to see how the result is used (which for the secrets contract is the location of the secret on the filesystem), you can look at this doc I wrote.