One way you could work around this is to add an internal final option within the submodule and then map to it in the apply function:

foo = mkOption {
  type = types.submodule (
    { config, options, ... }:
    {
      freeformType = types.attrsOf types.str;
      options.marker_a = mkOption {
        type = types.str;
      };
      options.marker_b = mkOption {
        type = types.str;
      };
      options.__result = mkOption {
        type = config._module.freeformType;
        internal = true;
      };

      config.__result =
        let
          cfg = builtins.removeAttrs config [
            "_module"
            "marker_a"
            "marker_b"
            "__result"
          ];
        in
        if options.marker_a.isDefined then
          cfg // { inherit (config) marker_a; } # TODO
        else if options.marker_b.isDefined then
          cfg // { inherit (config) marker_b; } # TODO
        else
          cfg # TODO: (empty? no marker?)
          ;
    }
  );

  apply = value: value.__result;
}

trace: { below = "_"; }                                                                                                                                                                                              
trace: { marker_b = "bar"; }                                                                                                                                                                                         
trace: { above = "_"; }                                                                                                                                                                                              
trace: { below = "_"; }                                                                                                                                                                                              
{ foo = { above = "_"; below = "_"; marker_b = "bar"; }; } 

For modules:

    { foo = { above = "_"; }; }
    {
      foo.marker_b = "bar";
    }
    { foo = { below = "_"; }; }

trace: { below = "_"; }                                                                                                                                                                                              
trace: { marker_b = "bar"; }                                                                                                                                                                                         
trace: { above = "_"; }                                                                                                                                                                                              
trace: { below = "_"; }                                                                                                                                                                                              
{ foo = { above = "_"; below = "_"; marker_b = "bar"; }; } 

For modules:

    { foo = { above = "_"; }; }
    {
      foo.marker_b = "bar";
    }
    { foo = { below = "_"; }; }

This won't support foo = { config,...}: { marker_b = "bar"; }.

Type checking is done before merging, but after resolving mkIf (etc) wrappers.

Type checking is used to determine which type to use for merging.

(
  { options, ... }:
  {
    freeformType = attrsOf (
      if options.marker_a.isDefined && options.marker_b.isDefined then
        throw "both `${options.marker_a}' and `${options.marker_b}' are defined"
      else if options.marker_a.isDefined then
        a
      else if options.marker_b.isDefined then
        b
      else
        throw "neither `${options.marker_a}' or `${options.marker_b}' are defined"
    );
  }
)

I suspect this works and then gets rejected by some kind of a final type check: error: The option `marker_a' was accessed but has no value defined. Try setting the option

(my attempt: https://gist.github.com/SomeoneSerge/29a9e6e10cdcd3596f865c41c7042610)

That's not a type check, that's an "empty value" stub. When an option has no definitions, its value is set to a stub that throws that error.

You're probably reading the value somewhere, e.g. using config.marker_a instead of options.marker_a, or not using removeAttrs when creating __result?

On line 45 you're defining apply which is meaningless in a module, it's a feature of options.

If you're evaluating these directly with lib.evalModules instead of using a submodule option, you can do the apply manually:

(lib.evalModules { /* ... */ }).config.__result

On line 45 you're defining apply which is meaningless in a module, it's a feature of options.

Yeah I was wondering if I misunderstood the "freeformType is an option"