[Help] Struggling to implement configuration file options

[gomain]

Toy program:

//cli.js
const optionA = new Option("--a <a>").makeOptionMandatory();
const optionB = new Option("--b <b>").makeOptionMandatory();

const useA = new Command("useA")
  .addOption(optionA)
  .action((opts) => console.log(opts.a));
const useAandB = new Command("useAandB")
  .addOption(optionA)
  .addOption(optionB)
  .action((opts) => console.log(opts.a, opts.b));

program
  .addCommand(useA)
  .addCommand(useAandB)
  .parse();

Now I want to include a config.json file. Say it is

{
  "a": "config-A"
}

Such that when run

$ node cli.js useA
config-A

$ node cli.js useA --a cli-A
cli-A

$node cli.js useAandB
error: required option '--b <b>' not specified

$node cli.js useAandB --a cli-A
error: required option '--b <b>' not specified

$node cli.js useAandB --b cli-B
config-A cli-B

$node cli.js useAandB --a cli-A --b cli-B
cli-A cli-B

Use cases will extend to combinations of options having .default or .env where precedence will be cli > config > env > default.

I have looked into .getOptionValue and friends but struggle to find the right place to use them. Documentations lack practical examples.

Some concerns:

• Options may have .argParser. I wish the json to store values as strings and be parsed as if provided on cli.
• Missing option errors are triggered after all option sources have been consulted.
• Not all options are consumed by every command (as in --b). I want to keep the unknown-option-error when provided at cli but unknown options in config file are ignored.
• Able to have global option --config <path> to pass location of config file.

[shadowspawn]

A couple of historical references.

There was a discussion about improvements to support config file processing in #1584. Several of the people commenting had already written implementations. The outcome was adding . getOptionValueSource() to make it easier to manage precedence.

The use case in this comment is processing a config file: #1713 (comment)

[shadowspawn]

Options may have .argParser. I wish the json to store values as strings and be parsed as if provided on cli.

There are not methods to do this directly. I think the safest route is probably looking up the option and doing the conversion yourself.

You could try emitting an option event (e.g. option:b) with the string value to trigger the built-in option processing, but the processing assumes it is from the CLI so you may run into problems. I have not done this myself from client code.

[shadowspawn]

where precedence will be cli > config > env > default.

A small heads-up. The built-in processing has env > config. However, this will only affect you if the built-in env processing comes after your config processing and you specify the source as config!

[gomain]

I was able to come up with this (in typescript using @commander-js/extra-typings):

//cli.ts
import fs from "fs";
import path from "path";
import { createCommand, createOption } from "@commander-js/extra-typings";

const parser = function (value: string): string {
  return value + "-parsed";
};

const optionA = createOption("--a <a>")
  .makeOptionMandatory()
  .env("A")
  .default("default-A-parsed")
  .argParser(parser);

const optionB = createOption("--b <b>")
  .makeOptionMandatory()
  .env("B")
  .argParser(parser);

const useA = createCommand("useA")
  .addOption(optionA)
  .action((opts) => console.log(opts.a));
const useAandB = createCommand("useAandB")
  .addOption(optionA)
  .addOption(optionB)
  .action((opts) => console.log(opts.a, opts.b));

await createCommand()
  .addOption(createOption("--config <path>"))
  .helpOption(false) // forward --help to inner program
  .allowUnknownOption() // ignore options intended for inner program
  .action(async (opts) => {
    const config = opts.config ? await readConfig(opts.config) : {};
    createCommand()
      .addOption(createOption("--config <path>", "Path to configuration file")) // for help display
      .hook("preSubcommand", (_, subCommand) => {
        for (const [key, value] of Object.entries(config)) {
          subCommand.setOptionValue(key, value);
        }
      })
      .addCommand(useA)
      .addCommand(useAandB)
      .showHelpAfterError()
      .parse();
  })
  .parseAsync();

async function readConfig(relativePath: string): Promise<any> {
  const modulePath = path.resolve(relativePath);
  if (fs.existsSync(modulePath) && fs.statSync(modulePath).isFile()) {
    const module = await import(modulePath);
    return module.default;
  } else {
    throw new Error(`Config file '${relativePath}' does not exist.`);
  }
}

Here I nest two root commands and use the 'preSubcommand' hook on the inner command to .setOptionValue on its sub-commands. The outer command is for reading the `"--config " option.

Some observations:

• This solution was enabled by exploiting a hook. It so happens that the pre-subcommand-hook is given a handle to the sub-command and is run before parsing options of the sub-command. This was discovered by trial and error. Perhaps the full life cycle of executing commands should be documented. Perhaps more lifecycle hooks could be provided, i.e. 'preParse' , 'postParse', 'preParseSubcommand', etc..
• .argParser is only run on option arguments passed by cli or env. Not on pre existing values - from .setOptionValue. The pre-existing value is, however, given to the parser - but only when parsing an argument from cli or env. The .getOptionSource is not accessible in this context. So it is uncertain of the value of this previous value. May be this callback should be given, also, the source of the values given.
• As a consequence, the type of the config value must match the output of the parser. This is already the case with .default. This makes it inconvenient to use JSON config files - having to self parse JSON arguments. As a cli first utility (where argument is always text), it feels consistent that arguments provided in whatever source should also be text and get parsed by the same provided, once, parser.

[shadowspawn]

Perhaps the full life cycle of executing commands should be documented. Perhaps more lifecycle hooks could be provided, i.e. 'preParse' , 'postParse', 'preParseSubcommand', etc..

Yes. A high level reply. I am open to adding more hooks as identify use cases that require them and how they can be identified, such as preSubcommand. I didn't want to scatter hooks through the flow, partly without there being a need, partly because supporting async can be tricky, and partly because the flow is complicated and I haven't worked out what phases could/should be referenced.

[shadowspawn]

As a cli first utility (where argument is always text), it feels consistent that arguments provided in whatever source should also be text and get parsed by the same provided, once, parser.

This is being considered and done for some new code where it makes sense, and for example the recent Option.preset() method takes the text-argument-like-the-cli approach.

I don't think it is feasible to change the old methods, it would change and break too much existing code.

[shadowspawn]

I am interested to see you came up with a double-parse approach. I had wondered about that as a way to extract the config option ahead of the main parsing, but not used for real: #1584 (comment)

[shadowspawn]

This solution was enabled by exploiting a hook.

👍 You had it worked out before I suggested it, good digging!

[shadowspawn]

Thanks for the detailed and thoughtful original question. I'll stop commenting, but ask if I haven't covered anything you haven't already worked out for yourself. 😄

[shadowspawn]

The sample program has a lot in it, with defaults and env and mandatory and parser. Nice!

For interest, I think there is no need to do the double parsing for this code. The config option is available in the hook after the top-level command has parsed the option.

await createCommand()
  .addOption(createOption("--config <path>", "Path to configuration file"))
  .hook("preSubcommand", async (hookedCommand, subCommand) => {
    const configPath = hookedCommand.opts().config;
    const config = configPath ? await readConfig(configPath) : {};
    for (const [key, value] of Object.entries(config)) {
      subCommand.setOptionValue(key, value);
    }
  })
  .addCommand(useA)
  .addCommand(useAandB)
  .showHelpAfterError()
  .parseAsync();

[shadowspawn]

Opened a PR with some life cycle and hooks detail: #1860

[shadowspawn]

Lots of good comments in this issue. The action item was to add some documentation on the parsing life cycle and hooks as a step forward.

Shipped that in 10.0.1

Thanks @gomain