Upgrade clap from 3.1 to 4.0

[kalil-pelissier]

Upgrade clap version from 3.1 to 4.0

Changelog

Sourced from clap changelog.

[4.0.0] - 2022-09-28

Highlights

Arg::num_args(range)

Clap has had several ways for controlling how many values will be captured without always being clear on how they interacted, including

• Arg::multiple_values(true)
• Arg::number_of_values(4)
• Arg::min_values(2)
• Arg::max_values(20)
• Arg::takes_value(true)

These have now all been collapsed into Arg::num_args which accepts both
single values and ranges of values. num_args controls how many raw arguments
on the command line will be captured as values per occurrence and independent
of value delimiters.

See Issue 2688 for more background.

Polishing Help

Clap strives to give a polished CLI experience out of the box with little
ceremony. With some feedback that has accumulated over time, we took this
release as an opportunity to re-evaluate our --help output to make sure it is
meeting that goal.

In doing this evaluation, we wanted to keep in mind:

• Whether other CLIs had ideas that make sense to apply
• Providing an experience that fits within the rest of applications and works across all shells

Before:

git
A fictional versioning CLI

USAGE:
    git <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    add      adds things
    clone    Clones repos
    help     Print this message or the help of the given subcommand(s)
    push     pushes things
    stash

After:

A fictional versioning CLI

Usage: git <COMMAND>

Commands:
  clone  Clones repos
  push   pushes things
  add    adds things
  stash
  help   Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help information

• name/version header was removed because we couldn't justify the space it occupied when
  • Usage already includes the name
  • --version is available for showing the same thing (if the program has a version set)
• Usage was dropped to one line to save space
• Focus is put on the subcommands
• Headings are now Title case
• The more general term "command" is used rather than being explicit about being "subcommands"
• The output is more dense with the expectation that it won't affect legibility but will allow more content
• We've moved to a more neutral palette for highlighting elements (not highlighted above)

In talking to users, we found some that liked clap's man-like experience.
When deviating from this, we are making the assumption that those are more
power users and that the majority of users wouldn't look as favorably on being
consistent with man.

See Issue 4132 for more background.

More Dynamicism

Clap's API has focused on &str for performance but this can make
dealing with owned data difficult, like #[arg(default_value_t)] generating a
String from the default value.

Additionally, to avoid ArgMatches from borrowing (and for some features we
decided to forgo), clap took the &str argument IDs and hashed them. This
prevented us from providing a usable API for iterating over existing arguments.

Now clap has switched to a string newtype that gives us the flexibility to
decide whether to use &'static str, Cow<'static, str> for fast dynamic behavior, or
Box<str> for dynamic behavior with small binary size.

As an extension of that work, you can now call ArgMatches::ids to iterate
over the arguments and groups that were found when parsing. The newtype Id
was used to prevent some classes of bugs and to make it easier to understand
when opaque Ids are used vs user-visible strings.

Clearing Out Deprecations

Instead of doing all development on clap 4.0.0, we implemented a lot of new features during clap 3's development, deprecating the old API while introducing the new API, including:

• Replacing the implicit behavior for args when parsing them with ArgAction
• Replacing various one-off forms of value validation with the ValueParser API
  • Allowing derives to automatically do the right thing for PathBuf (allowing invalid UTF-8)
• Replacing AppSettings and ArgSettings enums with getters/setters
• Clarifying terms and making them more consistent

Migrating

Steps:

0. Upgrade to v3 if you haven't already
1. Add CLI tests (including example below), -h and --help output at a minimum (recommendation: trycmd for snapshot testing)
2. If using Builder API: Explicitly set the arg.action(ArgAction::...) on each argument (StoreValue for options and IncOccurrences for flags)
3. Run cargo check --features clap/deprecated and resolve all deprecation warnings
4. Upgrade to v4
5. Update feature flags

• If default-features = false, run cargo add clap -F help,usage,error-context
• Run cargo add clap -F wrap_help unless you want to hard code line wraps

6. Resolve compiler errors
7. Resolve behavior changes (see "subtle changes" under BREAKING CHANGES)
8. At your leisure: resolve new deprecation notices

Example test (derive):

#[derive(clap::Parser)]
struct Cli {
    ...
}

#[test]
fn verify_cli() {
    use clap::CommandFactory;
    Cli::command().debug_assert()
}

Example test (builder):

fn cli() -> clap::Command {
    ...
}

#[test]
fn verify_cli() {
    cli().debug_assert();
}

Note: the idiomatic / recommended way of specifying different types of args in the Builder API has changed:

Before

.arg(Arg::new("flag").long("flag"))  # --flag
.arg(Arg::new("option").long("option").takes_value(true))  # --option <option>

After:

.arg(Arg::new("flag").long("flag").action(ArgAction::SetTrue))  # --flag
.arg(Arg::new("option").long("option"))  # --option <option>

In particular, num_args (the replacement for takes_value) will default appropriately
from the ArgAction and generally only needs to be set explicitly for the
other num_args use cases.

Breaking Changes

Subtle changes (i.e. compiler won't catch):

• arg! now sets one of (Bump tantivy version. #3795):
  • ArgAction::SetTrue, requiring ArgMatches::get_flag instead of ArgMatches::is_present
  • ArgAction::Count, requiring ArgMatches::get_count instead of ArgMatches::occurrences_of
  • ArgAction::Set, requiring ArgMatches::get_one instead of ArgMatches::value_of
  • ArgAction::Append, requiring ArgMatches::get_many instead of ArgMatches::values_of
• ArgAction::Set, ArgAction::SetTrue, and Arg::Action::SetFalse now
  conflict by default to be like ArgAction::StoreValue and
  ArgAction::IncOccurrences, requiring cmd.args_override_self(true) to override instead (Bump colored from 2.0.4 to 2.1.0 in /quickwit #4261)
• By default, an Args default action is ArgAction::Set, rather than ArgAction::IncOccurrence to reduce confusing magic through consistency (Store instrumentation scope #2687, Ensures the load of pipelines does not exceed 80%. #4032, see also Bump urllib3 from 2.0.4 to 2.0.7 in /quickwit/rest-api-tests #3977)
• mut_arg can no longer be used to customize help and version arguments, instead disable them (Command::disable_help_flag, Command::disable_version_flag) and provide your own (Spec and Implement the end of life of a shard. #4056)
• Removed lifetimes from Command, Arg, ArgGroup, and PossibleValue, assuming 'static. string feature flag will enable support for Strings (Put the index-uri parameter in the IndexConfig #1041, Consider ActorExitStatus::Quit as a successful exit? #2150, Issue/4184 assign shard logic #4223)
• arg!(--flag <value>) is now optional, instead of required. Add .required(true) at the end to restore the original behavior (Update to latest version of rust-rdkafka and remove consumer poll loop #4206)
• Added default feature flags, help, usage and error-context, requiring adding them back in if default-features = false (Add native support for Google Storage #4236)
• (parser) Always fill in "" argument for external subcommands to make it easier to distinguish them from built-in commands (Add support for ES exists query #3263)
• (parser) Short flags now have higher precedence than hyphen values with Arg::allow_hyphen_values, to be consistent with Command::allow_hyphen_values (Add the index ID and source ID in the warn logs #4187)
• (parser) Arg::value_terminator must be its own argument on the CLI rather than being in a delimited list (Improve behavior on corrupted checkpoint. #4025)
• (help) Line wrapping of help is now behind the existing wrap_help feature flag, either enable it or hard code your wraps (Instrument routers and ingesters #4258)
• (help) Make DeriveDisplayOrder the default and removed the setting. To sort help, set next_display_order(None) (Tracing: store trace and span IDs as byte fields #2808)
• (help) Subcommand display order respects Command::next_display_order instead of DeriveDisplayOrder and using its own initial display order value (Tracing: store trace and span IDs as byte fields #2808)
• (help) Subcommands are now listed before arguments. To get the old behavior, see Command::help_template (The ES compatible API accepts malformed requests #4132)
• (help) Help headings are now title cased, making any user-provided help headings inconsistent. To get the old behavior, see Command::help_template, Arg::help_heading, and Command::subcommand_help_heading (The ES compatible API accepts malformed requests #4132)
• (help) "Command" is used as the section heading for subcommands and COMMAND for the value name. To get the old behavior, see Command::subcommand_help_heading and Arg::subcommand_value_name (The ES compatible API accepts malformed requests #4132, Backward compatibilty tests #4155)
• (help) Whitespace in help output is now trimmed to ensure consistency regardless of how well a template matches the users needs. (The ES compatible API accepts malformed requests #4132, Make the ShardPositions model into a ShardPositionsService actor #4156)
• (help) name/version/author are removed by default from help output. To get the old behavior, see Command::help_template. (The ES compatible API accepts malformed requests #4132, (Minor fixes and improvements) Update README.md #4160)
• (help) Indentation for second-line usage changed. (The ES compatible API accepts malformed requests #4132, Update the swagger documentation on the POST mapping and ingest data #4188)
• (env) Parse --help and --version like any ArgAction::SetTrue flag (Normalize Error Messages #3776)
• (derive) Leave Arg::id as verbatim casing, requiring updating of string references to other args like in conflicts_with or requires (Increase default bucket limit? #3282)
• (derive) Doc comments for ValueEnum variants will now show up in --help ([Tracing] Span status is not indexed #3312)
• (derive) When deriving Args, and ArgGroup is created using the type's name, reserving it for future use (Bump tokio from 1.23.0 to 1.24.0 in /quickwit #2621, Bump openssl from 0.10.59 to 0.10.60 in /quickwit #4209)
• (derive) next_help_heading can now leak out of a #[clap(flatten)], like all other command settings (Bump @adobe/css-tools from 4.3.1 to 4.3.2 in /quickwit/quickwit-ui #4222)

Easier to catch changes:

• Looking up a group in ArgMatches now returns the arg Ids, rather than the values to reduce overhead and offer more flexibility. (Keyword repeat #4072)
• Changed Arg::number_of_values (average-across-occurrences) to Arg::num_args (per-occurrence) (raw CLI args, not parsed values) (Tracing: use pagination for queries fetching a huge number of spans #2688, Ingester close shards gRPC #4023)
  • num_args(0) no longer implies takes_value(true).multiple_values(true) (Ingester close shards gRPC #4023)
  • num_args(1) no longer implies multiple_values(true) (Ingester close shards gRPC #4023)
  • Does not check default or env values, only what the user explicitly passes in (Improve behavior on corrupted checkpoint. #4025)
  • No longer terminates on delimited values (Improve behavior on corrupted checkpoint. #4025)
• Replace Arg::min_values (across all occurrences) with Arg::num_args(N..) (per occurrence) to reduce confusion over different value count APIs (Ingester close shards gRPC #4023)
• Replace Arg::max_values (across all occurrences) with Arg::num_args(1..=M) (per occurrence) to reduce confusion over different value count APIs (Ingester close shards gRPC #4023)
• Replace Arg::multiple_values(true) with Arg::num_args(1..) and Arg::multiple_values(false) with Arg::num_args(0) to reduce confusion over different value count APIs (Ingester close shards gRPC #4023)
• Replace Arg::takes_value(true) with Arg::num_args(1) and Arg::takes_value(false) with Arg::num_args(0) to reduce confusion over different value count APIs
• Remove Arg::require_value_delimiter, either users could use Arg::value_delimiter or implement a custom parser with TypedValueParser as it was mostly to make multiple_values(true) act like multiple_values(false) and isn't needed anymore (Exposing pipeline metrics #4026)
• Arg::new("help") and Arg::new("version") no longer implicitly disable the
  built-in flags and be copied to all subcommands, instead disable
  the built-in flags (Command::disable_help_flag,
  Command::disable_version_flag) and mark the custom flags as global(true). (Spec and Implement the end of life of a shard. #4056)
• Arg::short('h') no longer implicitly disables the short flag for help,
  instead disable
  the built-in flags (Command::disable_help_flag,
  Command::disable_version_flag) provide your own Arg::new("help").long("help").action(ArgAction::Help).global(true). (Spec and Implement the end of life of a shard. #4056)
• ArgAction::SetTrue and ArgAction::SetFalse now prioritize Arg::default_missing_value over their standard behavior (Optimize searcher semaphore #4000)
• Changed Arg::requires_ifs and Arg::default_value*_ifs* to taking an ArgPredicate, removing ambiguity with None when accepting owned and borrowed types (Added support for v2 in the client. #4084)
• Removed PartialEq and Eq from Command so we could change external subcommands to use a ValueParser (Support compressed HTTP request bodies in REST API #3990)
• Various Arg, Command, and ArgGroup calls were switched from accepting &[] to [] via IntoIterator to be more flexible (Keyword repeat #4072)
• Arg::short_aliases and other builder functions that took &[] need the & dropped (Bumping quickwit's version to 0.6.5-dev #4081)
• ErrorKind and Result moved into the error module
• ErrorKind::EmptyValue replaced with ErrorKind::InvalidValue to remove an unnecessary special case (Refactoring of the CI. #3676, Quickwit doesn't respect URL prefix behind proxy #3968)
• ErrorKind::UnrecognizedSubcommand replaced with ErrorKind::InvalidSubcommand to remove an unnecessary special case (Refactoring of the CI. #3676)
• Changed the default type of allow_external_subcommands from String to OsString as that is less likely to cause bugs in user applications (Support compressed HTTP request bodies in REST API #3990)
• (help) Command::render_usage now returns a StyledStr (Control Plane somehow quits after a few minutes. #4248)
• (derive) Changed the default for arguments from parse to value_parser, removing parse support (reference blog post in cost section #3827, No messages outputted when indexing malformed json data #3981)
  • #[clap(value_parser)] and #[clap(action)] are now redundant
• (derive) subcommand_required(true).arg_required_else_help(true) is set instead of SubcommandRequiredElseHelp to give more meaningful errors when subcommands are missing and to reduce redundancy (Return an elasticsearch error response for elastic search endpoint #3280)
• (derive) Remove arg_enum attribute in favor of value_enum to match the new name (we didn't have support in v3 to mark it deprecated) (Bugfix: truncating the ingest v1 source on initialization. #4127)
• (parser) Assert when the CLI looksup an unknown args when external subcommand support is enabled to help catch bugs (validate tag timestamp #3703)
• (assert) Sometimes Arg::default_missing_value didn't require num_args(0..=N), now it does (Ingester close shards gRPC #4023)
• (assert) Leading dashes in Arg::long are no longer allowed (revisit parsing of configuration #3691)
• (assert) Disallow more value_names than num_args (Using rendez-vous hashing to sort nodes used for scheduling. #2695)
• (assert) Always enforce that version is specified when the ArgAction::Version is used
• (assert) Add missing #[track_caller]s to make it easier to debug asserts
• (assert) Ensure overrides_with IDs are valid
• (assert) Ensure no self-overrides_with now that Actions replace it
• (assert) Ensure subcommand names are not duplicated
• (assert) Assert on mut_arg receiving an invalid arg ID or mut_subcommand receiving an invalid command name

Compatibility

MSRV is now 1.60.0

Deprecated

• Arg::use_value_delimiter in favor of Arg::value_delimiter to avoid having multiple ways of doing the same thing
• Arg::requires_all in favor of Arg::requires_ifs now that it takes an ArgPredicate to avoid having multiple ways of doing the same thing
• Arg::number_of_values in favor of Arg::num_args to clarify semantic differences
• default_value_os, default_values_os, default_value_if_os, and default_value_ifs_os as the non _os variants now accept either a str or an OsStr (Remove IndexingStatus from IngestSource #4141)
• Arg::env_os in favor of Arg::env
• Command::dont_collapse_args_in_usage is now the default (Take multilang feature out of quickwit-doc-mapper/testsuite #4151)
• Command::trailing_var_arg in favor of Arg::trailing_var_arg to make it clearer which arg it is meant to apply to (Add the index ID and source ID in the warn logs #4187)
• Command::allow_hyphen_values in favor of Arg::allow_hyphen_values to make it clearer which arg it is meant to apply to (Add the index ID and source ID in the warn logs #4187)
• Command::allow_negative_numbers in favor of Arg::allow_negative_numbers to make it clearer which arg it is meant to apply to (Add the index ID and source ID in the warn logs #4187)
• (help) Deprecated Command::write_help and Command::write_long_help in favor of Command::render_help and Command::render_long_help (Control Plane somehow quits after a few minutes. #4248)
• (derive) structopt and clap attributes in favor of the more specific command, arg, and value to open the door for more features and clarify relationship to the builder ([Kafka source] Use consumer.recv() directly rather than consumer.stream() #1807, Fixed the serialize deserialization of the indexing state to include  #4180)
• (derive) #[clap(value_parser)] and #[clap(action)] defaulted attributes (its the default) (Bump serde_with from 3.3.0 to 3.4.0 in /quickwit #3976)

Behavior Changes

• (help) With wrap_help feature, if the terminal size cannot be determined, LINES and COLUMNS variables are used (update tantivy #4186)

Features

• Arg::num_args now accepts ranges, allowing setting both the minimum and maximum number of values per occurrence (Tracing: use pagination for queries fetching a huge number of spans #2688, Ingester close shards gRPC #4023)
• Allow non-bool value_parsers for ArgAction::SetTrue / ArgAction::SetFalse (Close fetch stream with an error if it does not reach EOF #4092)
• Add From<&OsStr>, From<OsString>, From<&str>, and From<String> to value_parser! (Shard load #4257)
• Allow resetting most builder methods
• Can now pass runtime generated data to Command, Arg, ArgGroup, PossibleValue, etc without managing lifetimes with the string feature flag (Consider ActorExitStatus::Quit as a successful exit? #2150, Issue/4184 assign shard logic #4223)
• New default error-context, help and usage feature flags that can be turned off for smaller binaries (Add native support for Google Storage #4236)
• Added StyledStr::ansi() to Display with ANSI escape codes (Control Plane somehow quits after a few minutes. #4248)
• (error) Error::apply for changing the formatter for dropping binary size (Bump axios from 1.5.1 to 1.6.1 in /quickwit/quickwit-ui #4111)
• (error) Error::renderfor formatting the error into a StyledStr
• (help) Show PossibleValue::help in long help (--help) ([Tracing] Span status is not indexed #3312)
• (help) New {tab} variable for Command::help_template ( Add local AGPL license link in LICENSE.md #4161)
• (help) Command::render_help and Command::render_long_help for formatting the error into a StyledStr (add documents about multi match and match phrase #3873, Control Plane somehow quits after a few minutes. #4248)
• (help) Command::render_usage now returns a StyledStr (Control Plane somehow quits after a few minutes. #4248)

Fixes

• Verify required is not used with conditional required settings (Disable multilang by default. #3660)
• Replaced cmd.allow_invalid_for_utf8_external_subcommands with cmd.external_subcommand_value_parser (nitpicks #3733)
• Arg::default_missing_value now applies per occurrence rather than if a value is missing across all occurrences (Apply query simplification optimization to find_trace #3998)
• arg!(--long [value]) to accept 0..=1 per occurrence rather than across all occurrences, making it safe to use with ArgAction::Append (Bump node from 18 to 21 #4001)
• Allow OsStrs for Arg::{required_if_eq,required_if_eq_any,required_if_eq_all} (Added support for v2 in the client. #4084)
• (help) With wrap_help feature, if the terminal size cannot be determined, LINES and COLUMNS variables are used (update tantivy #4186)
• (help) Use Command::display_name in the help title rather than Command::bin_name
• (help) Show when a flag is ArgAction::Count by adding an ... (feat: enable request bodies decompression #4003)
• (help) Use a more neutral palette for coloring (The ES compatible API accepts malformed requests #4132, Implement ingester graceful shutdown #4117)
• (help) Don't rely on ALL CAPS for help headers (The ES compatible API accepts malformed requests #4132, Bump tough-cookie from 4.1.2 to 4.1.3 in /quickwit/quickwit-ui #4123)
• (help) List subcommands first, focusing the emphasis on them (The ES compatible API accepts malformed requests #4132, Removing search/ingest specific client building method. #4125)
• (help) Do not include global args in cmd help help (Improve merge performance. #4131)
• (help) Use [positional] in list when relevant (Bound memory and disk usage at the queue level #4144)
• (help) Show all [positional] in usage (Take multilang feature out of quickwit-doc-mapper/testsuite #4151)
• (help) Polish up subcommands by referring to them as commands (The ES compatible API accepts malformed requests #4132, Backward compatibilty tests #4155)
• (help) Trim extra whitespace to avoid artifacts from different uses of templates (The ES compatible API accepts malformed requests #4132, Make the ShardPositions model into a ShardPositionsService actor #4156)
• (help) Hint to the user the difference between -h / --help when applicable (The ES compatible API accepts malformed requests #4132, Minor fixes & doc #4159)
• (help) Shorten help by eliding name/version/author (The ES compatible API accepts malformed requests #4132, (Minor fixes and improvements) Update README.md #4160)
• (help) When short help is long enough to activate next_line_help, don't add blank lines (The ES compatible API accepts malformed requests #4132, store split_fields in split #4190)
• (help) Make help output more dense (reducing horizontal whitespace) (The ES compatible API accepts malformed requests #4132, don't warmup redundantly an entire dictionary and some of its terms #4192)
• (help) Separate subcommand flags with "," like option flags (Avoid moving shards from one pipeline to another. instead: closed and open. #4232, Populates the control plane model will closed and open shards #4235)
• (help) Quote the suggested help flag (Add a header to be able to send logs or traces to a specified index throught the OTEL gRPC APIs #4220)
• (version) Use Command::display_name rather than Command::bin_name (fix sort order in UI #3966)
• (parser) Always fill in "" argument for external subcommands (Add support for ES exists query #3263)
• (parser) Short flags now have higher precedence than hyphen values with Arg::allow_hyphen_values, like Command::allow_hyphen_values (Add the index ID and source ID in the warn logs #4187)
• (parser) Prefer InvalidSubcommand over UnknownArgument in more cases (avoid unneeded copy on touching but non-overlaping byte ranges #4219)
• (derive) Detect escaped external subcommands that look like built-in subcommands (validate tag timestamp #3703)
• (derive) Leave Arg::id as verbatim casing (Increase default bucket limit? #3282)
• (derive) Default to #[clap(value_parser, action)] instead of #[clap(parse)] (reference blog post in cost section #3827)

[0xkelvin]

@kalil-pelissier i can help this

[fulmicoton]

@0xkelvin awesome! assinign the ticket to you!

[kalil-pelissier]

@0xkelvin I have no time to work on it at the moment so it's ok for me!

[EliKalter]

I am interested in collaborating with someone on their project so I can learn the language while they get a sidekick.
Let me Know if this is something that could work for you.
I Would need you to explain and guide me through my first steps in the language (I already started learning) and set the standard for me in code reviews but after a short while I'm sure I'll become an asset to the project.
(I have some experience in Typescript and Java and have learned C a few years ago).

[0xkelvin]

let me see can we pair-working on this, i still working on it, a bit slow because busy with full time job

@EliKalter connect me to discord : 0xqwert#2911

[EliKalter]

@0xkelvin I think you have a mix up, I am not a code owner nor am I a moderator

[fulmicoton]

@EliKalter I don't think there is any mix up. @0xkelvin is working on this, and you offered your help 2 weeks ago.

[fulmicoton]

It seems like someone sent a PR for this?
#3209

[EliKalter]

@0xkelvin said "connect me to discord", I can't connect him to anything,
Did he meant to say "contact me on discord"?

[0xkelvin]

It seems like someone sent a PR for this? #3209

oh, i was too slow, let me learn from it as well

[fmassot]

closed by #3209