Stablize Rust-Native Completion Engine Tracking Issue

[epage]

Maintainer's notes:

Remaining work for feature parity

• Communicate with bash (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete path ValueHints (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete possible values (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete short flags (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete long flags (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete subcommands (feat(complete): Skeleton for Rust-driven completions #3656)
• Communicate with zsh (zsh support for native completions #3916)
• Communicate with fish (fish support for native completions  #3917)
• Communicate with Powershell (powershell support for native completions  #3918)
• Communicate with Elvish (elvish support for native completions  #3919)
• Flag values (Support flags with values in native completions #3920)
• Multiple values (Support multiple values in native completions #3921)
• Hidden (Support hiding flags/subcommands/possible values for native compeletions #3951)
• Correctly handle appending a space
• Fix handling of OsString to preserve non-UTF8 paths
• Document that stdout should not be used before getting to completion processing in case the completions are requested
• FIGNORE support?

Non-blocking work

• Delimited values (Support delimited values for native completions #3922)
• is_require_equal_set support (Support is_require_equal_set in native completions #3923)
• Support subcommand flags in native completion engine #5284 (doesn't appear supported by static completions)
• Removing options based on conflicts (self and others)
• Any App or Arg setting not currently handled by the static completions
  • Priority to value delimiter and requires equals
• Complete more ValueHints
• Figure out where this will live as the dependency requires are dramatically different than the static completions and this shouldn't be behind a feature flag for forever
• Smarter selection of what to show (Dynamic Completions: Filter and order completions based on input and importance #5058)
• Custom help messages (Support custom help messages for completions #5063)
• Completing multiple path elements at once (Complete multiple path elements at once #5279)

Design considerations

• Make it easy to use for the natively supported shells but allow other shells to be supported

Prior art

• https://github.com/pacak/bpaf
• https://github.com/rsteube/carapace
• https://pypi.org/project/argcomplete

---

#3022 was a tipping point for me in realizing that maybe our current approach to completions doesn't work. We effectively have to implement a mostly-untested parser within each shell. Examples of other problems that seem to stem from this:

• clap_generate fish: ArgEnum completions not working on toplevel command #2729
• Allow completion of partial commands [Powershell] #2750
• clap_generate PowerShell script does not complete case sensitive options like -S. Only completes -s #2145
• zsh: optional flag with takes_value breaks completions #1822
• Allow the completed command to be an alias #1764

If we take the approach of argcomplete where we do the parsing in our core code, rather than in each completion script, this will help us share parsing logic between shell, share some or all parsing logic with clap itself, and make a subset of the logic more testable.

We also need to decide whether to morph the existing parser into supporting this or create a custom parser (since the needs are pretty special). If we do a custom parser, we should probably do #2915 first so we can reuse lexing logic between clap and the completion generation. I've also been considering #2912 which would allow reusing the completion logic with any CLI parser.

[epage]

Bash's expectations

Inputs for -F and -C:

• COMP_LINE: The current command line. This variable is available only in shell functions and external commands invoked by the programmable completion facilities
• COMP_POINT: The index of the current cursor position relative to the beginning of the current command. If the current cursor position is at the end of the current command, the value of this variable is equal to ${#COMP_LINE}. This variable is available only in shell functions and external commands invoked by the programmable completion facilities
• COMP_KEY: The key (or final key of a key sequence) used to invoke the current completion function.
• COMP_TYPE: Set to an integer value corresponding to the type of completion attempted that caused a completion function to be called: TAB, for normal completion, ‘?’, for listing completions after successive tabs, ‘!’, for listing alternatives on partial word completion, ‘@’, to list completions if the word is not unmodified, or ‘%’, for menu completion. This variable is available only in shell functions and external commands invoked by the programmable completion facilities
• 1: name of the command whose arguments are being completed
• 2: word being completed
• 3: word preceding the word being completed on the current command line

Inputs for -F:

• COMP_WORDS, COMP_CWORD when used with -F

Output for -C:

• print a list of completions, one per line, to the standard output. Backslash may be used to escape a newline, if necessary.

Output for -F:

• It must put the possible completions in the COMPREPLY array variable, one per array element.

See

• https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html#Programmable-Completion
• https://www.gnu.org/software/bash/manual/html_node/A-Programmable-Completion-Example.html#A-Programmable-Completion-Example
• https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion-Builtins.html#Programmable-Completion-Builtins
• https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html

[epage]

argcomplete emulates bash's interface in fish by

    set -x COMP_LINE (commandline -p)
    set -x COMP_POINT (string length (commandline -cp))
    set -x COMP_TYPE

and then just registers that function

https://github.com/kislyuk/argcomplete/blob/develop/argcomplete/shell_integration.py#L60

[epage]

Value hints are supported on

• zsh
• fish

Tooltips are supported on

• zsh
• powershell
• elvish
• fish

We should make sure we don't lose these features as part of this transition, ie we shouldn't drop to the lowest common denominator.

[epage]

For Powershell

Register-ArgumentCompleter
        -CommandName <String[]>
        -ScriptBlock <ScriptBlock>
        [-Native]
        [<CommonParameters>]

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/register-argumentcompleter?view=powershell-7.2

The block receives

When you specify the Native parameter, the script block must take the following parameters in the specified order. The names of the parameters aren't important because PowerShell passes in the values by position.

• $wordToComplete (Position 0) - This parameter is set to value the user has provided before they pressed Tab. Your script block should use this value to determine tab completion values.
• $commandAst (Position 1) - This parameter is set to the Abstract Syntax Tree (AST) for the current input line. For more information, see Ast Class.
• $cursorPosition (Position 2) - This parameter is set to the position of the cursor when the user pressed Tab.

The block provides CompletionResult

The CompletionResult object allows you to provide additional details to each returned value:

• completionText (String) - The text to be used as the auto completion result. This is the value sent to the command.
• listItemText (String) - The text to be displayed in a list, such as when the user presses Ctrl+Space. This is used for display only and is not passed to the command when selected.
• resultType (CompletionResultType) - The type of completion result.
• toolTip (String) - The text for the tooltip with details to be displayed about the object. This is visible when the user selects an item after pressing Ctrl+Space.

So it seems like Powershell can fit within rust-driven completions and provide the full feature set.

[epage]

I'm starting small and prototyping for just bash support. Looking at argcomplete, it seems they had to use their own shlex implementation. Hoping we can avoid that. The first step is comex/rust-shlex#12.

[happenslol]

Hi, I'm interested in helping with this this I'm developing a few personal tools using clap that would hugely benefit from dynamic completions. Is there any good issues that nobody is working on you could point me towards?

[epage]

@happenslol Thanks!

Anything unchecked in the "Remaining work" section is up for grabs; just post here that you are getting started on it. The highest priority is the support for each shell as that will help gauge the feasibility and provide feedback on the API. The rest is flushing out parsing implementation.

I should split those out into individual issues to make it easier for people to coordinate on those (and to add bounties) but I probably won't have time for another week or so.

[happenslol]

Alright, had a busy week, so I'm only getting back around to this now.

I've looked at the work done in #3656, and it looks like the next steps to add zsh and fish support would be to pull some of the functionality out of clap_complete::dynamic::bash, and adapt it to the other shells to provide the same functionality. Am I correct in assuming that?

I'd look at argcomplete for the other implementations here, since the current one also seems to be coming from there.

[epage]

@happenslol Yes, that is correct. I'd like to focus on shell support initially as that is where the most unknowns exist

[happenslol]

Started work here. I'm looking at cobra for zsh completions since argcomplete basically takes the easy way out and tells people to enable bash completion support for zsh 😅

[epage]

Oh if cobra is doing the same type of dynamic completions as argcomplete, that is great! That'll serve as a much better example!

[blaggacao]

I want to apport a data point:

• https://github.com/rsteube/carapace

Implements cobra completion with advanced things like a fs cache for dynamic completion querys (e.g. with provenience from remote APIs) for an x amount of seconds.

Imagine a aws CLI wrapper that interacts with your custom infrastructure and things like it.