Usage parsing and generation should match docopt

[epage]

Issue by epage
Tuesday Oct 26, 2021 at 20:53 GMT
Originally opened as clap-rs/clap#2951

---

Affected Version of clap

3.0.0-beta.4

Expected Behavior Summary

App:from parses according to http://docopt.org/

App::render_usage generates output that matches http://docopt.org/

Actual Behavior Summary

At least with "required", we are using <> instead of ()

See clap-rs/clap#2947

Notes

• Can we do develop this behind a feature flag?
  • Avoid this having to be done in a breaking-change-specific branch, pushing towards large, big releases
  • Allow people to opt-in to it early and start collecting feedback
• Should we stablize the usage generation separate from usage parsing? Its not a breaking change but having them match probably helps the person using App::from
• Can we support parsing both grammars at once? This would let us stablize parsing and give people room to transition

[epage]

Comment by pksunkara
Tuesday Oct 26, 2021 at 21:04 GMT

---

• What does docopt generate?
• How do we differ from it?
• Why do we think docopt is correct in those instances?

[epage]

Comment by epage
Tuesday Oct 26, 2021 at 21:16 GMT

---

What does docopt generate?

DocOpt is a convention for describing usage syntax, especially for creating arg parsers off of it.

How do we differ from it?

As mentioned in the Issue, we at least differ in syntax for required (<> vs ()). I've not exhaustively looked at where else we might differ, I see that as part of resolving this issue. I have a tiny hope that there is some kind of conformance suite ("tiny" because I see that being difficult).

Why do we think docopt is correct in those instances?

I believe @kbknapp said that the goal of App::from was to have an API similar to the docopt crate. That crate, while unmaintained, was an implementation of docopt. So by extension, clap is already meaning to support http://docopt.org/

The other benefit is the closer we are to a common syntax, the more likely people will be able to translate existing knowledge when programming in Rust which makes them more successful faster and are less reliant on how perfect / complete our documentation is.

If we decide docopt support isn't important, or if usage and parser can diverge, I'm fine with that. I'm more so just describing how I see the situation. Personally, I'll never use the docopt parser and the usage generation is good enough.

[epage]

Comment by kbknapp
Tuesday Oct 26, 2021 at 21:37 GMT

---

I could be wrong and misremembering, but I thought I remembered @BurntSushi mentioning once that there either wasn't a real spec, or it wasn't clear?

One the main issues and why I never fully pursued this further was docopt I believe relies on "whole document" information to infer some settings for arguments and goes through great lengths to make good guesses. clap's "usage parsing" is much more strict, but also tried to encode things that docopt has no concept of.

Ultimately, this was something I felt could be better handled by a crate wrapping clap, than clap itself. Like our derive and generating crates, I wouldn't mind them being part of this org, but a true docopt implementation is too much for clap proper.

[epage]

Comment by pksunkara
Tuesday Oct 26, 2021 at 21:46 GMT

---

So, there are two things here.

1. Is docopt the convention? Is there a spec for this or can we interpolate it by examining how existing non-docopt based tools behave?
2. Usage parser in clap.

For the second item, I think we are all aligned that it can be a separate crate. I personally don't mind either but it might be difficult because we would want to implement impl From<&str> for Arg in that crate and rust doesn't allow it IIRC. Anyway, that can be a separate discussion later on.

Let's focus this issue on the first item.

I personally think docopt is not the convention and <> actually conveys that it is required better than ().

---

I've not exhaustively looked at where else we might differ, I see that as part of resolving this issue.

I asked the question for future so that we can actually do that when we work on this.

[epage]

Comment by kbknapp
Tuesday Oct 26, 2021 at 22:59 GMT

---

it might be difficult because we would want to implement impl From<&str> for Arg in that crate and rust doesn't allow it IIRC

Correct. For that exact method we'd have to create and expose some kind of IntoArg trait. However, there is nothing preventing a crate from doing this without traits and just a bare function fn docopt_to_clap(&str) -> App. Is that optimal? No, not exactly. But at least for the experimentation phase and such it's perfectly fine.

[epage]

Comment by 0x7FFFFFFFFFFFFFFF
Wednesday Oct 27, 2021 at 00:31 GMT

---

I personally think docopt is not the convention and <> actually conveys that it is required better than ().

What do you think of {}?

[epage]

Comment by kbknapp
Wednesday Oct 27, 2021 at 00:58 GMT

---

What do you think of {}?

The only specification I've seen only says, (paraphrasing) "square brackets [] mean an argument is optional. Lack of square brackets mean the argument is required in that synopsis."

I've anecdotally seen <> or no brackets convey meaning the argument is required in 99% of all circumstances. I'd argue against anything beyond either of those options. And the no-brackets version sometimes has parsing difficulties when writing a general parser. The reason clap originally used <> (beyond it being a decently common convention) is it helps disambiguate positional arguments from subcommands.

[epage]

Comment by epage
Wednesday Oct 27, 2021 at 02:24 GMT

---

Ultimately, this was something I felt could be better handled by a crate wrapping clap, than clap itself. Like our derive and generating crates, I wouldn't mind them being part of this org, but a true docopt implementation is too much for clap proper.

I only had in mind getting ours more inline with docopt. Are you suggesting we split our existing usage parser into its own crate or if someone wanted a to go full docopt?

[epage]

Comment by BurntSushi
Wednesday Oct 27, 2021 at 12:37 GMT

---

It's been a long time since I've even thought about docopt, but IIRC, docopt.org is pretty much the only thing that exists for a spec. I remember following that and the test cases for the reference implementation.

IMO, this is a great piece of "fat" that could be trimmed from Clap.

It's also worth noting that the docopt project itself (not just the docopt Rust library) has been effectively unmaintained for many years.

[epage]

Comment by epage
Friday Oct 29, 2021 at 16:38 GMT

---

IMO, this is a great piece of "fat" that could be trimmed from Clap.

What are thoughts on deprecating the usage parser for 3.0, like we did the macro API?

• We'd want to revert the API changes since that is unnecessary churn.

[epage]

Comment by pksunkara
Friday Oct 29, 2021 at 16:41 GMT

---

I have seen quite a high number of people use it. Every alternate issue that we get seems to use it. Maybe we can mark it as deprecated (without reverting changes) but offer it as a side crate later?

[epage]

Comment by epage
Friday Oct 29, 2021 at 16:52 GMT

---

Thanks for the input on its usage.

If we do anything but plan to keep it as-is long term, we should revert the 3.0 API changes to reduce churn for those using it.

[epage]

Comment by pksunkara
Friday Oct 29, 2021 at 16:57 GMT

---

After 3.0, maybe we can do some metric collection and then decide on it?

[epage]

Comment by epage
Friday Oct 29, 2021 at 17:00 GMT

---

The only easy form of metric collection is crates..io. This is why I've been curious about the idea of breaking out each API into a distinct crate so we actually know what people use. Not saying its a great idea, it optimizes for one case but pessimizes for others.

[epage]

Comment by pksunkara
Friday Oct 29, 2021 at 17:06 GMT

---

Agree, We can definitely explore in that direction after 3.0

[epage]

Comment by epage
Friday Oct 29, 2021 at 17:13 GMT

---

If we are looking to radically change the API after 3.0, should we revert the API changes we did make?

• It leads to extra churn for the users
• The 3.0 API is a lot harder to search for because its implicit (implemented in From and the API accepts impl Into)

[epage]

Comment by epage
Tuesday Nov 09, 2021 at 01:19 GMT

---

As an alternative, what if we expose a subset of clap_app!, as clap_arg!, to provide compile-time usage parsing as a replacement for runtime usage parsing?