Syntax error recovery

[odersky]

This is a PR to improve syntax error recovery. It's meant to be integrated with one of the PRs by @adampauls to do trailing comma detection in Parser instead of Scanner.

I fixed the two CB issues that had illegal trailing commas. Here's the transcript of what I did. @anatoliykmetyuk might be able to point to a more worked out tutorial for this.

For specs2:

> cd community-build/community-projects/specs2
> git checkout -b fix-trailing-comma

Fix problem in editor

> git add -u
> git status
On branch fix-trailing-comma
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	modified:   core/shared/src/main/scala/org/specs2/matcher/ActionMatchers.scala

> git commit -m "fix trailing comma error"
[fix-trailing-comma e1ae96e7a] fix trailing comma error
 1 file changed, 1 insertion(+), 1 deletion(-)
> git push --set-upstream origin fix-trailing-comma
Enumerating objects: 21, done.
Counting objects: 100% (21/21), done.
Delta compression using up to 16 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (11/11), 732 bytes | 732.00 KiB/s, done.
Total 11 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
remote: 
remote: Create a pull request for 'fix-trailing-comma' on GitHub by visiting:
remote:      https://github.com/dotty-staging/specs2/pull/new/fix-trailing-comma
remote: 
To https://github.com/dotty-staging/specs2.git
 * [new branch]          fix-trailing-comma -> fix-trailing-comma
Branch 'fix-trailing-comma' set up to track remote branch 'fix-tr

Then, in the dotty directory:

> git add ../../community-build/community-projects/specs2
> git commit -m "fix trailing commas in community build"
> git push staging HEAD

Fixes #14507

[odersky]

I also had to disable two scalatest tests: ExpectationsSpec and DirectExpectationsSpec. These were test suites containing tests that checked the precise error message output of the dotty compiler. IMO such tests should never be in a regression test suite for the community build, since they essentially lock in current behavior. We do want tests about error messages of course, but these should be compiler tests that can easily be changed if the compiler's error diagnostics improve.

Not knowing scalatest, I was not able to figure out from the diagnostics which particular tests broke in the two suites, so I disabled them wholesale. I'll open a separate issue to bring back tests that are OK.

[odersky]

We should do much better now with error recovery which should mean many fewer spurious errors in compiles and in the IDE. Previously, a mismatch of some kind of parenthesis could lead to a large part of the source being skipped without parsing. This in turn could lead to many follow-on type errors errors since definitions were not recognized.

The new improvements are:

• When skipping we are more robust when dealing with mismatched parentheses. A missing closing parenthesis is now implicitly assumed when skipping determines that the parenthesis is probably missing. It does so by taking the indentation level into account. This means that outdents are recognized and inserted when skipping in a region opened by (. Also, that the region is discarded once an outdent goes back to at most of the indention level of the region enclosing the (.

  Before, such a missing parenthesis would have often triggered skipping until the next enclosing }. Which is particularly bad if your code uses indentation instead of braces; then it would skip to the end.

• A closing parenthesis (or }, or ]) is treated as a stop symbol for skipping only if there is a matching open parentheses. Previously, the stop symbol was unconditional, which means that recovery after skipping would tend to unwind the stack too far before advancing.

• A closing OUTDENT is treated as a stop symbol if it has the right indentation width.

• A comma is treated as a stop symbol if a comma is expected by some enclosing production. Previously, commas were never stop symbol, which meant we always skipped pass them.

[tgodzik]

Just one minor nitpick, otherwise looks great!

[tgodzik]

Looks like assumedOutdent is not needed?

[odersky]

Oh yes, that was a remainder of something else I tried that led nowhere.