REPL: JLine 3: fix various issues

[SethTisue]

adds some extra polish to #8036

• fixes JLine 3: tab completion behavior on tab after . is broken scala-dev#697
• fixes JLine 3: :paste is misbehaving (continuation prompt should be suppressed) scala-dev#703
• fixes JLine 3: replay session when entry seems to have slain the compiler scala-dev#705
• fixes JLine 3: parse errors should be suppressed during JLine-initiated tokenizing scala-dev#707
• fixes JLine 3: tab completion: showing method signatures is broken scala-dev#696
• fixes JLine 3: fix //print scala-dev#692

[dwijnand]

I'm pretty sure I've seen support around REPL replaying (SessionTest iirc). Isn't the fix for scala/scala-dev#705 testable?

[SethTisue]

Isn't the fix for scala/scala-dev#705 testable?

The problem was specifically when connected to an actual terminal.

Whether session replaying is now perfect, I don't know, my ambitions were limited (for now anyway) to fixing the immediate blocker that prevented it from working at all (on a real terminal).

[h-vetinari]

Has someone reached out to the spark maintainers about the (impact of the) JLine 3 changes / issues? It's 10 month after the release of 2.13.0, and they are still blocked on upgrading to 2.13 (and that despite @martijnhoekstra working tirelessly through the twitter dependency stack). One of the specific blockers is the REPL interaction, so it would IMO be a pity if something that's needed for spark-on-2.13 were to narrowly miss 2.13.2, and it would be another 4-6 month until they'd be unblocked by the following point release.

[lrytz]

Example that doesn't work in :paste -raw with the current tip of this PR:

Welcome to Scala 2.13.2-20200402-045016-1060b72 (Java HotSpot(TM) 64-Bit GraalVM EE 19.2.1, Java 1.8.0_231).
Type in expressions for evaluation. Or try :help.

scala> :paste -raw
// Entering paste mode (ctrl-D to finish)

package pa {
  object T
}

package pu {
  object U
}

java.lang.IllegalArgumentException: requirement failed: wordCursor 2 should be in range TokenData(105,51,52,false)
	at scala.tools.nsc.interpreter.jline.Reader$ScalaParsedLine.<init>(Reader.scala:176)

[lrytz]

@h-vetinari we haven't done cross-testing with spark because there is no 2.13 spark branch currently that we could use for that, so there is really no good way for us to test this. The only thing I could imagine is back-porting the JLine 3 upgrade to 2.12, but there are older refactorings in the REPL codebase in 2.13 which are much more likely to affect how spark integrates the REPL.

That said, we are eager to work with spark contributors on the 2.13 upgrade as soon as possible. We have chimed in on many of the 2.13-related tickets already. But as you note, the upgrade is currently blocked on issues outside our reach.

[h-vetinari]

@lrytz
Yeah, I wasn't talking about testing the JLine 3 things on spark yourselves, but maybe someone from the spark-maintainers would be able to immediately identify additional blocking issues among the remaining ones, for example.

It was just a thought; in any case, I think they'd very likely respond to a ping on GH/JIRA.

[SethTisue]

Example that doesn't work in :paste -raw with the current tip of this PR

@lrytz weird... I'm not able to reproduce the problem

[lrytz]

@lrytz weird... I'm not able to reproduce the problem

Hah, it seems to be unrelated to the code I posted, instead a trailing space trips it up. When I type class C then return in :paste mode I get

class C
java.lang.IllegalArgumentException: requirement failed: wordCursor 2 should be in range TokenData(10,6,7,true)
	at scala.tools.nsc.interpreter.jline.Reader$ScalaParsedLine.<init>(Reader.scala:176)
	at scala.tools.nsc.interpreter.jline.Reader$ScalaParser.tokenize(Reader.scala:157)

[SethTisue]

@lrytz ah, good catch, fixed.

[SethTisue]

This doesn't fix every last blocker, but I think it's about time to get this stuff merged, for wider testing.

@som-snytt do you have the time+inclination to do the review honors...?

(review from others is of course welcome as well!)

[SethTisue]

(rebased, no other changes)

[SethTisue]

merging, so we'll all be testing this. review feedback remains welcome