SI-12290: support JDK15 text blocks in Java parser

[harpocrates]

JDK15 introduced text blocks (JEP 378) for writing multiline strings.
This adds support for parsing these strings in the Java parser.

The logic for interpretting the literals is a little complicated, but
follows from the "3.10.6. Text Blocks" of the Java language specification.
The test cases include examples from there and from the JEP.

Fixes scala/bug#12290

[lrytz]

Thank you and welcome! :)

[lrytz]

Oh another detail: \s is a valid escape sequence now in Java (not only for text blocks, also for strings and char literals). So it should be added to getlitch.
https://docs.oracle.com/javase/specs/jls/se16/html/jls-3.html#jls-3.10.7

[harpocrates]

Oh another detail: \s is a valid escape sequence now in Java (not only for text blocks, also for strings and char literals).

I added support for \s and also for escaping newlines in text blocks (trailing \). All escape sequences should be covered now.

---

I've been force-pushing updates (since I read that Scala prefers to keep a clean commit log), but I can also just add commits if it is easier to review and squash them at the end.

[SethTisue]

I've been force-pushing updates (since I read that Scala prefers to keep a clean commit log), but I can also just add commits if it is easier to review and squash them at the end.

Either way is fine. We do it either way depending on the nature of the PR and on individual preference, rather than trying to establish a firm policy.

If in doubt, I would say default to squashing at the end, especially on anything where the reviewing gets complicated or appears likely to get complicated.

[lrytz]

Gave it another pass... But we're close!

I'll talk to Seth next week about testing. It would be good if we could mark tests as Java N+ and have some CI job that runs those regularly.

[lrytz]

Instead of / before returning, maybe we try to advance reader to the end of the text block? Otherwise there are a lot of errors issued when continuing to parse.

Although javac is not any better :-) So I'm fine if you leave it.

[harpocrates]

Maybe just not returning would do the trick? I'll try this out after work.

[lrytz]

Looks good now!

We'll get back to you about the testing situation.

A few more tests could be added

• mixed indentation with tabs and spaces (each count 1 character of indentation)
• escaped new line \
• \uXXXX escapes are handled before the text block (\u0020 is leading whitespace, \s is not)
• invalid text blocks
  • non-whitespace after initial """
  • difference to scala, which just uses the last 3 " to close a multiline string """this is "fine""""
• maybe CRLF, FF, exoctic unicode whitespaces...

[harpocrates]

I've added some extra tests to cover the cases you mentioned. I imagine that depending on how testing plays out there may be some modification needed to indicate the tests only run on JDK 14+. I'll be looking out for those updates to testing.

Thanks for the patient reviews!

[SethTisue]

WIP PR adding JDK 16 to Travis-CI: #9579

[SethTisue]

#9579 has landed, but now this is waiting on #9587

[som-snytt]

I don't think lrytz wanted to wait.