Support MatchType for SemanticDB #14608

tanishiking · 2022-03-02T13:29:18Z

/generated sources are generated from tanishiking/semanticdb-for-scala3#50
See the discussion about protobuf schema here scalameta/scalameta#2414

For example, the following program

type Elem[X] <: Any = X match
  case String => Char
  case Array[t] => t

is modeled as:

TypeSignature(
  Some(Scope(List(matchtype/MatchType$package.Elem#[X]),List())),
  MatchType(
    TypeRef(Empty,matchtype/MatchType$package.Elem#[X],List()),
    List(
      CaseType(
        TypeRef(Empty,scala/Predef.String#,List()),
        TypeRef(Empty,scala/Char#,List())
      ),
      CaseType(
        TypeRef(Empty,scala/Array#,List(TypeRef(Empty,local0,List()))),
        TypeRef(Empty,local0,List())
      )
    )
  ),
  TypeRef(Empty,scala/Any#,List())
)

Once this PR is approved I'm going to create a PR that support MatchType extracted from scalameta/scalameta#2414 :)

Generated from tanishiking/semanticdb-for-scala3#1 See the discussion here scalameta/scalameta#2414

bishabosha · 2022-03-02T13:46:24Z

to merge this we would have to wait for scalameta/scalameta#2414 to be approved?

tests/semanticdb/metac.expect

tanishiking · 2022-03-03T08:47:30Z

to merge this we would have to wait for scalameta/scalameta#2414 to be approved?

Actually, scalameta/scalameta#2414 has already been approved and was waiting for the Scala3 implementation (to see the protobuf schema really works for Scala3).

Anyway, I'll work on #14608 (comment) as they should be typed as you mentioned.

bishabosha · 2022-03-03T09:00:29Z

maybe unrelated but seeing as this will only be released for 3.2.0 or beyond, it seems a shame to not be able to support the same semanticdb for older scala 3 releases - what can we do about this? @tgodzik

tgodzik · 2022-03-03T19:06:38Z

maybe unrelated but seeing as this will only be released for 3.2.0 or beyond, it seems a shame to not be able to support the same semanticdb for older scala 3 releases - what can we do about this? @tgodzik

I can't really think of anything :/ We would need to split it out to a plugin again like with Scala 2.

bishabosha · 2022-03-03T19:49:55Z

I can't really think of anything :/ We would need to split it out to a plugin again like with Scala 2.

I had a an idea that maybe there could be backports of semanticdb changes to the old release branches of previous minor versions 🤣

tgodzik · 2022-03-03T20:00:45Z

I can't really think of anything :/ We would need to split it out to a plugin again like with Scala 2.

I had a an idea that maybe there could be backports of semanticdb changes to the old release branches of previous minor versions rofl

😨

tanishiking · 2022-03-04T12:58:01Z

compiler/src/dotty/tools/dotc/semanticdb/ExtractSemanticDB.scala

+            else if tree.rhs.isEmpty then
+              symkinds += SymbolKind.Abstract
+          // if symbol isType, it's type variable
+          case tree: Bind if (!tree.symbol.isType) =>


Added !tree.symbol.isType here, to misinterpret the type variable symbol as Val
This change makes type variables like local0 => case val method N$1 <: Nat to local0 => case type N$1 <: Nat.

Other diffs: just indented.

tanishiking · 2022-03-04T13:00:21Z

compiler/src/dotty/tools/dotc/semanticdb/TypeOps.scala

+          // prefix symbol (e.g. "scala/") + name (e.g. "Nothing") + Global symbol suffix ("#")
+          // see: https://scalameta.org/docs/semanticdb/specification.html#symbol
+          val ssym = s"${pre.typeSymbol.symbolName}${sym.mangledString}#"
+          s.TypeRef(s.Type.Empty, ssym, Seq.empty)


x *: xs part of

type Concat[Xs <: Tuple, +Ys <: Tuple] <: Tuple = Xs match case EmptyTuple => Ys case x *: xs => x *: Concat[xs, Ys]

have TypeRef(..., Nothing) (where Nothing is just a Name instead of Symbol), and we can't access the symbol for Nothing here. Therefore, craft the symbol for TyperRef for this.

tanishiking · 2022-03-04T13:07:09Z

Fixed #14608 (comment)

I can't really think of anything :/ We would need to split it out to a plugin again like with Scala 2.
I had a an idea that maybe there could be backports of semanticdb changes to the old release branches of previous minor versions 🤣

As this feature is kinda optional, I believe we don't need to backport to the older minor versions :)
(Though older version of Scala3 won't be able to show the inferred type of MatchType in Metals, it's ok I guess).

tanishiking · 2022-03-16T13:21:01Z

Now it's ready for review :)

tests/semanticdb/metac.expect

`t` of `case List[t] =>` has `CaseClass` flag. Maybe we should remove `CaseClass` flag from `t`, but as a workaround remove `Case` property from semanticdb from type variables.

bishabosha

nice! thank you

tanishiking · 2022-03-17T10:18:18Z

Can you wait for merging, until scalameta merges the PR to update the schema, just in case :) ?
~~I'll merge by hand, when it's prepared.~~ (No, I don't have a merge right, of course XD, I'll tell you :)

bishabosha · 2022-03-17T13:14:32Z

ok I have added the stat:do not merge label

tanishiking · 2022-03-17T16:37:16Z

@bishabosha Thanks! It's now ready for review :)

bishabosha · 2022-03-22T12:53:47Z

tests/semanticdb/metac.expect

+advanced/Structural#T#[A] => typeparam A 
+advanced/Structural#T#`<init>`(). => primary ctor <init> [typeparam A ](): T[A]
+advanced/Structural#T#bar(). => method bar (param b: <?>): Unit
+advanced/Structural#T#bar().(b) => param b: <?>


b should still have a type here - it seems a symbol does exist for foo.B because apparently it is local12

Edit: I have now seen your comment :) #14608 (comment)

tanishiking · 2022-03-22T12:53:50Z

The simplest example where tr @ TypeRef(pre, sym: Name) and tr.symbol == NoSymbol is foo.B (TypeRef to refinement (type B) in RefinedType foo : { ... }) in

https://github.com/lampepfl/dotty/blob/18b89018d090a9c38939c59f24098b89261b0f3f/tests/pos/t8177e.scala#L2

I guess this is because refinement actually doesn't have a symbol (we dealt with it by fake symbol). The problem is, unlike TypeParamRef, TypeRef doesn't know which RefinedType the name is bound (I'm trying to figure out how can we retrieve the symbol for foo.B 🤔

Added test: https://github.com/lampepfl/dotty/pull/14608/files#diff-18e8342519c9145a646a9b86ca13f58300b3fff458b90420ef3bb7e3e8bdd56fR75

bishabosha · 2022-03-22T12:59:38Z

I guess this is because refinement actually doesn't have a symbol (we dealt with it by fake symbol). The problem is, unlike TypeParamRef, TypeRef doesn't know which RefinedType the name is bound (I'm trying to figure out how can we retrieve the symbol for foo.B 🤔

what happens if you resolve the prefix to the symbol for foo, its info should be a refined type that defines B maybe?, or perhaps you can call pre.member(name)

bishabosha · 2022-03-22T13:03:56Z

Also see here for how tasty resolves selections of a type from prefix that has a refined type:
https://github.com/lampepfl/dotty/blob/ab885bfa66abe01bead62052333d20967068ff79/compiler/src/dotty/tools/dotc/core/tasty/TreeUnpickler.scala#L1110

tanishiking · 2022-03-22T13:38:47Z

Ok, we can access RefinedType by <foo>.widen

scala#14608 (comment)

compiler/src/dotty/tools/dotc/semanticdb/TypeOps.scala

tests/semanticdb/expect/dep-match.scala

bishabosha · 2022-03-30T09:37:15Z

tests/semanticdb/expect/Advanced.expect.scala

+
+  // see: https://github.com/lampepfl/dotty/pull/14608#discussion_r835642563
+  lazy val foo/*<-advanced::Test.foo.*/: (reflect.Selectable/*->scala::reflect::Selectable#*/ { type A/*<-local16*/ = Int/*->scala::Int#*/ }) &/*->scala::`&`#*/ (reflect.Selectable/*->scala::reflect::Selectable#*/ { type A/*<-local17*/ = Int/*->scala::Int#*/; val a/*<-local18*/: A/*->local17*/ }) = ???/*->scala::Predef.`???`().*/
+  def bar/*<-advanced::Test.bar().*/: foo/*->advanced::Test.foo.*/.A = foo/*->advanced::Test.foo.*/.a


do you have an idea why .A and .a do not have occurrences here?

I guess this is because .A and .a (access the field of refinements), don't have symbols (we're crafting fake symbols for them), and ExtractSemanticDB doesn't resolve the fake symbols by name.

We can see the same problem around here:
https://github.com/lampepfl/dotty/blob/60c336ef34fe0e77851a7652b910db13b84dfe6f/tests/semanticdb/expect/Advanced.expect.scala#L17

I'll take a look 👀

I guess looking again registerUse or registerUseGuarded still only assume real symbols, for example Select tree checks that the symbol exists, without checking for a registered fake symbol - this could be a follow up PR

As per foo.A, b964694 as you mentioned, you're right, we could fix the issue by looking up the fake symbols and registerUse them.

In terms of foo.a (field access to structural type), it is transformed to foo.selectDynamic("a") by Typer here ↓
https://github.com/lampepfl/dotty/blob/b9646942facccdcc110753f019c14b941fae870d/compiler/src/dotty/tools/dotc/typer/Dynamic.scala#L49-L65
We have to check if qual is something like Apply(Select(Ident(foo), name), List(Literal(Constant("a")))) if name == nme.applyDynamic || name == nme.selectDynamic || name == nme.updateDynamic || name == nme.applyDynamicNamed, and lookup symbol for a from foo (This is not yet fixed, can I work on this in another PR?)

I think this is fine for another PR

tests/semanticdb/expect/Advanced.expect.scala

tanishiking · 2022-03-31T13:36:23Z

I noticed that .A is local19, which does not have a definition occurrence.

Realized, we just failed to register symbols to refinementSymtab, we didn't need to register fake symbols :) fixed in 9375ab1

Let's see tests pass... (some failed in the previous commit)

bishabosha · 2022-03-31T15:11:16Z

Errors found on incorrect row numbers when compiling out/checkInit/neg/i5854
[1535](https://github.com/lampepfl/dotty/runs/5771849436?check_suite_focus=true#step:9:1535)
-> following the errors:
[1536](https://github.com/lampepfl/dotty/runs/5771849436?check_suite_focus=true#step:9:1536)
 at 0: Internal error in extracting SemanticDB while compiling tests/init/neg/i5854.scala: Ignoring A of symbol class Object

looks like the some new warnings are causing tests to have extra errors

```scala val a: String = (((1: Any): b.A): Nothing): String val b: { type A >: Any <: Nothing } = loop() ``` We can't resolve a symbol of `b.A` in `(1: Any): b.A` because semanticdb generator basically resolves the symbol of refinement members (like `type A` of `b`) by - Traverse trees and collect symbol information by **depth-first** way - during the traversal, we register the symbols of refinement members to a symbol table. - Then outer part of program can resolve the symbol of refinements by looking up symbols from the symbol table. However, in this case, we failed to resolve the symbol of b.A bacause - (1) We try to lookup the symbol of `b.A` first, which has not yet registered to the symtab. - (2) And then register a symbol for A in b by traversing `b`. Maybe we could fix this issue by - (a) Generate fake symbol for `b.A` in (1), and register it to the symtab - (b) in (2), when we register the "real" symbol of `A`, it should collide with the symbol registered in step (a) - (c) if they had collision, we mark those symbols (fake one and real one) as an alias - (d) on building the semanticdb symbol, we use the same symbol for both fake one and real one

tanishiking · 2022-04-01T09:22:00Z

I just suppressed the warning in ba24fe1 and added a test case for semanticdb.
Though I have some ideas to fix this issue, it may take some time. So let's ignore the issue in this PR and fix it in a separate PR/issue.

I found a problem in dealing with the refinement symbols (like i5854.scala). In this file, we fail to look up the symbol of b.A because we current ExtractSemanticDB depends on the order of traverse in looking up the symbol of refinements. In this case:

(1) We try to look up the symbol of b.A first, which has not yet registered to the refinementSymtab.
(2) And then register a symbol for A in b when traversing val b: { ... }.

Maybe we could fix this issue by

(a) Generate a fake symbol for b.A in (1) and register it to the symtab
(b) And see the symbols (fake symbol generated in step (a) and real symbol retrieved in step (2)) as a kind of alias (just remember they represent the same symbol).
(c) Generate the same SemanticDB symbol (localX) for those symbols.

bishabosha · 2022-04-01T16:22:01Z

Thank you for all your rest work! I agree that this is basically ready to merge, would you be able to make an issue to describe the problem with traversal order? then let's merge.

tanishiking · 2022-04-02T14:06:41Z

Sure! I opened an issue here #14828

bishabosha

Great work! thank you

tanishiking added 3 commits March 2, 2022 21:03

Add generated semanticdb source code

7817912

Generated from tanishiking/semanticdb-for-scala3#1 See the discussion here scalameta/scalameta#2414

Support MatchType in semanticdb

0480dec

Remove LambdaType

fa4cb01

tanishiking marked this pull request as ready for review March 2, 2022 13:29

bishabosha reviewed Mar 2, 2022

View reviewed changes

tests/semanticdb/metac.expect Outdated Show resolved Hide resolved

bishabosha assigned tanishiking Mar 3, 2022

bishabosha added the area:semanticdb label Mar 3, 2022

Assign types to type variables in match case types

28d59a0

tanishiking commented Mar 4, 2022

View reviewed changes

tanishiking requested a review from bishabosha March 4, 2022 13:07

bishabosha reviewed Mar 16, 2022

View reviewed changes

tests/semanticdb/metac.expect Outdated Show resolved Hide resolved

Remove case property from the symbol of type variable

2ca6fb2

`t` of `case List[t] =>` has `CaseClass` flag. Maybe we should remove `CaseClass` flag from `t`, but as a workaround remove `Case` property from semanticdb from type variables.

bishabosha approved these changes Mar 17, 2022

View reviewed changes

bishabosha enabled auto-merge March 17, 2022 09:51

tanishiking mentioned this pull request Mar 17, 2022

Support MatchType tanishiking/semanticdb-for-scala3#50

Merged

tanishiking disabled auto-merge March 17, 2022 10:17

tanishiking mentioned this pull request Mar 17, 2022

Support MatchType for Scala3 scalameta/scalameta#2681

Merged

bishabosha added the stat:do not merge label Mar 17, 2022

Add undefined type case for TypeRef with no symbols

18b8901

bishabosha reviewed Mar 22, 2022

View reviewed changes

Retrieve a symbol for typeref to refinements in RefinedType

b070667

scala#14608 (comment)

tanishiking commented Mar 22, 2022

View reviewed changes

compiler/src/dotty/tools/dotc/semanticdb/TypeOps.scala Outdated Show resolved Hide resolved

tanishiking commented Mar 22, 2022

View reviewed changes

compiler/src/dotty/tools/dotc/semanticdb/TypeOps.scala Outdated Show resolved Hide resolved

tanishiking commented Mar 22, 2022

View reviewed changes

tests/semanticdb/expect/dep-match.scala Show resolved Hide resolved

Resolve symbols in refinements by fake symbols

60c336e

bishabosha reviewed Mar 30, 2022

View reviewed changes

tests/semanticdb/expect/Advanced.expect.scala Outdated Show resolved Hide resolved

Resolve symbol occurrences of select type fields

be31d56

tanishiking force-pushed the typelambda-matchtype-2 branch from b964694 to be31d56 Compare March 31, 2022 11:56

tanishiking added 2 commits March 31, 2022 22:21

Resolve real symbol from nested refined type

9375ab1

No need to resolve fake symbols from RefinedTypes

50a43b9

tanishiking added 2 commits April 1, 2022 17:44

Dealias type for looking up symbols in refinements

8465366

tanishiking mentioned this pull request Apr 2, 2022

[SemanticDB] Missing symbol occurrence on refinement access #14828

Open

bishabosha approved these changes Apr 3, 2022

View reviewed changes

bishabosha merged commit a26a2c3 into scala:main Apr 3, 2022

tanishiking deleted the typelambda-matchtype-2 branch April 4, 2022 06:56

tanishiking mentioned this pull request Apr 5, 2022

Support MatchType in SemanticdbTreePrinter scalameta/metals#3787

Merged

Kordyjan added this to the 3.1.3 milestone Aug 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support MatchType for SemanticDB #14608

Support MatchType for SemanticDB #14608

tanishiking commented Mar 2, 2022

bishabosha commented Mar 2, 2022 •

edited

tanishiking commented Mar 3, 2022

bishabosha commented Mar 3, 2022

tgodzik commented Mar 3, 2022

bishabosha commented Mar 3, 2022 •

edited

tgodzik commented Mar 3, 2022

tanishiking Mar 4, 2022

tanishiking Mar 4, 2022

tanishiking commented Mar 4, 2022

tanishiking commented Mar 16, 2022

bishabosha left a comment

tanishiking commented Mar 17, 2022 •

edited

bishabosha commented Mar 17, 2022 •

edited

tanishiking commented Mar 17, 2022

bishabosha Mar 22, 2022 •

edited

tanishiking commented Mar 22, 2022 •

edited

bishabosha commented Mar 22, 2022 •

edited

bishabosha commented Mar 22, 2022

tanishiking commented Mar 22, 2022

bishabosha Mar 30, 2022

tanishiking Mar 30, 2022

bishabosha Mar 30, 2022 •

edited

tanishiking Mar 30, 2022

bishabosha Mar 31, 2022

tanishiking commented Mar 31, 2022

bishabosha commented Mar 31, 2022

tanishiking commented Apr 1, 2022

bishabosha commented Apr 1, 2022 •

edited

tanishiking commented Apr 2, 2022

bishabosha left a comment

Support MatchType for SemanticDB #14608

Support MatchType for SemanticDB #14608

Conversation

tanishiking commented Mar 2, 2022

bishabosha commented Mar 2, 2022 • edited

tanishiking commented Mar 3, 2022

bishabosha commented Mar 3, 2022

tgodzik commented Mar 3, 2022

bishabosha commented Mar 3, 2022 • edited

tgodzik commented Mar 3, 2022

tanishiking Mar 4, 2022

Choose a reason for hiding this comment

tanishiking Mar 4, 2022

Choose a reason for hiding this comment

tanishiking commented Mar 4, 2022

tanishiking commented Mar 16, 2022

bishabosha left a comment

Choose a reason for hiding this comment

tanishiking commented Mar 17, 2022 • edited

bishabosha commented Mar 17, 2022 • edited

tanishiking commented Mar 17, 2022

bishabosha Mar 22, 2022 • edited

Choose a reason for hiding this comment

tanishiking commented Mar 22, 2022 • edited

bishabosha commented Mar 22, 2022 • edited

bishabosha commented Mar 22, 2022

tanishiking commented Mar 22, 2022

bishabosha Mar 30, 2022

Choose a reason for hiding this comment

tanishiking Mar 30, 2022

Choose a reason for hiding this comment

bishabosha Mar 30, 2022 • edited

Choose a reason for hiding this comment

tanishiking Mar 30, 2022

Choose a reason for hiding this comment

bishabosha Mar 31, 2022

Choose a reason for hiding this comment

tanishiking commented Mar 31, 2022

bishabosha commented Mar 31, 2022

tanishiking commented Apr 1, 2022

bishabosha commented Apr 1, 2022 • edited

tanishiking commented Apr 2, 2022

bishabosha left a comment

Choose a reason for hiding this comment

bishabosha commented Mar 2, 2022 •

edited

bishabosha commented Mar 3, 2022 •

edited

tanishiking commented Mar 17, 2022 •

edited

bishabosha commented Mar 17, 2022 •

edited

bishabosha Mar 22, 2022 •

edited

tanishiking commented Mar 22, 2022 •

edited

bishabosha commented Mar 22, 2022 •

edited

bishabosha Mar 30, 2022 •

edited

bishabosha commented Apr 1, 2022 •

edited