Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiler support for scala-async, with a state machine transform phase running after erasure [ci: last-only] #8816

Merged
merged 94 commits into from Jun 18, 2020

Commits on Jun 10, 2020

  1. make xprompt work in sbt's scalac task

    adriaanm authored and retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    9a0cbe8 View commit details
    Browse the repository at this point in the history
  2. Import async

    adriaanm authored and retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    3d09049 View commit details
    Browse the repository at this point in the history
  3. patch for async debugging

    adriaanm authored and retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    9535c94 View commit details
    Browse the repository at this point in the history
  4. Implement async transform with a macro and a post-erasure phase

    The macro wraps the (untransformed) expression in the skeleton of the
    state machine class. Compiler phases between this point and the late
    async transform will correctly add outer pointers, lift captured vars
    into Ref objects etc. The state machine class itself will also undergo
    any transforms that it needs, such as specialization.
    
    The macro is wired up using the FastTrack facility in line with other
    macros that are quasi-intrinsic (ie, implemented in the scala-compiler.jar).
    
    After post-erasure, we run the ANF and state machine transform.
    
    The old code used the macro refletion API -- these usages are
    now being replaced with direct use of the richer Global API.
    adriaanm authored and retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    6a665e9 View commit details
    Browse the repository at this point in the history
  5. Improvements to async phase

    Passes the correct owner symbol to `useFields` to fix a bug
    with local modules.
    
    Refactors to store state about the current async block in a var
    which allows us to mix in all the implementation once into
    AsyncPhase.
    
    Remove some testing FutureSystems. We'll refactor the tests
    to use ScalaConcurrentFutureSystem.
    
    Allow a custom front end to supply an alternative FutureSystem
    attached to the block defining the state machine wrapper class.
    Demonstrate this with a test case for an annotation driven
    front end that targets an alternative Future implementation.
    
    Remove some now-unused customization points from FutureSystem.
    For example, we don't need to abstract over the choice of
    base class for the state machine as a custom front end now
    creates the state machine tree directly.
    
    FutureSystem now deals directly in Trees, rather than in
    `c.Expr`. I've kept a light weight option (a type alias
    `type Expr[T] = Tree`) so we can have a little compiler support
    for getting the types right and documentation for
    `FutureSystem` implementors.
    
    Refactor the ANF transform (and others) to directly be a
    TypingTransformer.
    
    Remove the need to subclass Function0 in the state machine
    by kicking things off with `Future.unit.onComplete(stateMachine)`.
    This reduces bloat in the generated code due to Function0
    specialization.
    
    I've also elminated the macro application in the early transform
    in favour of wrapping the argument in `Block(<arg> :: Nil, q"()"}`.
    This ran into some "pure expression has no effect" warnings which
    I've suppressed with a new tree annotation.
    
    Most of the `auto_` test cases are converted to use the
    standard scala.concurrent version of async. Now that the
    transform is always happening post-erasure, there is no need to
    keep those tests under the annotation driven async test (which
    used to be the only way to get post-typer expansion).
    
    A few test can only be expressed with the annotation driven
    version of async, namely those that have an async boundary
    in an extractor. These have been moved into the
    `AnnotationDrivenAsyncTest`.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    a7bbec5 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    008137d View commit details
    Browse the repository at this point in the history
  7. Add JUnit support to partest

    szeiger authored and retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    5dd9495 View commit details
    Browse the repository at this point in the history
  8. Add tests from scala-async

      - most regular scala.async.Async tests pass
      - AsyncId converted to ScalaConcurrentAsync
      - non-standard tests not properly integrated yet
      - actual test failure in `run/futures.scala`
      - neg/NakedAwait still missing (alternative first)
      - remove late.scala (converted to AnnotationDrivenAsync)
      - Remove uncheckedBounds.scala (the problem no longer exists post-erasure)
    szeiger authored and retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    08fa083 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    3b38d95 View commit details
    Browse the repository at this point in the history
  10. Delete dead code.

    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    b61e65b View commit details
    Browse the repository at this point in the history
  11. Convert tests to ScalaConcurrent Future System

    Also:
      - remove late.scala (converted to AnnotationDrivenAsync)
      - Remove uncheckedBounds.scala (the problem no longer exists post-erasure)
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    5a9517b View commit details
    Browse the repository at this point in the history
  12. Fix nested async transforms

    UseFields needs to do what explicit outer would have done:
    access inner classes must access fields via the outer
    parameter and we must ensure fields accessed from inners
    are not JVM private.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    50150ca View commit details
    Browse the repository at this point in the history
  13. We'll never see isAsync again in nested trees.

    This code path was only relevant when async was run as a macro
    in the presentation compiler.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    8de6b1b View commit details
    Browse the repository at this point in the history
  14. Deal with await in ArrayValue

    As a late-running ANF transform must do.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    9666175 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    e82ed14 View commit details
    Browse the repository at this point in the history
  16. Companion Detection should be post-erasure aware

    Deal with the representation of local modules at async's
    new place in the compiler pipeline.
    
    Use more efficient/idiomatic collection of local def trees.
    (`Tree#children` creates temporary lists.)
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    0619d4e View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    2b580ed View commit details
    Browse the repository at this point in the history
  18. A simpler and faster ANF transform

    A long overdue overhaul.
    
    Originally, we defined this in terms of a pair of mutually recursive
    transforms (`anf` and `linearize`). We can simplfy the implementation
    by collapsing these the cases of single Transformer.
    
    This transformer is designed for "thicket" based transforms, where
    a single input tree can map to a list of output trees which will
    be flattened into the enclosing `Block`. An enclosing block will be
    automatically added if there was none before.
    
    Make the transform more idiomatic by more widespread use of
    utilities to generated attributed trees rather than passing
    untyped trees through `localTyper.typed`.
    
    Rework special casing of Unit/Nothing typed expressions to
    avoid temporary vals of these types to avoid littering the
    resulting trees with `BoxedUnit` references.
    
    Avoid unnecessary temporaries in:
      - the suffix of the args of an Apply the follows the final `await`
      - the result of If that only awaits the condition
      - similarly, the result of a Match that only awaits the scrutinee
      - "safe to inline" expressions like simple Idents or constants.
    
    Remove special cases for derived value classes which are
    eliminated now before the ANF transform.
    
    Furthermore:
    
      - Deal with ArrayValue tree
      - Remove cast which was used ot suppress typer warning
      - Don't need to deal with nested Applys after uncurry
      - Remove an new-unneeded adjustment from the entry point to the ANF
        and from MatchResultTransformer.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    ec81663 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    b1c43ce View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    d0ac056 View commit details
    Browse the repository at this point in the history
  21. Sort cases by state ID

    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    ee66301 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    c7cb301 View commit details
    Browse the repository at this point in the history
  23. Use reporter.error directly

    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    210bfac View commit details
    Browse the repository at this point in the history
  24. Cleanup AsyncPhase's transformer

    Transform the ClassDef and DefDef in separate cases.
    This reduces some of manual atOwner and treeCopy busywork.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    90568be View commit details
    Browse the repository at this point in the history
  25. Harden Context.make debug logging against null trees.

    TypingTransformers can easily feed null trees into here
    if they don't assign `curTree`.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    2d20aeb View commit details
    Browse the repository at this point in the history
  26. Integrate async's logging into debuglog

    The ANF transform trace is demoted to the
    "edit a boolean and re-compile" UI.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    e276921 View commit details
    Browse the repository at this point in the history
  27. Refactor and document UseFields transform

    Also, lifted lazy var fields need not be MUTABLE.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    588cab4 View commit details
    Browse the repository at this point in the history
  28. Avoid overhead in tree attachments

    Deal with the underlying Set directly rather than
    using the ClassTag indexed lookup/removal.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    528ffd2 View commit details
    Browse the repository at this point in the history
  29. Reduce number of states in async state machine

    We used to lift the control flow for any pattern match or
    if/else into fine grained states if it enclosed even a single
    await call.
    
    Instead, we now keep as much of the control flow as possible
    in the current state. The program point after the control flow
    statement is lifted into a new state iff it is reachable from
    any branch that contains an await.
    
    LabelDef-s representing cases and the matchEnd of patterns are now
    only lifted into states if they are the target of a label jump
    from a resumption handler.
    
    To make this analysis possible, we form a context chain of
    AsyncBlockBuilders to let the translation of the label jumps compare
    the current state to that current state of the enclosing block builder
    that holds the LabelDefs. If these differ, we can conclude that we
    have crossed an async boundary and mark the LabelDef for lifting
    into state.
    
    Here's how the simplification helps:
    
    https://gist.github.com/retronym/50839a051db9b6d1fcd0719d99ecbfc2
    
    Refactorings along the way:
    
      - introduce a ThicketTransformer to transform `Tree => List[Tree]`
      - make `symLookup` more convenient to access
      - add some internal flags to disable state compaction and field
        nulling which help to debug the phase
      - unify onto a single `AsyncState` class. Different use cases
        are served by eagerly adding additional logic, rather than
        having this added later on in the variants of
        `mkCompletionHandler`
      - detect and remove empty states in `compactStates`. It seems easier
        now to do this than to avoid constructing them.
      - Try a bit harder to avoid redundant `()` in stats.
      - Move synthesis of async block generation into
        StateBuilder.build (formrly known as resultSimple). This is
        now the only factory method on StateBuilder.
      - Fix propagation of the enclosing subsequent state into
        nested expression positions
      - In combination, this lets us perform async block completion
        inline in all terminal states, there is no need to write the
        result to lifted field and make a state transition.
      - Remove the `currTree` var in favour of delegating to
        the field in the `stateBuilder` var
      - Encapsulate state transition bookkeeping
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    b617575 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    7fdeed3 View commit details
    Browse the repository at this point in the history
  31. Reinstate live variables test and fix null assignment

    While looking more closely at the implementation, I
    realised our approach of nulling dead variables at the
    start of the subsequent state was wrong, we retained
    the references for longer than needed.
    
    I've moved the nulling into the conclusion of the
    state of last usages.
    
    Remove the warnings test which isn't relevant with post-typer ANF.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    1db7f9d View commit details
    Browse the repository at this point in the history
  32. Use compiler APIs.

    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    32e15bb View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    c62fb75 View commit details
    Browse the repository at this point in the history
  34. Remove PhasedTransform

    We don't have any phase-dependent logic left.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    ffc613b View commit details
    Browse the repository at this point in the history
  35. Configuration menu
    Copy the full SHA
    789811a View commit details
    Browse the repository at this point in the history
  36. Remove unused imports

    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    b86f268 View commit details
    Browse the repository at this point in the history
  37. Allow a nested TypingTransformer to use an existing local typer.

    This saves allocations for root imports etc.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    2f75120 View commit details
    Browse the repository at this point in the history
  38. Merge AsyncTransformState with SymLookup

    Also:
     - remove some redundant or simply unused parameters.
     - tighten access on some members
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    4753b39 View commit details
    Browse the repository at this point in the history
  39. Add a fast path to the async phase

    If no methods have been marked for tranform, disable
    the phase for the entire unit.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    bcf25d7 View commit details
    Browse the repository at this point in the history
  40. Configuration menu
    Copy the full SHA
    f512e2f View commit details
    Browse the repository at this point in the history
  41. Refactor lifting to to change owner sooner

    And reuse existing utility method to change module class owner.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    dd0001b View commit details
    Browse the repository at this point in the history
  42. Refactor field nulling.

    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    ad00a91 View commit details
    Browse the repository at this point in the history
  43. Remove unused imports.

    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    60db8f9 View commit details
    Browse the repository at this point in the history
  44. Remove dead cases from adaptToUnit

    The input trees are always typed now which means the early cases
    subsume the ones being deleted here.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    5755c74 View commit details
    Browse the repository at this point in the history
  45. Configuration menu
    Copy the full SHA
    ef04e59 View commit details
    Browse the repository at this point in the history
  46. Configuration menu
    Copy the full SHA
    6f1546c View commit details
    Browse the repository at this point in the history
  47. Refactor asyncTransform

    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    93748bd View commit details
    Browse the repository at this point in the history
  48. Configuration menu
    Copy the full SHA
    22014fc View commit details
    Browse the repository at this point in the history
  49. Remove dead code.

    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    0cea808 View commit details
    Browse the repository at this point in the history
  50. Configuration menu
    Copy the full SHA
    02bbc1d View commit details
    Browse the repository at this point in the history
  51. Configuration menu
    Copy the full SHA
    a3aaeda View commit details
    Browse the repository at this point in the history
  52. Configuration menu
    Copy the full SHA
    1285669 View commit details
    Browse the repository at this point in the history
  53. Configuration menu
    Copy the full SHA
    4124c25 View commit details
    Browse the repository at this point in the history
  54. Configuration menu
    Copy the full SHA
    aabe723 View commit details
    Browse the repository at this point in the history
  55. Configuration menu
    Copy the full SHA
    01a83cb View commit details
    Browse the repository at this point in the history
  56. Configuration menu
    Copy the full SHA
    c89c21b View commit details
    Browse the repository at this point in the history
  57. Inline case class Awaitable

    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    33ca19f View commit details
    Browse the repository at this point in the history
  58. Configuration menu
    Copy the full SHA
    c5bc867 View commit details
    Browse the repository at this point in the history
  59. Configuration menu
    Copy the full SHA
    2631d47 View commit details
    Browse the repository at this point in the history
  60. Overhaul integration with front end macros

      - The state machine generated by the front end macro must Now
        implement a small set of methods as a facade over the
        particular Future/Awaitable/Task type being used. This
        replaces the AST factory methods in `FutureSystem`
      - Remove intrinsic implemnentation for scala-async's async
        macro. Show what it will need to do in a test implementation,
        `s.t.n.partest.Async` and update all test cases to use this.
      - Add a heuristic to interpret `@compileTimeOnly` annotated
        methods to register `await` methods. Defer the check in refchecks
        for all of these and instead check that no references survive
        the async phase.
      - Refactor the test implementations of async front ends to
        share an interface for the state machine. We don't ship
        this but it could be copy/pasted into third party integrations.
      - Add a test integration with Java's CompletableFuture.
      - Expose a method through macro `Internals` API to let
        front ends mark a method in the state machine for async
        translation.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    11733cb View commit details
    Browse the repository at this point in the history
  61. Configuration menu
    Copy the full SHA
    791c015 View commit details
    Browse the repository at this point in the history
  62. Configuration menu
    Copy the full SHA
    36ad222 View commit details
    Browse the repository at this point in the history
  63. Configuration menu
    Copy the full SHA
    d7a2764 View commit details
    Browse the repository at this point in the history
  64. Allow compilation of async's partest suite in Junit

    This lets us use IntelliJ's code coverage analysis.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    36d452e View commit details
    Browse the repository at this point in the history
  65. Configuration menu
    Copy the full SHA
    dd463d1 View commit details
    Browse the repository at this point in the history
  66. Fix detection of ill-nested awaits

     - Adapt detection logic to the post-erasure tree shapes
     - Lift restriction about || and && in favour of a
       rewrite to `If` in the ANF transform
     - import relevant parts of the test from scala-async.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    de4fc23 View commit details
    Browse the repository at this point in the history
  67. Move ThicketTransformer to s.t.n.transform

    It seems generally useful enough to move it out of
    the async utilities. I've also added a unit test.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    7d46e6b View commit details
    Browse the repository at this point in the history
  68. Configuration menu
    Copy the full SHA
    469cfc1 View commit details
    Browse the repository at this point in the history
  69. Configuration menu
    Copy the full SHA
    074b480 View commit details
    Browse the repository at this point in the history
  70. Configuration menu
    Copy the full SHA
    3a2882f View commit details
    Browse the repository at this point in the history
  71. More granular fresh names.

    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    b68e118 View commit details
    Browse the repository at this point in the history
  72. Configuration menu
    Copy the full SHA
    0775a0d View commit details
    Browse the repository at this point in the history
  73. Configuration menu
    Copy the full SHA
    d1d3daa View commit details
    Browse the repository at this point in the history
  74. Configuration menu
    Copy the full SHA
    8c770f2 View commit details
    Browse the repository at this point in the history
  75. Improve error reporting in asynj junit test.

    Show source file context for reported compiler
    errors and warnings, as we would expect from
    the normal ConsoleReporter.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    4d0f6bb View commit details
    Browse the repository at this point in the history
  76. Support state getter/setter methods

    Rather than assuming its a var.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    a5298c5 View commit details
    Browse the repository at this point in the history
  77. Better integration with compiler fresh names

     - No longer add the `$async` suffix to fresh names. Add a test to
       show that we no longer risk clashing with lambda lifted names
       because we're coordinating through the unit's FreshNameCreator.
     - Add a facility to FreshNameCreator to support clients that
       want to generate many fresh names from a single prefix
       and avoid the overhead of going through `newTermName`
       when subsequent units also want to generate `x$0`, etc.
     - Add a little efficiency to name concatenation by use of
       presized of string builders.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    499296c View commit details
    Browse the repository at this point in the history
  78. Configuration menu
    Copy the full SHA
    e73f502 View commit details
    Browse the repository at this point in the history
  79. Overhaul/fix live variables

    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    bcb5dc6 View commit details
    Browse the repository at this point in the history
  80. Configuration menu
    Copy the full SHA
    d81ef53 View commit details
    Browse the repository at this point in the history
  81. Configuration menu
    Copy the full SHA
    502c127 View commit details
    Browse the repository at this point in the history
  82. Move async run tests to jvm category

    Some will be made Scala.js compatible in a subsequent commit and
    moved back to run.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    e63a133 View commit details
    Browse the repository at this point in the history
  83. Configuration menu
    Copy the full SHA
    f9d7952 View commit details
    Browse the repository at this point in the history
  84. Configuration menu
    Copy the full SHA
    215f173 View commit details
    Browse the repository at this point in the history
  85. Tweak names in generated code

     - Remove '_' and line number from state machine classs names.
     - Rename `state$async` to `state`.
    retronym committed Jun 10, 2020
    Configuration menu
    Copy the full SHA
    5455445 View commit details
    Browse the repository at this point in the history
  86. Configuration menu
    Copy the full SHA
    db3ea69 View commit details
    Browse the repository at this point in the history
  87. Configuration menu
    Copy the full SHA
    923e57b View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2020

  1. Configuration menu
    Copy the full SHA
    b76a0ec View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2020

  1. Configuration menu
    Copy the full SHA
    c94951e View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2020

  1. Rework async expression wrapping

    Wrap the async expression in `locally { ... }` to
    force value class boxing. This seems to work better
    than `{ ... }: Any`, which ends up with `Function`
    trees typed with `Any` which violoates an assumption
    in `delambdafy`.
    retronym committed Jun 15, 2020
    Configuration menu
    Copy the full SHA
    7f22119 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    230684d View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2020

  1. remove old ThicketTransformer

    lrytz committed Jun 16, 2020
    Configuration menu
    Copy the full SHA
    0e27334 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2020

  1. Configuration menu
    Copy the full SHA
    ae91a3c View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2020

  1. Refactor detection of ill nested await calls

    We can defer the check until the current location of the "fallback"
    check without sacrificing the specific error messages.
    retronym committed Jun 18, 2020
    Configuration menu
    Copy the full SHA
    89f48d2 View commit details
    Browse the repository at this point in the history