New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compiler support for scala-async, with a state machine transform phase running after erasure [ci: last-only] #8816
Compiler support for scala-async, with a state machine transform phase running after erasure [ci: last-only] #8816
Commits on Jun 10, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 9a0cbe8 - Browse repository at this point
Copy the full SHA 9a0cbe8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3d09049 - Browse repository at this point
Copy the full SHA 3d09049View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9535c94 - Browse repository at this point
Copy the full SHA 9535c94View commit details -
Implement async transform with a macro and a post-erasure phase
The macro wraps the (untransformed) expression in the skeleton of the state machine class. Compiler phases between this point and the late async transform will correctly add outer pointers, lift captured vars into Ref objects etc. The state machine class itself will also undergo any transforms that it needs, such as specialization. The macro is wired up using the FastTrack facility in line with other macros that are quasi-intrinsic (ie, implemented in the scala-compiler.jar). After post-erasure, we run the ANF and state machine transform. The old code used the macro refletion API -- these usages are now being replaced with direct use of the richer Global API.
Configuration menu - View commit details
-
Copy full SHA for 6a665e9 - Browse repository at this point
Copy the full SHA 6a665e9View commit details -
Passes the correct owner symbol to `useFields` to fix a bug with local modules. Refactors to store state about the current async block in a var which allows us to mix in all the implementation once into AsyncPhase. Remove some testing FutureSystems. We'll refactor the tests to use ScalaConcurrentFutureSystem. Allow a custom front end to supply an alternative FutureSystem attached to the block defining the state machine wrapper class. Demonstrate this with a test case for an annotation driven front end that targets an alternative Future implementation. Remove some now-unused customization points from FutureSystem. For example, we don't need to abstract over the choice of base class for the state machine as a custom front end now creates the state machine tree directly. FutureSystem now deals directly in Trees, rather than in `c.Expr`. I've kept a light weight option (a type alias `type Expr[T] = Tree`) so we can have a little compiler support for getting the types right and documentation for `FutureSystem` implementors. Refactor the ANF transform (and others) to directly be a TypingTransformer. Remove the need to subclass Function0 in the state machine by kicking things off with `Future.unit.onComplete(stateMachine)`. This reduces bloat in the generated code due to Function0 specialization. I've also elminated the macro application in the early transform in favour of wrapping the argument in `Block(<arg> :: Nil, q"()"}`. This ran into some "pure expression has no effect" warnings which I've suppressed with a new tree annotation. Most of the `auto_` test cases are converted to use the standard scala.concurrent version of async. Now that the transform is always happening post-erasure, there is no need to keep those tests under the annotation driven async test (which used to be the only way to get post-typer expansion). A few test can only be expressed with the annotation driven version of async, namely those that have an async boundary in an extractor. These have been moved into the `AnnotationDrivenAsyncTest`.
Configuration menu - View commit details
-
Copy full SHA for a7bbec5 - Browse repository at this point
Copy the full SHA a7bbec5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 008137d - Browse repository at this point
Copy the full SHA 008137dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5dd9495 - Browse repository at this point
Copy the full SHA 5dd9495View commit details -
- most regular scala.async.Async tests pass - AsyncId converted to ScalaConcurrentAsync - non-standard tests not properly integrated yet - actual test failure in `run/futures.scala` - neg/NakedAwait still missing (alternative first) - remove late.scala (converted to AnnotationDrivenAsync) - Remove uncheckedBounds.scala (the problem no longer exists post-erasure)
Configuration menu - View commit details
-
Copy full SHA for 08fa083 - Browse repository at this point
Copy the full SHA 08fa083View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3b38d95 - Browse repository at this point
Copy the full SHA 3b38d95View commit details -
Configuration menu - View commit details
-
Copy full SHA for b61e65b - Browse repository at this point
Copy the full SHA b61e65bView commit details -
Convert tests to ScalaConcurrent Future System
Also: - remove late.scala (converted to AnnotationDrivenAsync) - Remove uncheckedBounds.scala (the problem no longer exists post-erasure)
Configuration menu - View commit details
-
Copy full SHA for 5a9517b - Browse repository at this point
Copy the full SHA 5a9517bView commit details -
UseFields needs to do what explicit outer would have done: access inner classes must access fields via the outer parameter and we must ensure fields accessed from inners are not JVM private.
Configuration menu - View commit details
-
Copy full SHA for 50150ca - Browse repository at this point
Copy the full SHA 50150caView commit details -
We'll never see
isAsync
again in nested trees.This code path was only relevant when async was run as a macro in the presentation compiler.
Configuration menu - View commit details
-
Copy full SHA for 8de6b1b - Browse repository at this point
Copy the full SHA 8de6b1bView commit details -
As a late-running ANF transform must do.
Configuration menu - View commit details
-
Copy full SHA for 9666175 - Browse repository at this point
Copy the full SHA 9666175View commit details -
Configuration menu - View commit details
-
Copy full SHA for e82ed14 - Browse repository at this point
Copy the full SHA e82ed14View commit details -
Companion Detection should be post-erasure aware
Deal with the representation of local modules at async's new place in the compiler pipeline. Use more efficient/idiomatic collection of local def trees. (`Tree#children` creates temporary lists.)
Configuration menu - View commit details
-
Copy full SHA for 0619d4e - Browse repository at this point
Copy the full SHA 0619d4eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2b580ed - Browse repository at this point
Copy the full SHA 2b580edView commit details -
A simpler and faster ANF transform
A long overdue overhaul. Originally, we defined this in terms of a pair of mutually recursive transforms (`anf` and `linearize`). We can simplfy the implementation by collapsing these the cases of single Transformer. This transformer is designed for "thicket" based transforms, where a single input tree can map to a list of output trees which will be flattened into the enclosing `Block`. An enclosing block will be automatically added if there was none before. Make the transform more idiomatic by more widespread use of utilities to generated attributed trees rather than passing untyped trees through `localTyper.typed`. Rework special casing of Unit/Nothing typed expressions to avoid temporary vals of these types to avoid littering the resulting trees with `BoxedUnit` references. Avoid unnecessary temporaries in: - the suffix of the args of an Apply the follows the final `await` - the result of If that only awaits the condition - similarly, the result of a Match that only awaits the scrutinee - "safe to inline" expressions like simple Idents or constants. Remove special cases for derived value classes which are eliminated now before the ANF transform. Furthermore: - Deal with ArrayValue tree - Remove cast which was used ot suppress typer warning - Don't need to deal with nested Applys after uncurry - Remove an new-unneeded adjustment from the entry point to the ANF and from MatchResultTransformer.
Configuration menu - View commit details
-
Copy full SHA for ec81663 - Browse repository at this point
Copy the full SHA ec81663View commit details -
Configuration menu - View commit details
-
Copy full SHA for b1c43ce - Browse repository at this point
Copy the full SHA b1c43ceView commit details -
Configuration menu - View commit details
-
Copy full SHA for d0ac056 - Browse repository at this point
Copy the full SHA d0ac056View commit details -
Configuration menu - View commit details
-
Copy full SHA for ee66301 - Browse repository at this point
Copy the full SHA ee66301View commit details -
Configuration menu - View commit details
-
Copy full SHA for c7cb301 - Browse repository at this point
Copy the full SHA c7cb301View commit details -
Configuration menu - View commit details
-
Copy full SHA for 210bfac - Browse repository at this point
Copy the full SHA 210bfacView commit details -
Cleanup AsyncPhase's transformer
Transform the ClassDef and DefDef in separate cases. This reduces some of manual atOwner and treeCopy busywork.
Configuration menu - View commit details
-
Copy full SHA for 90568be - Browse repository at this point
Copy the full SHA 90568beView commit details -
Harden Context.make debug logging against null trees.
TypingTransformers can easily feed null trees into here if they don't assign `curTree`.
Configuration menu - View commit details
-
Copy full SHA for 2d20aeb - Browse repository at this point
Copy the full SHA 2d20aebView commit details -
Integrate async's logging into debuglog
The ANF transform trace is demoted to the "edit a boolean and re-compile" UI.
Configuration menu - View commit details
-
Copy full SHA for e276921 - Browse repository at this point
Copy the full SHA e276921View commit details -
Refactor and document UseFields transform
Also, lifted lazy var fields need not be MUTABLE.
Configuration menu - View commit details
-
Copy full SHA for 588cab4 - Browse repository at this point
Copy the full SHA 588cab4View commit details -
Avoid overhead in tree attachments
Deal with the underlying Set directly rather than using the ClassTag indexed lookup/removal.
Configuration menu - View commit details
-
Copy full SHA for 528ffd2 - Browse repository at this point
Copy the full SHA 528ffd2View commit details -
Reduce number of states in async state machine
We used to lift the control flow for any pattern match or if/else into fine grained states if it enclosed even a single await call. Instead, we now keep as much of the control flow as possible in the current state. The program point after the control flow statement is lifted into a new state iff it is reachable from any branch that contains an await. LabelDef-s representing cases and the matchEnd of patterns are now only lifted into states if they are the target of a label jump from a resumption handler. To make this analysis possible, we form a context chain of AsyncBlockBuilders to let the translation of the label jumps compare the current state to that current state of the enclosing block builder that holds the LabelDefs. If these differ, we can conclude that we have crossed an async boundary and mark the LabelDef for lifting into state. Here's how the simplification helps: https://gist.github.com/retronym/50839a051db9b6d1fcd0719d99ecbfc2 Refactorings along the way: - introduce a ThicketTransformer to transform `Tree => List[Tree]` - make `symLookup` more convenient to access - add some internal flags to disable state compaction and field nulling which help to debug the phase - unify onto a single `AsyncState` class. Different use cases are served by eagerly adding additional logic, rather than having this added later on in the variants of `mkCompletionHandler` - detect and remove empty states in `compactStates`. It seems easier now to do this than to avoid constructing them. - Try a bit harder to avoid redundant `()` in stats. - Move synthesis of async block generation into StateBuilder.build (formrly known as resultSimple). This is now the only factory method on StateBuilder. - Fix propagation of the enclosing subsequent state into nested expression positions - In combination, this lets us perform async block completion inline in all terminal states, there is no need to write the result to lifted field and make a state transition. - Remove the `currTree` var in favour of delegating to the field in the `stateBuilder` var - Encapsulate state transition bookkeeping
Configuration menu - View commit details
-
Copy full SHA for b617575 - Browse repository at this point
Copy the full SHA b617575View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7fdeed3 - Browse repository at this point
Copy the full SHA 7fdeed3View commit details -
Reinstate live variables test and fix null assignment
While looking more closely at the implementation, I realised our approach of nulling dead variables at the start of the subsequent state was wrong, we retained the references for longer than needed. I've moved the nulling into the conclusion of the state of last usages. Remove the warnings test which isn't relevant with post-typer ANF.
Configuration menu - View commit details
-
Copy full SHA for 1db7f9d - Browse repository at this point
Copy the full SHA 1db7f9dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 32e15bb - Browse repository at this point
Copy the full SHA 32e15bbView commit details -
Configuration menu - View commit details
-
Copy full SHA for c62fb75 - Browse repository at this point
Copy the full SHA c62fb75View commit details -
We don't have any phase-dependent logic left.
Configuration menu - View commit details
-
Copy full SHA for ffc613b - Browse repository at this point
Copy the full SHA ffc613bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 789811a - Browse repository at this point
Copy the full SHA 789811aView commit details -
Configuration menu - View commit details
-
Copy full SHA for b86f268 - Browse repository at this point
Copy the full SHA b86f268View commit details -
Allow a nested TypingTransformer to use an existing local typer.
This saves allocations for root imports etc.
Configuration menu - View commit details
-
Copy full SHA for 2f75120 - Browse repository at this point
Copy the full SHA 2f75120View commit details -
Merge AsyncTransformState with SymLookup
Also: - remove some redundant or simply unused parameters. - tighten access on some members
Configuration menu - View commit details
-
Copy full SHA for 4753b39 - Browse repository at this point
Copy the full SHA 4753b39View commit details -
Add a fast path to the async phase
If no methods have been marked for tranform, disable the phase for the entire unit.
Configuration menu - View commit details
-
Copy full SHA for bcf25d7 - Browse repository at this point
Copy the full SHA bcf25d7View commit details -
Configuration menu - View commit details
-
Copy full SHA for f512e2f - Browse repository at this point
Copy the full SHA f512e2fView commit details -
Refactor lifting to to change owner sooner
And reuse existing utility method to change module class owner.
Configuration menu - View commit details
-
Copy full SHA for dd0001b - Browse repository at this point
Copy the full SHA dd0001bView commit details -
Configuration menu - View commit details
-
Copy full SHA for ad00a91 - Browse repository at this point
Copy the full SHA ad00a91View commit details -
Configuration menu - View commit details
-
Copy full SHA for 60db8f9 - Browse repository at this point
Copy the full SHA 60db8f9View commit details -
Remove dead cases from adaptToUnit
The input trees are always typed now which means the early cases subsume the ones being deleted here.
Configuration menu - View commit details
-
Copy full SHA for 5755c74 - Browse repository at this point
Copy the full SHA 5755c74View commit details -
Configuration menu - View commit details
-
Copy full SHA for ef04e59 - Browse repository at this point
Copy the full SHA ef04e59View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6f1546c - Browse repository at this point
Copy the full SHA 6f1546cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 93748bd - Browse repository at this point
Copy the full SHA 93748bdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 22014fc - Browse repository at this point
Copy the full SHA 22014fcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0cea808 - Browse repository at this point
Copy the full SHA 0cea808View commit details -
Configuration menu - View commit details
-
Copy full SHA for 02bbc1d - Browse repository at this point
Copy the full SHA 02bbc1dView commit details -
Configuration menu - View commit details
-
Copy full SHA for a3aaeda - Browse repository at this point
Copy the full SHA a3aaedaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1285669 - Browse repository at this point
Copy the full SHA 1285669View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4124c25 - Browse repository at this point
Copy the full SHA 4124c25View commit details -
Configuration menu - View commit details
-
Copy full SHA for aabe723 - Browse repository at this point
Copy the full SHA aabe723View commit details -
Configuration menu - View commit details
-
Copy full SHA for 01a83cb - Browse repository at this point
Copy the full SHA 01a83cbView commit details -
Configuration menu - View commit details
-
Copy full SHA for c89c21b - Browse repository at this point
Copy the full SHA c89c21bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 33ca19f - Browse repository at this point
Copy the full SHA 33ca19fView commit details -
Configuration menu - View commit details
-
Copy full SHA for c5bc867 - Browse repository at this point
Copy the full SHA c5bc867View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2631d47 - Browse repository at this point
Copy the full SHA 2631d47View commit details -
Overhaul integration with front end macros
- The state machine generated by the front end macro must Now implement a small set of methods as a facade over the particular Future/Awaitable/Task type being used. This replaces the AST factory methods in `FutureSystem` - Remove intrinsic implemnentation for scala-async's async macro. Show what it will need to do in a test implementation, `s.t.n.partest.Async` and update all test cases to use this. - Add a heuristic to interpret `@compileTimeOnly` annotated methods to register `await` methods. Defer the check in refchecks for all of these and instead check that no references survive the async phase. - Refactor the test implementations of async front ends to share an interface for the state machine. We don't ship this but it could be copy/pasted into third party integrations. - Add a test integration with Java's CompletableFuture. - Expose a method through macro `Internals` API to let front ends mark a method in the state machine for async translation.
Configuration menu - View commit details
-
Copy full SHA for 11733cb - Browse repository at this point
Copy the full SHA 11733cbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 791c015 - Browse repository at this point
Copy the full SHA 791c015View commit details -
Configuration menu - View commit details
-
Copy full SHA for 36ad222 - Browse repository at this point
Copy the full SHA 36ad222View commit details -
Configuration menu - View commit details
-
Copy full SHA for d7a2764 - Browse repository at this point
Copy the full SHA d7a2764View commit details -
Allow compilation of async's partest suite in Junit
This lets us use IntelliJ's code coverage analysis.
Configuration menu - View commit details
-
Copy full SHA for 36d452e - Browse repository at this point
Copy the full SHA 36d452eView commit details -
Configuration menu - View commit details
-
Copy full SHA for dd463d1 - Browse repository at this point
Copy the full SHA dd463d1View commit details -
Fix detection of ill-nested awaits
- Adapt detection logic to the post-erasure tree shapes - Lift restriction about || and && in favour of a rewrite to `If` in the ANF transform - import relevant parts of the test from scala-async.
Configuration menu - View commit details
-
Copy full SHA for de4fc23 - Browse repository at this point
Copy the full SHA de4fc23View commit details -
Move ThicketTransformer to s.t.n.transform
It seems generally useful enough to move it out of the async utilities. I've also added a unit test.
Configuration menu - View commit details
-
Copy full SHA for 7d46e6b - Browse repository at this point
Copy the full SHA 7d46e6bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 469cfc1 - Browse repository at this point
Copy the full SHA 469cfc1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 074b480 - Browse repository at this point
Copy the full SHA 074b480View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3a2882f - Browse repository at this point
Copy the full SHA 3a2882fView commit details -
Configuration menu - View commit details
-
Copy full SHA for b68e118 - Browse repository at this point
Copy the full SHA b68e118View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0775a0d - Browse repository at this point
Copy the full SHA 0775a0dView commit details -
Configuration menu - View commit details
-
Copy full SHA for d1d3daa - Browse repository at this point
Copy the full SHA d1d3daaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8c770f2 - Browse repository at this point
Copy the full SHA 8c770f2View commit details -
Improve error reporting in asynj junit test.
Show source file context for reported compiler errors and warnings, as we would expect from the normal ConsoleReporter.
Configuration menu - View commit details
-
Copy full SHA for 4d0f6bb - Browse repository at this point
Copy the full SHA 4d0f6bbView commit details -
Configuration menu - View commit details
-
Copy full SHA for a5298c5 - Browse repository at this point
Copy the full SHA a5298c5View commit details -
Better integration with compiler fresh names
- No longer add the `$async` suffix to fresh names. Add a test to show that we no longer risk clashing with lambda lifted names because we're coordinating through the unit's FreshNameCreator. - Add a facility to FreshNameCreator to support clients that want to generate many fresh names from a single prefix and avoid the overhead of going through `newTermName` when subsequent units also want to generate `x$0`, etc. - Add a little efficiency to name concatenation by use of presized of string builders.
Configuration menu - View commit details
-
Copy full SHA for 499296c - Browse repository at this point
Copy the full SHA 499296cView commit details -
Configuration menu - View commit details
-
Copy full SHA for e73f502 - Browse repository at this point
Copy the full SHA e73f502View commit details -
Configuration menu - View commit details
-
Copy full SHA for bcb5dc6 - Browse repository at this point
Copy the full SHA bcb5dc6View commit details -
Configuration menu - View commit details
-
Copy full SHA for d81ef53 - Browse repository at this point
Copy the full SHA d81ef53View commit details -
Configuration menu - View commit details
-
Copy full SHA for 502c127 - Browse repository at this point
Copy the full SHA 502c127View commit details -
Move async run tests to jvm category
Some will be made Scala.js compatible in a subsequent commit and moved back to run.
Configuration menu - View commit details
-
Copy full SHA for e63a133 - Browse repository at this point
Copy the full SHA e63a133View commit details -
Configuration menu - View commit details
-
Copy full SHA for f9d7952 - Browse repository at this point
Copy the full SHA f9d7952View commit details -
Configuration menu - View commit details
-
Copy full SHA for 215f173 - Browse repository at this point
Copy the full SHA 215f173View commit details -
- Remove '_' and line number from state machine classs names. - Rename `state$async` to `state`.
Configuration menu - View commit details
-
Copy full SHA for 5455445 - Browse repository at this point
Copy the full SHA 5455445View commit details -
Configuration menu - View commit details
-
Copy full SHA for db3ea69 - Browse repository at this point
Copy the full SHA db3ea69View commit details -
Configuration menu - View commit details
-
Copy full SHA for 923e57b - Browse repository at this point
Copy the full SHA 923e57bView commit details
Commits on Jun 11, 2020
-
Configuration menu - View commit details
-
Copy full SHA for b76a0ec - Browse repository at this point
Copy the full SHA b76a0ecView commit details
Commits on Jun 12, 2020
-
Configuration menu - View commit details
-
Copy full SHA for c94951e - Browse repository at this point
Copy the full SHA c94951eView commit details
Commits on Jun 15, 2020
-
Rework async expression wrapping
Wrap the async expression in `locally { ... }` to force value class boxing. This seems to work better than `{ ... }: Any`, which ends up with `Function` trees typed with `Any` which violoates an assumption in `delambdafy`.
Configuration menu - View commit details
-
Copy full SHA for 7f22119 - Browse repository at this point
Copy the full SHA 7f22119View commit details -
Configuration menu - View commit details
-
Copy full SHA for 230684d - Browse repository at this point
Copy the full SHA 230684dView commit details
Commits on Jun 16, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 0e27334 - Browse repository at this point
Copy the full SHA 0e27334View commit details
Commits on Jun 17, 2020
-
Configuration menu - View commit details
-
Copy full SHA for ae91a3c - Browse repository at this point
Copy the full SHA ae91a3cView commit details
Commits on Jun 18, 2020
-
Refactor detection of ill nested await calls
We can defer the check until the current location of the "fallback" check without sacrificing the specific error messages.
Configuration menu - View commit details
-
Copy full SHA for 89f48d2 - Browse repository at this point
Copy the full SHA 89f48d2View commit details