Document execution engine

platforms/core-execution/execution/.editorconfig

@@ -0,0 +1,2 @@
+[*.adoc]
+indent_size = 2

platforms/core-execution/execution/README.adoc

@@ -0,0 +1,69 @@
+// Copyright (C) 2023 Gradle, Inc.
+//
+// Licensed under the Creative Commons Attribution-Noncommercial-ShareAlike 4.0 International License.;
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      https://creativecommons.org/licenses/by-nc-sa/4.0/
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+= Execution Engine
+
+The role of the **execution engine** is to take some work with well-defined inputs and outputs, and using the inputs manifest the outputs as efficiently as possible.
+
+The engine is implemented as a network of steps.
+During execution the engine passes a _context_ object along that accumulates information about the work as it gets passed deeper in the hierarchy of steps.
+When the outputs have been manifested, a _result_ object gets passed back along the same path, accumulating metadata about the manifested outputs along the way.
+
+== Semantic structure of pipeline
+
+[mermaid]
+....
+flowchart TD
+  Identify[Identify work]
+  IdentityCache{{In identity cache?}}
+  ChoosePipeline{{Work type?}}
+
+  Identify -->|identity| IdentityCache
+  IdentityCache -->|hit| Done
+  IdentityCache -->|miss| ChoosePipeline
+  Done[Done]
+
+  ChoosePipeline -->|"non-incremental work\n(excluding tasks)"| ImmutablePipeline
+  subgraph ImmutablePipeline[Immutable pipeline]
+    LoadImmutableHistory["Load history\n(check if workspace exists)"]
+
+    LoadImmutableHistory -->|hit| CheckImmutableWorkspace[Check workspace]
+    LoadImmutableHistory -->|miss| AllocateTemporaryWorkspace[Allocate temporary workspace]
+    AllocateTemporaryWorkspace --> CheckIfImmutableEmpty{{Empty sources?}}
+    CheckIfImmutableEmpty -->|no| ExecuteNonIncrementalWork["Execute work\n(or load from cache)"]
+    ExecuteNonIncrementalWork -->|success| StoreImmutableHistory
+    CheckIfImmutableEmpty -->|yes| StoreImmutableHistory
+    StoreImmutableHistory["Store history"] --> MoveWorkspaceToImmutableArea[Move workspace\nto immutable area]
+  end
+  CheckImmutableWorkspace --> Done
+  ExecuteNonIncrementalWork -->|failure| Done
+  MoveWorkspaceToImmutableArea --> Done
+
+  ChoosePipeline --->|"tasks and incremental work"| MutablePipeline
+  subgraph MutablePipeline["Mutable pipeline"]
+    AllocateMutableWorkspace[Allocate mutable workspace\nunder lock]
+
+    AllocateMutableWorkspace --> LoadMutableHistory[Load history]
+    LoadMutableHistory --> CleanStaleOutputs[Clean stale outputs]
+    CleanStaleOutputs --> SkipIfSourcesEmpty{{Empty sources?}}
+    SkipIfSourcesEmpty -->|no| SkipUpToDate{{Is up-to-date?}}
+    SkipIfSourcesEmpty -->|yes| DeletePreviousOutputs[Delete previous outputs]
+    SkipUpToDate -->|no| ExecuteIncrementalWork["Execute work\n(or load from cache)"]
+    DeletePreviousOutputs ----> StoreMutableHistory[Store history]
+    ExecuteIncrementalWork --> RecordOutputs["Record outputs\n(for stale output cleanup later)"]
+    RecordOutputs --> StoreMutableHistory
+  end
+  StoreMutableHistory --> Done
+  SkipUpToDate -->|yes| Done
+....

platforms/core-execution/execution/Steps.md

@@ -0,0 +1,94 @@
+```mermaid
+flowchart TD
+  Identify[Identify work] <--> MutablePipeline[Mutable pipeline]
+  Identify <--> ImmutablePipeline[Immutable pipeline]
+  MutablePipeline <--> Execution[Execution pipeline]
+  ImmutablePipeline <--> Execution
+```
+
+## Identification of the work
+
+```mermaid
+sequenceDiagram
+  Execution engine ->> Identify: ExecutionRequestContext
+    Identify -> Identify: Capture identity inputs
+    Identify ->> IdentityCache: IdentityContext
+    alt Not in identity cache
+      IdentityCache -->> ChoosePipeline: IdentityContext
+      alt Mutable work
+        ChoosePipeline ->> Mutable pipeline: IdentityContext
+        Mutable pipeline ->> ChoosePipeline: Result
+      else Immutable work
+        ChoosePipeline ->> Immutable pipeline: IdentityContext
+        Immutable pipeline ->> ChoosePipeline: Result
+      end
+      ChoosePipeline -->> IdentityCache: Result
+    end
+    IdentityCache ->> Identify: Result
+  Identify ->> Execution engine: Result
+```
+
+## Immutable pipeline
+
+```mermaid
+sequenceDiagram
+  AssignImmutableWorkspace ->> AssignTemporaryWorkspace: ImmutableWorkspaceContext
+  alt Immutable workspace missing
+    AssignTemporaryWorkspace ->> Validate: TemporaryWorkspaceContext
+      Validate ->> ResolveCachingState: ValidationFinishedContext
+        ResolveCachingState ->> SkipEmptyImmutableWork: CachingContext
+        alt Non-empty sources
+          SkipEmptyImmutableWork ->> BuildCache: CachingContext
+          alt Cache miss
+            BuildCache ->> BroadcastChangingOutputs: CachingContext
+              BroadcastChangingOutputs ->> Execution pipeline: ChangingOutputsContext
+              Execution pipeline ->> BroadcastChangingOutputs: Result
+            BroadcastChangingOutputs ->> BuildCache: Result
+          end
+          BuildCache ->> SkipEmptyImmutableWork: CachingResult
+        end
+        SkipEmptyImmutableWork ->> ResolveCachingState: CachingResult
+      ResolveCachingState ->> Validate: CachingResult
+    Validate ->> AssignTemporaryWorkspace: CachingResult
+    AssignTemporaryWorkspace ->> AssignTemporaryWorkspace: Store history
+    AssignTemporaryWorkspace ->> AssignTemporaryWorkspace: Move workspace to immutable location
+  end
+  AssignTemporaryWorkspace ->> AssignImmutableWorkspace: CachingResult
+```
+
+## Mutable pipeline
+
+```mermaid
+sequenceDiagram
+
+  AssignPersistentWorkspace ->> HandleStaleOutputs: WorkspaceContext
+    HandleStaleOutputs ->> LoadPreviousExecutionState: WorkspaceContext
+      LoadPreviousExecutionState ->> SkipEmptyIncrementalWork: PreviousExecutionContext
+        SkipEmptyIncrementalWork ->> CaptureStateBeforeExecution: PreviousExecutionContext
+          CaptureStateBeforeExecution ->> Validate: BeforeExecutionState
+            Validate ->> ResolveCachingState: ValidationFinishedContext
+              ResolveCachingState ->> ResolveChanges: CachingContext
+                ResolveChanges ->> SkipUpToDate: IncrementalChangesContext
+                  SkipUpToDate ->> StoreExecutionState: IncrementalChangesContext
+                    StoreExecutionState ->> BuildCache: PreviousExecutionContext
+                      BuildCache ->> ResolveInputChanges: IncrementalChangesContext
+                        ResolveInputChanges ->> CaptureStateAfterExecution: InputChangesContext
+                          CaptureStateAfterExecution ->> BroadcastChangingOutputs: BeforeExecutionState
+                            BroadcastChangingOutputs ->> RemovePreviousOutputs: ChangingOutputsContext
+                              RemovePreviousOutputs ->> Execution pipeline: ChangingOutputsContext
+                              Execution pipeline ->> RemovePreviousOutputs: Result
+                            RemovePreviousOutputs ->> BroadcastChangingOutputs: Result
+                          BroadcastChangingOutputs ->> CaptureStateAfterExecution: Result
+                        CaptureStateAfterExecution ->> ResolveInputChanges: AfterExecutionResult
+                      ResolveInputChanges ->> BuildCache: Result
+                    BuildCache ->> StoreExecutionState: AfterExecutionResult
+                  StoreExecutionState ->> SkipUpToDate: AfterExecutionResult
+                SkipUpToDate ->> ResolveChanges: UpToDateResult
+              ResolveChanges ->> ResolveCachingState: Result
+            ResolveCachingState ->> Validate: CachingResult
+          Validate ->> CaptureStateBeforeExecution: CachingResult
+        CaptureStateBeforeExecution ->> SkipEmptyIncrementalWork: CachingResult
+      SkipEmptyIncrementalWork ->> LoadPreviousExecutionState: AfterExecutionResult
+    LoadPreviousExecutionState ->> HandleStaleOutputs: AfterExecutionResult
+  HandleStaleOutputs ->> AssignPersistentWorkspace: AfterExecutionResult
+```