WIP: input tracking #5496

stefreak · 2023-11-29T16:09:06Z

What this PR does / why we need it:

Input tracking, to allow excluding inputs in a future PR
Separate parsing template strings from evaluating template strings by generating an AST in the parser
Implement lazy evaluation
implement a proxy so we avoid changing too much existing code that uses templates

Todo:

Integrate all that with the existing code base (Parsing, validating and processing configs)

Special notes for your reviewer:

Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com>

146 passing (206ms) 95 failing Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com>

stefreak · 2023-11-29T16:18:34Z

core/src/template-string/ast.ts

+        value: v.value,
+        inputs: {
+          // key might be something like ["var", "foo", "bar"]
+          // We also add the keypath to get separate keys for ever


Suggested change

// We also add the keypath to get separate keys for ever

// We also add the keypath to get separate keys for every leaf

core/src/template-string/ast.ts

core/src/template-string/functions.ts

core/src/template-string/parser.pegjs

TimBeyer · 2023-11-30T10:31:01Z

core/src/template-string/inputs.ts

+    }
+  }
+
+  relevantValues.forEach((r) => {


I think we should see if we can add this linter rule to our setup: https://github.com/github/eslint-plugin-github/blob/main/docs/rules/array-foreach.md

The reasons for why are well explained in the link.

stefreak · 2023-12-01T12:20:18Z

core/src/template-string/ast.ts

+  constructor(
+    loc: Location,
+    public readonly condition: TemplateExpression,
+    public ifTrue: TemplateExpression,


I think ifTrue is also optional actually, for example for this expression it would be undefined: ${if ...}{else}Hello world{endif} (It would be undefined because there are no expressions for the then case between the if and the else)

Would be good to add tests for these cases

stefreak · 2023-12-01T14:05:53Z

core/src/template-string/ast.ts

+      )
+    }
+
+    return mergeInputs(transformed, left, right)


No need to merge inputs for concatenating arrays. We likely should skip this step for collections.

stefreak · 2023-12-01T14:19:16Z

core/src/template-string/ast.ts

+      })
+    }
+
+    return new TemplateLeaf({


This wipes the inputs of the inner expression;
We need to actually:

Suggested change

return new TemplateLeaf({

return inner

core/src/template-string/ast.ts

stefreak · 2023-12-07T18:58:13Z

core/src/template-string/lazy.ts

+      if (v instanceof LazyValue) {
+        return new MergeInputsLazily(this.source, v, unwrappedRelevantValues)
+      }
+      return mergeInputs(this.source, v satisfies TemplateLeaf<TemplateLeafValue>, ...unwrappedRelevantValues)


Possible infinite recursion if unwrappedRelevantValues is a collection that contains lazy values?

…nement Co-authored-by: Tim Beyer <TimBeyer@users.noreply.github.com> Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com>

behaviour

Co-authored-by: Tim Beyer <TimBeyer@users.noreply.github.com> Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com>

stefreak · 2023-12-13T07:43:33Z

core/src/config/base.ts

  } else if (allowInvalid) {
-    return spec
+    // we shouldn't throw


Do we really need allowInvalid? Why does it exist?

stefreak · 2023-12-13T07:48:24Z

core/src/config/project.ts

@@ -335,8 +336,7 @@ export const projectSchema = createSchema({
    ),
    defaultEnvironment: joi
      .environment()
-      .allow("")
-      .default("")
+      .default((parent) => parent.environments[0].name)


Suggested change

.default((parent) => parent.environments[0].name)

.default((parent) => parent.environments?.[0]?.name || "")

Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com> Co-authored-by: Tim Beyer <TimBeyer@users.noreply.github.com>

Co-authored-by: Tim Beyer <TimBeyer@users.noreply.github.com> Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com>

stefreak · 2023-12-13T18:59:01Z

core/src/config/project.ts

+    defaultPath: defaultVarfilePath,
+  })
+
+  const environmentConfigContextVariables = new LayeredContext(


We need tests for some edge cases here:
e.g.

variables: my deep: structure: foo

with override
variables.other.deep.structure: bar

stefreak · 2023-12-13T19:04:37Z

core/src/config/template-contexts/base.ts

          message = res.message
          partial = !!res.partial
+        } else {
+          // TODO: improve error message
+          throw new ConfigurationError({


Maybe instead of error, it should just result in a plain object with all the values the context contains.

stefreak · 2023-12-13T19:06:59Z

core/src/config/template-contexts/base.ts

+        opts: {
+          ...opts,
+          // Throw an error if we can't find the value in the last layer, unless allowPartial is set
+          allowPartial: isLastLayer ? opts.allowPartial : false,


Unnecessary; If we do not find anything, we return CONTEXT_RESOLVE_KEY_NOT_FOUND instead of throwing.

stefreak · 2023-12-13T20:08:33Z

core/test/unit/src/template-string/validation.ts

+// In the future we might
+// const varsFromFirstConfig = firstConfig.atPath("var") // can contain lazy values
+// const actionConfigWithVars = actionConfig.merge(firstConfig)


Suggested change

// In the future we might

// const varsFromFirstConfig = firstConfig.atPath("var") // can contain lazy values

// const actionConfigWithVars = actionConfig.merge(firstConfig)

stefreak · 2023-12-13T21:52:55Z

core/src/template-string/lazy.ts

+  abstract visitAll(): TemplateExpressionGenerator
+}
+
+export class OverrideKeyPathLazily extends LazyValue {


Potentially not needed?

Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com> Co-authored-by: Tim Beyer <TimBeyer@users.noreply.github.com>

stefreak · 2023-12-14T18:07:14Z

core/src/template-string/lazy.ts

+    context: ConfigContext,
+    opts: ContextResolveOpts
+  ): Record<string, CollectionOrValue<TemplateValue>> {
+    // Resolve $merge keys, depth-first, leaves-first


Suggested change

// Resolve $merge keys, depth-first, leaves-first

thsig and others added 18 commits November 21, 2023 13:45

Save

93d2931

test: add test and first take at implementation for simple, 1 ref

595e652

test: add simple test for ensuring we get leafs

9d9ecf0

fix: resolve references only once. Uncomment the throw.

e2ebf08

fix: build

3c06ecd

test: records template references (array, 1 reference)

73dcad9

fix: path accounting

c125de4

fix: add reference target logic

00d8a82

Save

4bff56a

test: more test cases

24ea2e1

wip

fcc37d4

wip (does not compile)

d47d5e1

wip (compiles now)

c80a2cf

wip: rename the types in inputs.ts

3f9fa88

wip: temporary hack to avoid changing the parser right now

49ca406

Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com>

wip: remove recorder. compiles, but all tests fail.

1f7ee2a

Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com>

wip: fix some tests

1484e9d

wip: start implementing template string AST

5504c56

146 passing (206ms) 95 failing Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com>

stefreak commented Nov 29, 2023

View reviewed changes

core/src/template-string/ast.ts Outdated Show resolved Hide resolved

stefreak commented Nov 29, 2023

View reviewed changes

core/src/template-string/ast.ts Outdated Show resolved Hide resolved

stefreak commented Nov 29, 2023

View reviewed changes

core/src/template-string/functions.ts Outdated Show resolved Hide resolved

TimBeyer reviewed Nov 30, 2023

View reviewed changes

core/src/template-string/parser.pegjs Outdated Show resolved Hide resolved

TimBeyer reviewed Nov 30, 2023

View reviewed changes

stefreak added 2 commits November 30, 2023 14:57

wip: all template tests pass, except for blocks and partial

3e72e3d

ast: all template string tests pass

e99008c

stefreak commented Dec 1, 2023

View reviewed changes

core/src/template-string/ast.ts Outdated Show resolved Hide resolved

stefreak added 3 commits December 7, 2023 15:45

wip: working proxy and more tests

88e67b2

add test for partial proxy

5320506

partial proxy & logical operators

70670d9

stefreak commented Dec 7, 2023

View reviewed changes

stefreak and others added 12 commits December 8, 2023 14:09

wip

30eece3

wip: mutable proxy overlay

dd634d4

wip: make proxy read-only and record changes during validation / refi…

7a3f170

…nement Co-authored-by: Tim Beyer <TimBeyer@users.noreply.github.com> Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com>

test(validation): add failing test for withContext and schema override

f342808

behaviour

fix: ensure Object.keys works correctly on array

04adf6b

chore: remove confusing lazy class

e944ea2

chore: add some docs for diffing logic

dfbee86

refactor: do overlays on the config object

f8fc3f4

wip

3960f9a

allow refining multiple times

58e4297

wip: add joi refine method

1ad69f9

Co-authored-by: Tim Beyer <TimBeyer@users.noreply.github.com> Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com>

wip: integrate 1. next: pickEnvironment

edab666

stefreak commented Dec 13, 2023

View reviewed changes

stefreak and others added 2 commits December 13, 2023 15:28

wip: provider initialization

7fb1f04

Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com> Co-authored-by: Tim Beyer <TimBeyer@users.noreply.github.com>

wip: scanAndAddConfigs

da81199

Co-authored-by: Tim Beyer <TimBeyer@users.noreply.github.com> Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com>

stefreak commented Dec 13, 2023

View reviewed changes

stefreak and others added 2 commits December 14, 2023 13:25

wip. next: renderConfigTemplate

44a0a5e

Co-authored-by: Thorarinn Sigurdsson <thorarinnsigurdsson@gmail.com> Co-authored-by: Tim Beyer <TimBeyer@users.noreply.github.com>

wip: renderModules and renderConfigTemplate

9038449

stefreak commented Dec 14, 2023

View reviewed changes

stefreak mentioned this pull request Jan 25, 2024

[FEATURE]: filter sensitive values from log output #5666

Open

stefreak mentioned this pull request Mar 12, 2024

0.13: [Bug]: Template variables are being interpolated on input values #5825

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: input tracking #5496

WIP: input tracking #5496

stefreak commented Nov 29, 2023 •

edited

stefreak Nov 29, 2023

TimBeyer Nov 30, 2023

stefreak Dec 1, 2023 •

edited

stefreak Dec 1, 2023

stefreak Dec 1, 2023

stefreak Dec 7, 2023

stefreak Dec 13, 2023

stefreak Dec 13, 2023

stefreak Dec 13, 2023

stefreak Dec 13, 2023

stefreak Dec 13, 2023 •

edited

stefreak Dec 13, 2023

stefreak Dec 13, 2023

stefreak Dec 14, 2023

	// We also add the keypath to get separate keys for ever
	// We also add the keypath to get separate keys for every leaf

	.default((parent) => parent.environments[0].name)
	.default((parent) => parent.environments?.[0]?.name \|\| "")

	// In the future we might
	// const varsFromFirstConfig = firstConfig.atPath("var") // can contain lazy values
	// const actionConfigWithVars = actionConfig.merge(firstConfig)

WIP: input tracking #5496

Are you sure you want to change the base?

WIP: input tracking #5496

Conversation

stefreak commented Nov 29, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefreak Dec 1, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefreak Dec 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefreak commented Nov 29, 2023 •

edited

stefreak Dec 1, 2023 •

edited

stefreak Dec 13, 2023 •

edited