dynamic path/tag variables #372

epcim · 2023-03-14T07:43:13Z

Description

This is POC/Draft for discussion only.

Implements #345

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update
This change requires a new example

epcim · 2023-04-12T08:25:01Z

@codablock pls review/comment so I can finish it, thx

codablock

Added a few comments. Sorry for this taking so long.

pkg/deployment/deployment_item.go

codablock · 2023-04-17T07:31:24Z

pkg/deployment/deployment_item.go

+	}
+	div.RelPath = pe
+	div.RevPath = pr
+	div.RelRoot = strings.Repeat("../", len(strings.Split(di.RelToSourceItemDir, "/")))


what does this try to achieve?

it creates ../../../ ,, assume as prefix to get project root.

what ever you are nested, to load any shared component from project root is as easy as

so to load sth. from compose dir - with shared components (that are re-used many times in various app deployments)

/compose/_dockerconfig /targets .kluctl.yml deployment.yaml

Example:

targets/baseLayer/etcd/kustomization.yml 4:- {{ this.relRoot }}/compose/_dockerconfig 5:- {{ this.relRoot }}/compose/_wingmansigner

Simply using /compose/_dockerconfig does not work? If I remember correct, it was at least supposed to work...if not, it's maybe a bug.

For kustomize itself it would certainly not work if the structure is like below

├── compose │ ├── dockerconfig │ ├── cluster │ ├── deployment.yml │ ├── argo │ │ ├── kustomization.yml ├── deployment.yml ├── .kluctl.yml

✗ Building kustomize objects ✗ 1 error occurred: * building kustomize objects for /Users/xxxx/Workspace/sre-staging-model/deployment/kube/cluster/argo failed. accumulating components: loader.New "new root '/compose/dockerconfig' cannot be absolute"

to make it working, you would have to on temp. build dir update paths of bases:, resources:, components:

some "magic" could be done here: https://github.com/kluctl/kluctl/blob/main/pkg/deployment/deployment_item.go#L443

Non-related topic

in this discussion we would even evaluate whether we can copy to "build dir" the linked resources. I can explain in detail but if Kustomize.yml under target refers a file (in bases: or components:) out of its subdirectories (above paths) then it needs to be copied to .build dir either.

I was using this, but it has many unexpected consequences.

deployments: # WORKAROUND to append kustomize resources/dependencies to build dir #- service directory is rendered once using global params {% for s in params.keys(). %} - path: ../service/{{ s }} onlyRender: true vars: - file: ../service/{{ s }}/context.yml ignoreMissing: true {% endfor %}

Ok I see, even if kluctl templating would be able to load from / (pointing to the project root), kustomize would fail as it simply does not allow absolute pathes.

I'll think about this and maybe create one or two commits in regard to naming of these vars.

pkg/deployment/deployment_item.go

codablock · 2023-04-19T08:12:53Z

This will be important for you as well: #416
It moves to using encoding/json and json tags instead of yaml tags. It then uses conversion between json/yaml to achieve yaml support. It's basically what k8s is doing since forever and I decided to go the same route from now on.

For you this means that your structs need to be updated with json tags.

epcim · 2023-04-21T09:34:59Z

pkg/deployment/deployment_project.go

 		if item.Path != nil {
+			// Validate or fix item.Path, as k8s labels has constrains
+			// regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')
+			if *item.Path == "." {


This change make this possible ((path: .) & highlight that for example _stage0 as invalid - we can consider some validation later, possibly even later on di.Tags

deployments: - path: . #<-- current dir, sharing deployment.yml & kustomization.yml - git: url: git@gitlab.com:

Can you put this into a separate PR so that it properly appears in the changelog later?

epcim · 2023-04-26T18:44:44Z

remved "draft" prefix, but still want to add some documentation & example.

I kept two topics open for @codablock for another round of discussion.

codablock

I've create a few commits that I think make sense here:
d8049b0
1bd9651
0cc97f2

The first one already renames the "d" variable to "di", but I'm still unsure if this is the best naming. I'm also considering the fully written version "deploymentItem"...

codablock · 2023-05-22T14:31:36Z

pkg/deployment/deployment_project.go

 		if item.Path != nil {
+			// Validate or fix item.Path, as k8s labels has constrains
+			// regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')
+			if *item.Path == "." {


Can you put this into a separate PR so that it properly appears in the changelog later?

codablock · 2023-05-22T14:59:48Z

Also, a rebase is required here

epcim · 2023-05-23T06:01:13Z

Sure, will check your updates.One think I would like to add it is to have DI.name for even git reference - which either will fix missing include tag for it. item.Path == "." i was actually about tu place it separate. "di" Is nice and for full name we could have an alias. PMi22. 5. 2023 v 16:59, Alexander Block ***@***.***>: @codablock requested changes on this pull request. I've create a few commits that I think make sense here: d8049b0 1bd9651 0cc97f2 The first one already renames the "d" variable to "di", but I'm still unsure if this is the best naming. I'm also considering the fully written version "deploymentItem"... In pkg/deployment/deployment_project.go:

if item.Path != nil {

+ // Validate or fix item.Path, as k8s labels has constrains + // regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?') + if *item.Path == "." { Can you put this into a separate PR so that it properly appears in the changelog later? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

epcim · 2023-06-05T11:17:29Z

Also, a rebase is required here

removed, rebased

epcim · 2023-06-20T12:36:48Z

Latest commit fix di.relPath for latest kluctl rebase master and fix behavior on latest kluctl.

If the kluctl is in subfolder of the git repo, then the DeployItem relative dir (original variable) is path within the git repo - this was recently introduced as a fix on main branch.

which somehow with code: for deployments.yml in PATH/IN/REPO folder:

deployments:
  - path: ./compose
    onlyRender: true

ends up to upload "compose" to ./build/0/PATH/IN/REPO/compose vs. original behaviour ./build/0/compose (or today if the kluctl is not executed from "git repo".

So I had to somehow get the relative only to the project dir - which ended up as new definition for DeploymentProjectVars.TopPath variable.

@codablock I am now not any way dependant on this PR. But still seams to me nice feature. You want to make some changes as (vr renames/better path handling).

epcim force-pushed the dyn-var-draft branch 2 times, most recently from ebbb5d4 to e4dcfdf Compare March 30, 2023 18:36

codablock reviewed Apr 17, 2023

View reviewed changes

epcim commented Apr 21, 2023

View reviewed changes

epcim changed the title ~~Draft: dynamic path/tag variables~~ dynamic path/tag variables Apr 26, 2023

codablock requested changes May 22, 2023

View reviewed changes

epcim force-pushed the dyn-var-draft branch from e7ea03b to 0a0d4e3 Compare June 5, 2023 11:16

epcim force-pushed the dyn-var-draft branch from 0a0d4e3 to b124e69 Compare June 5, 2023 11:28

dynamic path/tag variables

0c0cc9e

epcim force-pushed the dyn-var-draft branch from b124e69 to 0c0cc9e Compare June 5, 2023 11:30

fix di.relPath for latest kluctl

c4e6277

senbax-admin mentioned this pull request Jul 16, 2023

Passing vars/templates/data between git-included deployments #669

Open

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dynamic path/tag variables #372

dynamic path/tag variables #372

epcim commented Mar 14, 2023

epcim commented Apr 12, 2023

codablock left a comment

codablock Apr 17, 2023

epcim Apr 17, 2023 •

edited

codablock Apr 19, 2023

epcim Apr 21, 2023 •

edited

codablock Apr 27, 2023

codablock commented Apr 19, 2023

epcim Apr 21, 2023 •

edited

codablock May 22, 2023

epcim commented Apr 26, 2023

codablock left a comment

codablock May 22, 2023

codablock commented May 22, 2023

epcim commented May 23, 2023 via email

epcim commented Jun 5, 2023

epcim commented Jun 20, 2023

dynamic path/tag variables #372

Are you sure you want to change the base?

dynamic path/tag variables #372

Conversation

epcim commented Mar 14, 2023

Description

Type of change

epcim commented Apr 12, 2023

codablock left a comment

Choose a reason for hiding this comment

codablock Apr 17, 2023

Choose a reason for hiding this comment

epcim Apr 17, 2023 • edited

Choose a reason for hiding this comment

codablock Apr 19, 2023

Choose a reason for hiding this comment

epcim Apr 21, 2023 • edited

Choose a reason for hiding this comment

Non-related topic

codablock Apr 27, 2023

Choose a reason for hiding this comment

codablock commented Apr 19, 2023

epcim Apr 21, 2023 • edited

Choose a reason for hiding this comment

codablock May 22, 2023

Choose a reason for hiding this comment

epcim commented Apr 26, 2023

codablock left a comment

Choose a reason for hiding this comment

codablock May 22, 2023

Choose a reason for hiding this comment

codablock commented May 22, 2023

epcim commented May 23, 2023 via email

epcim commented Jun 5, 2023

epcim commented Jun 20, 2023

epcim Apr 17, 2023 •

edited

epcim Apr 21, 2023 •

edited

epcim Apr 21, 2023 •

edited