The `test` option has become fussy with command line option ordering #450

jim-barber-he · 2022-10-19T09:43:43Z

Operating system

Debian stable (bullseye 11)

Helmfile Version

0.147.0

Helm Version

v3.10.1

Bug description

Running

helmfile -e $ENVIRONMENT_NAME -l name=$RELEASE test

results in the error:

err: no releases found that matches specified selector() and environment(-l), in any helmfile

But running it like so:

helmfile test -e $ENVIRONMENT_NAME -l name=$RELEASE

works:

$ helmfile test -e test -l name=catalyst
Building dependency release=catalyst, chart=helm/chart/catalyst
Testing catalyst

I'm not sure what version this broke at, but it was working for version 0.145.2 of helmfile (with helm 3.9.2).

Example helmfile.yaml

---
environments:
  prod:
  test:

helmDefaults:
  atomic: true
  cleanupOnFail: true
  createNamespace: false
  timeout: {{ env "HELM_TIMEOUT" | default "600" }}

releases:
  - name: catalyst
    chart: ./helm/chart/catalyst
    namespace: {{ env "RELEASE_NAMESPACE" | default "catalyst" }}
    values:
      - helm/vars/values.yaml.gotmpl

Error message you've seen (if any)

err: no releases found that matches specified selector() and environment(-l), in any helmfile

Steps to reproduce

Any helm chart that has a tests directory with tests to run

Working Helmfile Version

0.145.2

Relevant discussion

No response

The text was updated successfully, but these errors were encountered:

yxxhero · 2022-10-19T10:02:56Z

@jim-barber-he Please show the debug output by adding the --debug option.

jim-barber-he · 2022-10-19T12:26:20Z

Debug output

$ helmfile --debug -e test -l name=catalyst test
processing file "helmfile.yaml" in directory "."
changing working directory to "/home/jim/git/catalyst/master"
first-pass rendering starting for "helmfile.yaml.part.0": inherited=&{-l map[] map[]}, overrode=<nil>
first-pass uses: &{-l map[] map[]}
first-pass rendering output of "helmfile.yaml.part.0":
 0: ---
 1: environments:
 2:   prod:
 3:   test:
 4: 
 5: helmDefaults:
 6:   atomic: true
 7:   cleanupOnFail: true
 8:   createNamespace: false
 9:   timeout: 600
10: 
11: releases:
12:   - name: catalyst
13:     chart: ./helm/chart/catalyst
14:     labels:
15:       app: catalyst
16:     missingFileHandler: Info
17:     namespace: catalyst
18:     values:
19:     
20:       - helm/vars/values.yaml.gotmpl
21:     
22: 

first-pass produced: &{-l map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {-l map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "helmfile.yaml.part.0":
 0: ---
 1: environments:
 2:   prod:
 3:   test:
 4: 
 5: helmDefaults:
 6:   atomic: true
 7:   cleanupOnFail: true
 8:   createNamespace: false
 9:   timeout: 600
10: 
11: releases:
12:   - name: catalyst
13:     chart: ./helm/chart/catalyst
14:     labels:
15:       app: catalyst
16:     missingFileHandler: Info
17:     namespace: catalyst
18:     values:
19:     
20:       - helm/vars/values.yaml.gotmpl
21:     
22: 

changing working directory back to "/home/jim/git/catalyst/master"
err: no releases found that matches specified selector() and environment(-l), in any helmfile

jim-barber-he · 2022-10-19T12:28:27Z

And also debug output when switching the command line option around so that test is before the -e and -l options:

$ helmfile --debug test -e test -l name=catalyst
processing file "helmfile.yaml" in directory "."
changing working directory to "/home/jim/git/catalyst/master"
first-pass rendering starting for "helmfile.yaml.part.0": inherited=&{test map[] map[]}, overrode=<nil>
first-pass uses: &{test map[] map[]}
first-pass rendering output of "helmfile.yaml.part.0":
 0: ---
 1: environments:
 2:   prod:
 3:   test:
 4: 
 5: helmDefaults:
 6:   atomic: true
 7:   cleanupOnFail: true
 8:   createNamespace: false
 9:   timeout: 600
10: 
11: releases:
12:   - name: catalyst
13:     chart: ./helm/chart/catalyst
14:     labels:
15:       app: catalyst
16:     missingFileHandler: Info
17:     namespace: catalyst
18:     values:
19:     
20:       - helm/vars/values.yaml.gotmpl
21:     
22: 

first-pass produced: &{test map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {test map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "helmfile.yaml.part.0":
 0: ---
 1: environments:
 2:   prod:
 3:   test:
 4: 
 5: helmDefaults:
 6:   atomic: true
 7:   cleanupOnFail: true
 8:   createNamespace: false
 9:   timeout: 600
10: 
11: releases:
12:   - name: catalyst
13:     chart: ./helm/chart/catalyst
14:     labels:
15:       app: catalyst
16:     missingFileHandler: Info
17:     namespace: catalyst
18:     values:
19:     
20:       - helm/vars/values.yaml.gotmpl
21:     
22: 

merged environment: &{test map[] map[]}
helm:pwDSS> v3.10.1+g9f88ccb
Building dependency release=catalyst, chart=helm/chart/catalyst
exec: helm dependency build helm/chart/catalyst
1 release(s) matching name=catalyst found in helmfile.yaml

Testing catalyst
exec: helm test catalyst --namespace catalyst --timeout 600s
helm:oulmW> NAME: catalyst
LAST DEPLOYED: Wed Oct 19 08:11:26 2022
NAMESPACE: catalyst
STATUS: deployed
REVISION: 3
TEST SUITE:     catalyst-integration-tests
Last Started:   Wed Oct 19 20:27:05 2022
Last Completed: Wed Oct 19 20:27:08 2022
Phase:          Succeeded
NAME: catalyst
LAST DEPLOYED: Wed Oct 19 08:11:26 2022
NAMESPACE: catalyst
STATUS: deployed
REVISION: 3
TEST SUITE:     catalyst-integration-tests
Last Started:   Wed Oct 19 20:27:05 2022
Last Completed: Wed Oct 19 20:27:08 2022
Phase:          Succeeded

release "catalyst" processed
changing working directory back to "/home/jim/git/catalyst/master"

jim-barber-he · 2022-10-19T12:42:09Z

It also has the same problem with version 0.146.0 of helmfile.

If I go back to helmfile 0.145.2 but still using helm 3.10.1 then it works okay.
So I started trying the releases in between and found that the problem appears at version 0.145.3

yxxhero · 2022-10-19T12:47:15Z

@jim-barber-he Thanks for your feedback.

yxxhero · 2022-10-19T12:53:46Z

@yxxhero I think this issue is only when the env name is test because subcommand is test too.

yxxhero · 2022-10-19T12:58:16Z

helmfile --debug -e=test -l name=catalyst test

Could you try to use it like the above?

yxxhero · 2022-10-19T12:58:28Z

@jim-barber-he

jim-barber-he · 2022-10-19T23:58:40Z

That works.

$ helmfile version    

▓▓▓ helmfile

  Version            0.147.0
  Git Commit         ad25755
  Build Date         09 Oct 22 15:58 AWST (1 week ago)
  Commit Date        09 Oct 22 09:13 AWST (1 week ago)
  Dirty Build        no
  Go version         1.18.6
  Compiler           gc
  Platform           linux/amd64
$ helmfile --debug -e=test -l=name=catalyst test
processing file "helmfile.yaml" in directory "."
changing working directory to "/home/jim/git/catalyst/master"
first-pass rendering starting for "helmfile.yaml.part.0": inherited=&{test map[] map[]}, overrode=<nil>
first-pass uses: &{test map[] map[]}
first-pass rendering output of "helmfile.yaml.part.0":
 0: ---
 1: environments:
 2:   prod:
 3:   test:
 4: 
 5: helmDefaults:
 6:   atomic: true
 7:   cleanupOnFail: true
 8:   createNamespace: false
 9:   timeout: 600
10: 
11: releases:
12:   - name: catalyst
13:     chart: ./helm/chart/catalyst
14:     labels:
15:       app: catalyst
16:     missingFileHandler: Info
17:     namespace: catalyst
18:     values:
19:     
20:       - helm/vars/values.yaml.gotmpl
21:     
22: 

first-pass produced: &{test map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {test map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "helmfile.yaml.part.0":
 0: ---
 1: environments:
 2:   prod:
 3:   test:
 4: 
 5: helmDefaults:
 6:   atomic: true
 7:   cleanupOnFail: true
 8:   createNamespace: false
 9:   timeout: 600
10: 
11: releases:
12:   - name: catalyst
13:     chart: ./helm/chart/catalyst
14:     labels:
15:       app: catalyst
16:     missingFileHandler: Info
17:     namespace: catalyst
18:     values:
19:     
20:       - helm/vars/values.yaml.gotmpl
21:     
22: 

merged environment: &{test map[] map[]}
helm:xJKRd> v3.10.1+g9f88ccb
Building dependency release=catalyst, chart=helm/chart/catalyst
exec: helm dependency build helm/chart/catalyst
1 release(s) matching name=catalyst found in helmfile.yaml

Testing catalyst
exec: helm test catalyst --namespace catalyst --timeout 600s
helm:ZWZCp> NAME: catalyst
LAST DEPLOYED: Wed Oct 19 08:11:26 2022
NAMESPACE: catalyst
STATUS: deployed
REVISION: 3
helm:ZWZCp> TEST SUITE:     catalyst-integration-tests
Last Started:   Thu Oct 20 07:56:46 2022
Last Completed: Thu Oct 20 07:58:17 2022
Phase:          Succeeded
NAME: catalyst
LAST DEPLOYED: Wed Oct 19 08:11:26 2022
NAMESPACE: catalyst
STATUS: deployed
REVISION: 3
TEST SUITE:     catalyst-integration-tests
Last Started:   Thu Oct 20 07:56:46 2022
Last Completed: Thu Oct 20 07:58:17 2022
Phase:          Succeeded

release "catalyst" processed
changing working directory back to "/home/jim/git/catalyst/master"

yxxhero · 2022-10-20T00:00:19Z

@jim-barber-he Thanks for your feedback. I will create an issue for the cobra project.

yxxhero · 2022-10-20T02:10:21Z

@jim-barber-he see: spf13/cobra#1781

yxxhero · 2022-10-20T06:04:54Z

@jim-barber-he if cobra fixed. I will post a PR for this issue.

stale · 2022-11-03T10:16:13Z