"Override Parameters" retry only uses parameters specified when submitting workflowtemplate

[amfage]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issue exists when I tested with :latest
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

Behaviour in v3.5.5
When using the example WorkflowTemplate provided below to submit a workflow using the Argo CLI, when a Retry is selected in the UI on a failed workflow, only the argument parameters specified on the command line when submitting are available for override. The parameters that were not specified on the command line (where the default values in the WorkflowTemplate were used in the intial submit) are not available for override and are also not submitted on the retry, so they are missing and unable to be resolved.

Using the example WorkflowTemplate supplied below, the submit command would be:

argo submit --namespace argo --from workflowtemplate/test-simple-fail-for-retry --parameter param1="FOO"

param2 is not specified, the default value from the WorkflowTemplate is used.

This gives the output:

FOO
bar

before failing.
Through the UI, choose to retry, checking "Override Parameters"

Only the parameter param1 supplied when submitted is shown. param2 does not appear in the list.

On retry, param2 is not resolved, and the output is:

baz
{{workflow.parameters.param2}}

A retry without parameter override can resolve all WorkflowTemplate parameters successfully, including those not specified on the CLI when submitting.

Expected Behaviour
All WorkflowTemplate parameters are available for override and can be resolved, even if not specified when submitting through the CLI (i.e. the WorkflowTemplate default value has been used when submitting).

Note
Retrying from the CLI shows the same behaviour, but it is possible to specify the "missing" parameters to work around this issue.

Version

v3.5.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

# This example sleeps for 30 seconds then fails, so it can be retried
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: test-simple-fail-for-retry
spec:
  entrypoint: template-fail
  arguments:
    parameters:
      - name: param1
        value: 'foo'
      - name: param2
        value: 'bar'
  templates:
    - name: template-fail
      script:
        image: 'python:alpine3.6'
        command: ["python"]
        # fail
        source: |
          import sys;
          import time;
          print("{{workflow.parameters.param1}}");
          print("{{workflow.parameters.param2}}");
          time.sleep(30);
          sys.exit(1);

Logs from the workflow controller

N/A

Logs from in your workflow's wait container

N/A

[agilgur5]

If I'm not mistaken, that sounds like correct behavior. Can you provide the Workflow after you submitted it? I'm pretty sure it would only have one parameter at the top-level.

I also don't think this is specific to the UI. Likely not retries either -- if you did a resubmit I think it would end up similar for the same reasons.

Retrying from the CLI shows the same behaviour, but it is possible to specify the "missing" parameters to work around this issue.

Right I think that adds up, because in the CLI you can specify arbitrary parameters

The same behavior could be supported in the UI if you want to reword this as a feature request, but I don't believe this is a bug as it's based on the submitted Workflow (which has its own set of parameters that can be independent of the WorkflowTemplate)

[amfage]

Thanks for the explanation.

Yes, it does show only the one parameter in the top-level of the Workflow that is run. For us, since there are some parameters we almost always use the default value for, we can't use the UI feature to override a parameter. I guess Argo uses a different method when retrying the workflow without any parameter overrides being specified.

[agilgur5]

Yes, it does show only the one parameter in the top-level of the Workflow that is run.

Ok, that would confirm that this is correct behavior.

For us, since there are some parameters we almost always use the default value for, we can't use the UI feature to override a parameter. I guess Argo uses a different method when retrying the workflow without any parameter overrides being specified.

Sorry, not quite sure I understand what you mean here. The CLI sufficed for you but an equivalent UI feature would not suffice? The CLI and UI both talk to the same API, so the "method" is the same in the end. Not sure if you mean something else regarding retry w/ and w/o overrides?

[amfage]

Sorry, not quite sure I understand what you mean here. The CLI sufficed for you but an equivalent UI feature would not suffice? The CLI and UI both talk to the same API, so the "method" is the same in the end. Not sure if you mean something else regarding retry w/ and w/o overrides?

I was meaning that if you retry the workflow without overriding the parameters, Argo will use the same default and specified parameters that were originally submitted.
If you retry the workflow and override a parameter that was originally specified, Argo will not use the defaults.

[agilgur5]

If you retry the workflow and override a parameter that was originally specified, Argo will not use the defaults.

Wouldn't that be expected? (as the definition of an override).
You also didn't answer why an equivalent UI feature to the CLI's arbitrary parameters would not suffice

Trying to understand if we want to reword this as a feature request for arbitrary parameters in the UI or if there's something else missing that you'd like.

Also, it helps to think of Workflows as a sequence of functions. You have top-level global parameters for the whole file/Workflow and then input parameters for each function/template. A templateRef effectively in-lines a function call/template from another file/Workflow. A workflowTemplateRef is similar.
Parameters you specify when submitting are the top-level ones. When running a file/Workflow, any functions/templates within are encapsulated, including their input parameters.

[amfage]

Wouldn't that be expected? (as the definition of an override).

I expected it for the parameter that I overrode, but not for the other parameters. But now I know that is intended behaviour we can work around it on the command line.

You also didn't answer why an equivalent UI feature to the CLI's arbitrary parameters would not suffice

Apologies, yes I think that would suffice.

[agilgur5]

I expected it for the parameter that I overrode, but not for the other parameters.

Ah I gotcha now. Basically you're providing a default list in your WorkflowTemplate; so an override is an override of the list.

But now I know that is intended behaviour we can work around it on the command line.

Depending on your use-case, there might be a way to correctly specify this. You can set input parameters for a template, say, your entrypoint, and have an expression, something like:

inputs:
  parameters:
    - myParam: "{{= workflow.parameters.myParam ? workflow.parameters.myParam : 'myDefault' }}"

It's been a while, but I've done something like that in templates when I was heavily using Workflows at my last job. Syntax might not be 100% correct and potentially could be simplified.

You also didn't answer why an equivalent UI feature to the CLI's arbitrary parameters would not suffice

Apologies, yes I think that would suffice.

Ok, wanted to make sure. At this point this thread is pretty windy, could you file a separate feature request for that to keep things tidy?

[agilgur5]

Ok, wanted to make sure. At this point this thread is pretty windy, could you file a separate feature request for that to keep things tidy?

I filed #13035 to cover this feature request