API Permissions needed for /api/v1/workflows/{namespace}/submit

[wesleyhaws]

I worked through this doc: https://argoproj.github.io/argo-workflows/access-token/

and got an access token. I'm able to:

curl -X GET https://{endpoint}/api/v1/workflow-templates/{namespace}/{template} -H "Authorization: Bearer $ARGO_TOKEN"

just fine. However, if I run:

curl -X POST \
  --url https://workflows-dev-main.shippodev.com/api/v1/workflows/argo-workflows/submit \
  --header 'Authorization: Bearer $ARGO_TOKEN' \
  --data '{ "resourceKind": "WorkflowTemplate", "namespace": "{namespace}", "resourceName": "{template}", "submitOptions": { "parameters": [ "{param1}={param1_value}", "{param2}={param2_value}" ], "submitOptions": { "labels": "workflows.argoproj.io/workflow-template={template}", "name": "{arbitrary name}" } } }'

I get the following error:

{"code":16,"message":"Unauthorized"}

for the life of me I cannot figure out what permissions I might need to do this. I can't seem to find any discussion or docs anywhere on the matter. What might I be doing wrong?

[wesleyhaws]

NOTE: I have tried the following permissions on the role that the service account i'm using is using:

      {
        api_groups = ["argoproj.io"]
        resources  = ["*"]
        verbs      = ["list", "get", "watch"]
      },
      {
        api_groups = ["argoproj.io"]
        resources  = ["workflows", "workflowtaskresults"]
        verbs      = ["create", "patch"]
      }

I even tried it with resources *, and it didn't do anything. So there must be something else somewhere that I'm missing.

[wesleyhaws]

How do I regenerate my token to have a valid one then, as it seems like this token is auto generated by something else?

[agilgur5]

It's just the ServiceAccount's Secret, per the docs.
Bearer is part of it the token, so when you used BearerToken that is correctly invalid. You didn't get that error when using Bearer <secret>, meaning that your initial header was correct.

[wesleyhaws]

I think I'm getting closer here:

curl -X 'POST' \
  'https://{endpoint}/api/v1/workflows/{namespace}/submit' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "resourceKind": "WorkflowTemplate",
    "namespace": "{namespace}",
    "resourceName": "scripts-testing",
    "submitOptions": {
        "entryPoint": "failure",
        "parameters": [
            "id=0",
            "name=sample"
        ],
        "labels": "workflows.argoproj.io/workflow-template=scripts-testing",
        "name": "scripts-testing-callback"
    }
}'

Will now spit out:

{"code":2,"message":"The POST operation against Workflow.argoproj.io could not be completed at this time, please try again."}

Just to make sure it's not a role issue, I made sure to try this out with verbs of * on workflows and workflowtaskresults. No matter how many times I try to issue the POST it says the same thing. The logs on the server say the same thing with no additional helpful info sadly.

Role permissions:

PolicyRule:
  Resources                                Non-Resource URLs  Resource Names  Verbs
  ---------                                -----------------  --------------  -----
  workflows.argoproj.io                    []                 []              [*]
  workflowtaskresults.argoproj.io          []                 []              [*]
  *.argoproj.io                            []                 []              [list watch get]
  *.dataflow.argopro.io                    []                 []              [list watch get]

[wesleyhaws]

Okay I found this helpful link: #7580

Sure enough when I changed the name it ran (at least a dryrun, nothing is displaying in the UI yet). I guess I need to scour the API paths to guess how I can submit the same workflow name with different parameters (or re-sumbit)

[agilgur5]

Ah different issue. generateName is commonly used to provide different names

[wesleyhaws]

So people don't have to read through the chain, because I know more people are going to run across this, and save yourself days worth of debugging. Here is how to get this to work.

1. Deploy the workflow template
2. Deploy your SA, Rolebinding, and secret
3. You will need these permissions (maybe not all of these but this is what I tried it with and it worked)

PolicyRule:
  Resources                                Non-Resource URLs  Resource Names  Verbs
  ---------                                -----------------  --------------  -----
  workflows.argoproj.io                    []                 []              [create patch]
  workflowtaskresults.argoproj.io          []                 []              [create patch]
  *.argoproj.io                            []                 []              [list watch get]
  *.dataflow.argopro.io                    []                 []              [list watch get]

4. You can get your api token by doing:

export ARGO_TOKEN=$(kg secret <secret_name> -o=jsonpath='{.data.token}' -n <namespace> | base64 --decode)

NOTE: Still unsure if this is a short lived token or not.

5. You MUST use Authorization: Bearer <token> in your curl headers, even though the API shows Authorization: <token> or even Authorization: BearerToken <token>.

6. Issue the submit curl:

curl -X 'POST' \
  'https://{endpoint}/api/v1/workflows/{namespace}/submit' \
  -H 'accept: application/json' \
  -H "Authorization: Bearer ${ARGO_TOKEN}" \
  -H 'Content-Type: application/json' \
  -d '{
    "resourceKind": "WorkflowTemplate",
    "namespace": "{namespace}",
    "resourceName": "{workflow_template_name}",
    "submitOptions": {
        "dryrun": true,
        "entryPoint": "{template_name}",
        "parameters": [
            ...
        ],
        "labels": "workflows.argoproj.io/workflow-template={workflow_template_name}",
        "name": "{Whatever_you_want_Here}"
    }
}'

NOTE: The above entryPoint is NOT the spec.entrypoint, but the spec.templates[?].name.
NOTE2: You can use submitOptions.generateName to name the workflow yourself otherwise it inherits the name of the workflow-{random_string}

Some caveats. If the workflow has already been run with the provided name it will throw this error:

The POST operation against Workflow.argoproj.io could not be completed at this time, please try again.

To fix it, just change the name of your run. Alternatively you can just "rerun" the workflow by resubmitting it with the previous runs name:

curl -X 'PUT' \
  'https://{endpoint}/api/v1/workflows/{namespace}/{prev_run_name}/resubmit' \
  -H 'accept: application/json' \
  -H "Authorization: Bearer ${ARGO_TOKEN}" \
  -H 'Content-Type: application/json' \
  -d '{
        "dryrun": true,
        "parameters": [
            ...
        ],
        "name": "{prev_run_name}"
}'

[agilgur5]

5. You MUST use Authorization: Bearer <token> in your curl headers, even though the API shows Authorization: <token> or even Authorization: BearerToken <token>.

In the docs, $ARGO_TOKEN includes Bearer . You are referring to the ServiceAccount Secret, which is the second half of $ARGO_TOKEN (the part after Bearer ).

Where does the API say say "BearerToken "? That should definitely be corrected. Feel free to submit a PR as well.

[agilgur5]

The main reference I could find for "BearerToken" was in the generated SDKs. Those are generated via OpenAPI tooling, so not directly written by Argo. I also did not find a reference that specifically said Authorization: BearerToken <token>.

[agilgur5]

Made some small clarifications to two places in the docs that were a bit inconsistent in #11714