Inconsistent Listing of Workflows

[om3171991]

Hi Team,

We are facing a weird issue where listing of workflow is inconsistent which mean sometime its show few workflow and sometime not. For example

In screenshot1, We tried to list all workflow with label "ProjectName=Build" but its only show few workflow. Like in screenshot2, workflow name which has "gitops" are not coming in the list though same label is used. Also if we removed all filters from Argo Workflow dashboard, then as well can't see all workflow. Can you please suggest what I am doing wrong here?

[agilgur5]

Pagination with sorting will show a warning, which might be related. Does the problem persist without pagination?

Also there is a possible regression around the UI list: #9696. Is it inconsistent with the same filters when using the CLI?

[agilgur5]

Thanks for providing those, that's useful debugging info. So the UI is only ignoring Running Workflows then? That would suggest they might be missing a status.startedAt, but that shouldn't ever happen 🤔

If you're able to reliably reproduce this, could you provide the k8s object for one of the Workflows that is in the CLI and not the UI? status.phase, status.startedAt, and metadata.creationTimestamp fields specifically are useful, though the whole Workflow object might have some more debugging info.

This does look like the same issue as #9696 then if the CLI is correct. I am planning on making another change to the UI to make the two have equivalent output. The basic difference though is that --since tracks metadata.creationTimestamp while the UI uses status.startedAt.
The UI currently only shows Workflows with no status.startedAt if they are Pending. This potentially means that the Workflow got into a Running state without its status.startedAt getting set -- which would be a bug. That or potentially a flawed network request somewhere

[om3171991]

So the UI is only ignoring Running Workflows then ?
I don't think so, other workflow are missing as well. Check in below screenshot. No last 6hrs workflow is coming in UI whether completed, running or pending etc

could you provide the k8s object
I have removed org specific details from below K8 Object but this not visible in UI.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  annotations:
    workflows.argoproj.io/pod-name-format: v2
  creationTimestamp: "2023-09-15T06:08:58Z"
  generateName: website-monorepo-1a-695e8a1-
  generation: 5
  labels:
    ProjectName: website-monorepo
    Team: UI
    events.argoproj.io/action-timestamp: "1694758138353"
    events.argoproj.io/sensor: website-monorepo-1-sensor-pr-update
    events.argoproj.io/trigger: tm-website-pr-updated-trigger
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/phase: Succeeded
  name: website-monorepo-1a-695e8a1-jfxkp
  namespace: argo-events
spec:
  entrypoint: dag
  securityContext:
    runAsNonRoot: true
    runAsUser: 10000
  templates:
  - dag:
      tasks:
      - arguments: {}
        name: npm
        template: npm
    inputs: {}
    metadata: {}
    name: dag
    outputs: {}
  - name: npm
    metadata: {}
    outputs: {}
    script:
      command:
      - bash
      image: ubuntu:22.04
      name: ""
      resources: {}
      source: |
        #!/bin/sh
        echo "-----------------------------"
        echo "      STARTING PIPELINE      "
        echo "-----------------------------"
        echo "      PIPELINE RUNNING     "
        echo "-----------------------------"
        echo "     PIPELINE COMPLETED      "
        echo "-----------------------------"
      volumeMounts:
      - mountPath: /workspace
        name: output
      workingDir: /workspace/src
  volumeClaimGC:
    strategy: OnWorkflowCompletion
  volumeClaimTemplates:
  - metadata:
      creationTimestamp: null
      name: workspace
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 25Gi
    status: {}
  volumes:
  - name: output
    persistentVolumeClaim:
      claimName: website-monorepo-1a-695e8a1-jfxkp-workspace
status:
  conditions:
  - status: "False"
    type: PodRunning
  - status: "True"
    type: Completed
  finishedAt: "2023-09-15T06:16:03Z"
  nodes:
    website-monorepo-1a-695e8a1-jfxkp:
      children:
      - website-monorepo-1a-695e8a1-jfxkp-1649639236
      displayName: website-monorepo-1a-695e8a1-jfxkp
      finishedAt: "2023-09-15T06:16:03Z"
      id: website-monorepo-1a-695e8a1-jfxkp
      name: website-monorepo-1a-695e8a1-jfxkp
      outboundNodes:
      - website-monorepo-1a-695e8a1-jfxkp-1649639236
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 648
        memory: 648
      startedAt: "2023-09-15T06:08:58Z"
      templateName: dag
      templateScope: local/website-monorepo-1a-695e8a1-jfxkp
      type: DAG
    website-monorepo-1a-695e8a1-jfxkp-1649639236:
      boundaryID: website-monorepo-1a-695e8a1-jfxkp
      displayName: npm
      finishedAt: "2023-09-15T06:15:53Z"
      hostNodeName: ip-10-90-111-126.ec2.internal
      id: website-monorepo-1a-695e8a1-jfxkp-1649639236
      name: website-monorepo-1a-695e8a1-jfxkp.npm
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 648
        memory: 648
      startedAt: "2023-09-15T06:08:58Z"
      templateName: npm
      templateScope: local/website-monorepo-1a-695e8a1-jfxkp
      type: Pod
  phase: Succeeded
  progress: 1/1
  resourcesDuration:
    cpu: 648
    memory: 648
  startedAt: "2023-09-15T06:08:58Z"

--since This is used to get limited data else cli list command is timing out.

[agilgur5]

I don't think so, other workflow are missing as well. Check in below screenshot. No last 6hrs workflow is coming in UI whether completed, running or pending etc

Well that's not right... That is duplicative of #9696, though I wasn't able to reproduce that locally.

I have removed org specific details from below K8 Object but this not visible in UI.

Huh... this object has all the needed fields (status.startedAt, status.finishedAt, metadata.creationTimestamp, status.phase)... not sure why it isn't showing up then...

[agilgur5]

I raised a PR to make the UI date filters work identically to the CLI: #11840.
I wasn't able to reproduce this specific issue as I mentioned above, but hopefully once that PR's merged and you upgrade to it, this should be resolved.

I'm also going to close this out as duplicative of #9696. Can follow-up there or in a new issue if there are still bugs after #11840 (would need to be able to create or access a repro of this if so), as this would be a bug as opposed to a discussion.

[om3171991]

@agilgur5 Thanks for help. Will wait for PR to get merge.

[agilgur5]

#11840 was merged and is currently out as part of 3.5.0-rc2 (may cherry-pick it to 3.4.12, not sure). Could you try with 3.5.0-rc2 and see if the problem is fixed?

[om3171991]

Thanks. We have began a testing with new version.

But We have notice another bug with 3.5.0-rc2. Whenever we refreshes the workflow page in browser then we see a warning message and it goes to quickly that we aren't able to read message. Attaching a screenshot of message. Not sure if it introduced with this release or it exist in master code.

[terrytangyuan]

Could you raise a separate issue?

[om3171991]

Don't see any UI warning. Yes, It exist without pagination as well.