3.5: Cannot list archived workflow in UI with /archived-workflows/

[zhucan]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

expect to list archived workflow like version 3.4.9:

but under version v3.5.8, can not list archived workflow:

Version(s)

v3.5.8

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

only create a normal wf, and waiting the wf was archived.

Logs from the workflow controller

no need to check the logs of controller.

Logs from in your workflow's wait container

no need to check the logs of wait container.

[zhucan]

other question is, user curl cmd to get wf, can reponse successfully, but list wf with limit, hangs
Get on wf:

curl -H "Authorization: $ARGO_TOKEN" 10.245.10.132:2746/api/v1/workflows/argo-data-closeloop/prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6
{"metadata":{"name":"prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6","generateName":"prod--prod--perception-autolabel-detection3d-taging-1-1-36--cdc37f6d-fe55-41cf-9cdf-ffcff5bd34ef--","namespace":"argo-data-closeloop","uid":"34c1d357-897f-40d3-baac-7aa6c2f93876","resourceVersion":"953580576","generation":2,"creationTimestamp":"2024-07-17T09:28:58Z","labels":{"workflows.argoproj.io/completed":"false","workflows.argoproj.io/creator":"system-serviceaccount-argo-data-closeloop-argo-data-closeloop","workflows.argoproj.io/phase":"Running"},"annotations":{"workflows.argoproj.io/pod-name-format":"v2"},"managedFields":[{"manager":"argo","operation":"Update","apiVersion":"argoproj.io/v1alpha1","time":"2024-07-17T09:28:58Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:workflows.argoproj.io/creator":{}}}}},{"manager":"workflow-controller","operation":"Update","apiVersion":"argoproj.io/v1alpha1","time":"2024-07-17T09:28:58Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:workflows.argoproj.io/pod-name-format":{}},"f:labels":{"f:workflows.argoproj.io/completed":{},"f:workflows.argoproj.io/phase":{}}},"f:spec":{},"f:status":{}}}]},"spec":{"templates":[{"name":"job-entrypoint","inputs":{},"outputs":{},"metadata":{},"script":{"name":"","image":"reg.deeproute.ai/deeproute-public/deelooper/prod__perception_autolabel_detection3d_taging:1.1.36","command":["bash"],"env":[{"name":"DEELOOPER_EXE_ENV","value":"prod"},{"name":"DEELOOPER_WORK_INSTANCE_ID","value":"3D_M8_TO_LABEL_20240717:3D_M8_TO_LABEL_20240717-13392"},{"name":"DEELOOPER_TASK_INSTANCE_ID","value":"3D_M8_TO_LABEL_20240717:3D_M8_TO_LABEL_20240717-13392:13941911034944"},{"name":"DEELOOPER_JOB_INSTANCE_ID","value":"669787e872f5e7dc2f4a3113"},{"name":"DEELOOPER_WORK_NAME","value":"3D_M8_TO_LABEL_20240717"},{"name":"DEELOOPER_TASK_NAME","value":"PERC_3D_obstacle_bag粗筛"},{"name":"DEELOOPER_TASK_CODE","value":"13941911034944"},{"name":"DEELOOPER_SWIMLANE","value":"baseline"},{"name":"DEELOOPER_BAG_NAME"},{"name":"DEELOOPER_TASK_IMAGE_ID","value":"reg.deeproute.ai/deeproute-public/deelooper/prod__perception_autolabel_detection3d_taging:1.1.36"},{"name":"DEELOOPER_PRIORITY","value":"1"},{"name":"DEELOOPER_DRMLP_USERNAME","valueFrom":{"secretKeyRef":{"name":"drmlp-account","key":"DRMLP_USERNAME"}}},{"name":"DEELOOPER_DRMLP_PASSWORD","valueFrom":{"secretKeyRef":{"name":"drmlp-account","key":"DRMLP_PASSWORD"}}}],"resources":{},"volumeMounts":[{"name":"dshm","mountPath":"/dev/shm"},{"name":"uwzsnjgkmoa43nwyexea4rkew4w3dsx3hau45xt6pw3wkhy4thmq","readOnly":true,"mountPath":"/mnt/csi-data-gfs"},{"name":"lyeoihi3y4r56jxsaevmeg4ncpsxd6txrgi7eybv4hghhqkhtmua","mountPath":"/mnt/csi-data-aly/shared/public"},{"name":"p3avl2bgnhlh4f7qaemtnog2ln4ezt7d3i7ev4e4ahnxl6tb2suq","mountPath":"/mnt/core-data"}],"securityContext":{"capabilities":{"add":["SYS_PTRACE"]}},"source":"(echo \"### mountpoint gfs:\" \u0026\u0026 mountpoint /mnt/csi-data-gfs || true) \u0026\u0026 (echo \"### mountpoint public:\" \u0026\u0026 mountpoint /mnt/csi-data-aly/shared/public || true) \u0026\u0026 (echo \"### mountpoint core-data:\" \u0026\u0026 mountpoint /mnt/core-data || true) \u0026\u0026 (echo \"### mountpoint dshm:\" \u0026\u0026 mountpoint /dev/shm || true) \u0026\u0026 (echo \"### mountpoint label-data:\" \u0026\u0026 mountpoint /mnt/label_data || true) \u0026\u0026 bash ./start.sh"},"retryStrategy":{"limit":6,"retryPolicy":"Always","backoff":{"duration":"6s"},"expression":"lastRetry.status == \"Error\" || asInt(lastRetry.exitCode) == 89"},"schedulerName":"volcano","podSpecPatch":"{\n  \"containers\": [\n    {\n      \"name\": \"main\",\n      \"resources\": {\n        \"requests\": {\n          \"cpu\": \"1\",\n          \"memory\": \"1Gi\"\n        },\n        \"limits\": {\n          \"cpu\": \"1\",\n          \"memory\": \"1Gi\"\n        }\n      }\n    }\n  ]\n}"}],"entrypoint":"job-entrypoint","arguments":{},"serviceAccountName":"argo-workflow","volumes":[{"name":"dshm","emptyDir":{"medium":"Memory","sizeLimit":"64Mi"}},{"name":"uwzsnjgkmoa43nwyexea4rkew4w3dsx3hau45xt6pw3wkhy4thmq","hostPath":{"path":"/mnt/csi-data-gfs","type":"Directory"}},{"name":"lyeoihi3y4r56jxsaevmeg4ncpsxd6txrgi7eybv4hghhqkhtmua","hostPath":{"path":"/mnt/csi-data-aly/shared/public","type":"Directory"}},{"name":"p3avl2bgnhlh4f7qaemtnog2ln4ezt7d3i7ev4e4ahnxl6tb2suq","hostPath":{"path":"/mnt/core-data","type":"Directory"}}],"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"creator":"argo"}},"topologyKey":"kubernetes.io/hostname"}}]}},"tolerations":[{"key":"cloud.deeproute.cn/team","operator":"Equal","value":"mlp","effect":"NoSchedule"}],"activeDeadlineSeconds":7776000,"priority":1,"schedulerName":"volcano","podGC":{"strategy":"OnPodSuccess"},"podSpecPatch":"{\"containers\":[{\"name\":\"main\", \"resources\":{\"requests\":{\"cpu\": \"1\", \"memory\": \"2Gi\" }, \"limits\":{\"cpu\": \"2\", \"memory\": \"8Gi\" }}}]}","podMetadata":{"labels":{"creator":"argo","deelooper.deeproute.cn/task-code":"13941911034944","deelooper.deeproute.cn/work-id":"619e1f6b0bae4bfa0e46e396207964b5179fc2cb"}}},"status":{"phase":"Running","startedAt":"2024-07-17T09:28:58Z","finishedAt":null,"progress":"0/1","nodes":{"prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6":{"id":"prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6","name":"prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6","displayName":"prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6","type":"Retry","templateName":"job-entrypoint","templateScope":"local/prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6","phase":"Running","startedAt":"2024-07-17T09:28:58Z","finishedAt":null,"progress":"0/1","children":["prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6-3593406971"]},"prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6-3593406971":{"id":"prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6-3593406971","name":"prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6(0)","displayName":"prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6(0)","type":"Pod","templateName":"job-entrypoint","templateScope":"local/prod--prod--perception-autolabel-detection3d-taging-1-1-36zv6b6","phase":"Pending","startedAt":"2024-07-17T09:28:58Z","finishedAt":null,"progress":"0/1","nodeFlag":{"retried":true}}},"artifactRepositoryRef":{"default":true,"artifactRepository":{"archiveLogs":true,"s3":{"endpoint":"s3.deeproute.cn:80","bucket":"argo-workflow-artifacts","insecure":true,"accessKeySecret":{"name":"argo-server-sso","key":"accesskey"},"secretKeySecret":{"name":"argo-server-sso","key":"secretkey"}}}},"artifactGCStatus":{"notSpecified":true}}}

list wf:

[zhucan]

https://terrytangyuan.github.io/2023/08/14/argo-workflows-v3.5/ I know it use the "Unified workflows list UI and API"，but the list cmd with curl no any responses and hangs.

[agilgur5]

Duplicate of #12545, #13126 (reply in thread), etc.

https://terrytangyuan.github.io/2023/08/14/argo-workflows-v3.5/ I know it use the "Unified workflows list UI and API"

Correct, there is no split UI in 3.5+, as the blog post says. They are combined. The archived workflows API still exists, but the workflows API will now return archived workflows as well and the UI pages are combined.

---

successfully, but list wf with limit, hangs

list wf:

This is unrelated. Please keep issues on a single topic.

You also used a screenshot despite me asking you not to several times before as they are not accessible.

As a result I had to stare at your screenshot and manually type your API query. Again, please do not use screenshots of code or logs or config. Use text.

Your API query in the screenshot is:

curl -H "Authorization: $ARGO_TOKEN" 10.245.10.132:2746/api/v1/workflows/argo-data-closeloop?\&limit=50 -v

The ?\& looks like an incorrect query parameter to me. It should be ?limit=50.

That invalid query parameter may mean that you're making a request with no limit, hence it will take very long. You can look at your Server logs.

Additionally, your bearer token is also visible in the screenshot. You'll want to rotate your Secret.

[zhucan]

The ?& looks like an incorrect query parameter to me. It should be ?limit=50.

it's correct, you can take a look at debug informations.

Additionally, your bearer token is also visible in the screenshot. You'll want to rotate your Secret.

thanks for your reminder, dose no matter, it's only test env

As a result I had to stare at your image and manually type your API query. Again, please do not use images of code or logs or config.

we make the image from source code, can the push the image to our internal image repo

[agilgur5]

we make the image from source code, can the push the image to our internal image repo

I think you misunderstood me. In this case, by image I meant your screenshot (PNG). I have edited my comment to clarify that.

it's correct, you can take a look at debug informations.

That too would be easier in text form than with a screenshot that I have to zoom into.

If it's correct, I can't reproduce this hanging and neither can our tests. I would drill that down to a minimal reproduction and if it still seems like a bug to you, you can open a separate bug report with that reproduction. And as I wrote before, you'd want to check the Server logs.

There are some remaining performance issues in 3.5 like #13295 but I haven't seen any reports of hanging indefinitely. In those cases, which particularly impact larger workflows, it may take 10+ seconds, but it still returns eventually. Again the Server logs will reveal information, as will your DB logs etc.

[zhucan]

@agilgur5 Could you give me a exmaples for the curl request?

1. Get token firstly: kubectl exec -ti argo-workflow-server-586dcb9dd-vxb67 -n argo-workflow -- argo auth token
2. Export the reponses of the first step's result: export ARGO_TOKEN="xxxxx"
3. Use curl to get wf: curl -H "Authorization: $ARGO_TOKEN" 10.245.10.132:2746/api/v1/workflows/argo-data-closeloop?\&limit=50 -v

like this?

[agilgur5]

I usually follow the Access Token docs.

1. Get token firstly:

An exec into the Server seems unnecessary and insecure

3. Use curl to get wf:

Again, I'd use the ?limit=50 syntax.

Note that the UI uses the API, as does the CLI (when in Server mode)