3.5.6+ items.status.nodes disappeared from /api/v1/workflows endpoint for completed Workflows

[jacek-jablonski]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

While updating argo-workflows to version v3.5.6, I noticed that the status.nodes field disappeared for completed workflows in endpoint /api/v1/workflows. If the workflow is in the Running status, this field still appears. Is this expected behavior?

I cannot force this field to appear with: https://argo-workflows.xxx/api/v1/workflows/?listOptions.limit=50&fields=items.status.nodes
Up to v3.5.5 it was working fine.

Version

v3.5.6

[agilgur5]

Follow-up from this Slack thread. Per thread, nodeStatusOffload is not enabled and this only occurs on the list endpoint.

Given that it only occurs on list and completed Workflows, I'm guessing this is only impacting Archived Workflows, and so would be caused by #12912 (comment)

[agilgur5]

I cannot force this field to appear with: https://argo-workflows.xxx/api/v1/workflows/?listOptions.limit=50&fields=items.status.nodes
Up to v3.5.5 it was working fine.

This part would be the regression -- fields is supposed to be dynamic.

If the workflow is in the Running status, this field still appears. Is this expected behavior?

So in 3.4, the Archived Workflows list wouldn't return nodes by default either per the diff I commented on in #11121 (comment).

The UI's default list request also doesn't include it.

So I think it being included in 3.5 was actually an unintended bug -- part of the whole performance regression that #12912 fixed that I discovered in #11121 (comment)

[agilgur5]

So in 3.4, the Archived Workflows list wouldn't return nodes by default either per the diff I commented on in #11121 (comment).

Looking at it a bit more, it may not have been possible at all for Archived Workflows list in 3.4?

So this is perhaps intended behavior for Archived Workflows lists, just that in 3.5 they're also available in the regular list API as well.

[jacek-jablonski]

So, to summarize: status.nodes field appeared in 3.5 list endpoint by a mistake and it shouldn't be there. And by design it won't return?

[agilgur5]

Specifically for Archived Workflows, not necessarily live Workflows.

[jacek-jablonski]

@agilgur5 items.metadata.ownerReferences seems also missing for archived workflows. There seems to be quite a big discrepancy between what is returned for archived and live workflows on a single API endpoint.

[agilgur5]

Yea anything that doesn't have it's own column in the archived DB is not returned right now. And adding more columns or plucking more from the JSON blob causes performance issues per #13295 etc 😕

[jacek-jablonski]

Is it a subject to change in 3.6? Or will it stay the same?

[agilgur5]

They're technically supposed to be set dynamically per your fields parameters as I mentioned in #12912 (comment). If you can make that work, I'd probably consider that a patch fix in 3.5 even, but at a glance it looks non-trivial to do

[jacek-jablonski]

@agilgur5 I've set in my config:

worfklowDefaults:
  spec:
    ttlStrategy:
      secondsAfterCompletion: 345600 # 4 days

so that I could get some more information on the completed workflows, but despite the fact they are still live in k8s cluster, some fields are still missing (seems that, if it is archived, the data goes from the db, not from the cluster).
It's quite problematic if you have a custom GUI and prevents us from updating to 3.5.6+.

[jacek-jablonski]

If I disable the archive, the fields are visible again for completed workflows, so that confirms the above situation.