v3.5.7: namespaced install causes the server to complain about list+watch on cluster scoped workflows

[Joibel]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

To reproduce install 3.5.7 namespaced
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.5.7/namespace-install.yaml

You will see

W0613 08:46:47.382693       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1alpha1.Workflow: workflows.argoproj.io is forbidden: User "system:serviceaccount:argo:argo-server" cannot list
 resource "workflows" in API group "argoproj.io" at the cluster scope                                                                                                                                                                          
E0613 08:46:47.382720       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Workflow: failed to list *v1alpha1.Workflow: workflows.argoproj.io is forbidden: User "system:servicea
ccount:argo:argo-server" cannot list resource "workflows" in API group "argoproj.io" at the cluster scope    

This does not happen with 3.5.6 namespaced installs.

I believe this is coming from #12736 / #13021 on lines

argo-workflows/server/workflow/workflow_server.go

Line 73 in b28486c

return wfClientSet.ArgoprojV1alpha1().Workflows(metav1.NamespaceAll).List(context.Background(), options)

and

argo-workflows/server/workflow/workflow_server.go

Line 76 in b28486c

return wfClientSet.ArgoprojV1alpha1().Workflows(metav1.NamespaceAll).Watch(context.Background(), options)

It feels like these should be namespaced when the server is running namespaced.

Version

v3.5.7

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Not relevant.

Logs from the workflow controller

Not relevant. Server logs above.

Logs from in your workflow's wait container

Not relevant

[Joibel]

@jiachengxu, could you look at this and see if you can come up with a fix please?

[jiachengxu]

@jiachengxu, could you look at this and see if you can come up with a fix please?

sure @Joibel could you assign this to me? I don't have permission to do that

[agilgur5]

It feels like these should be namespaced when the server is running namespaced.

Yes, yes they should.
This is the first time we've had an Informer/Reflector on the Server, so all the namespacing logic that the Controller has didn't exist there. Prior to it, all requests had a namespace, so that logic was limited in the Server.

Specifically, if the Server is in --namespaced mode, it should use its own namespace, unless it also has a --managed-namespace, in which case it should use that. If unset, the ListWatch should be cluster-scope as it is now.