fix: periodically detect and update cluster connection status

[chetan-rns]

Argo CD doesn't report connection errors on the cluster UI page unless the cache is invalidated (manually/periodically every 24 hours) or there is a sync error. This error is only updated during auto-sync if there is a new commit to sync. This PR adds a new field ConnectionStatus that gets updated whenever Argo CD fails to access the cluster. The ClusterInfoUpdater will periodically fetch this cluster info and update the cache.

The cluster cache will start a goroutine with a configurable interval that pings (calls the version API) the remote cluster and checks if the cluster is reachable. It only pings the remote cluster if it detects any watch errors in the cluster cache thereby avoiding unnecessary calls. The implementation can be found here: argoproj/gitops-engine#559

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[codecov]

Codecov Report

Attention: Patch coverage is 20.68966% with 23 lines in your changes are missing coverage. Please review.

Project coverage is 44.91%. Comparing base (21b1514) to head (cd0f052).
Report is 9 commits behind head on master.

Files	Patch %	Lines
controller/cache/cache.go	8.00%	22 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #18143      +/-   ##
==========================================
- Coverage   44.93%   44.91%   -0.03%     
==========================================
  Files         354      354              
  Lines       47742    47722      -20     
==========================================
- Hits        21454    21435      -19     
- Misses      23485    23488       +3     
+ Partials     2803     2799       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ikauzak]

hey there. We are looking for this feature as well. Any estimation date of when this will be merged? thank you

[andrii-korotkov-verkada]

What if the connections errors are intermittent? Reporting on the UI can be noisy.
Can you open an issue and give a more detailed context on how it would solve the problems, please?