Add etcd snapshot metrics

[brandond]

Proposed Changes

Add etcd snapshot metrics:

# HELP k3s_etcd_snapshot_reconcile_duration_seconds Total time taken to sync the list of etcd snapshots
# HELP k3s_etcd_snapshot_reconcile_local_duration_seconds Total time taken to list local snapshot files
# HELP k3s_etcd_snapshot_reconcile_s3_duration_seconds Total time taken to list S3 snapshot files
# HELP k3s_etcd_snapshot_save_duration_seconds Total time taken to complete the etcd snapshot process
# HELP k3s_etcd_snapshot_save_local_duration_seconds Total time taken to save a local snapshot file
# HELP k3s_etcd_snapshot_save_s3_duration_seconds Total time taken to upload a snapshot file to S3

Note that the total number of ETCDSnapshotFile resources present does NOT have a metric here - that can already be queried from apiserver storage object count metrics:
apiserver_storage_objects{resource="etcdsnapshotfiles.k3s.cattle.io"} 1

Types of Changes

enhancement

Verification

Check metrics on etcd node:

brandond@dev01:~$ kubectl get --raw /metrics | grep 'etcd_snapshot_.*_count'
k3s_etcd_snapshot_reconcile_duration_seconds_count{status="success"} 9
k3s_etcd_snapshot_reconcile_local_duration_seconds_count{status="success"} 11
k3s_etcd_snapshot_reconcile_s3_duration_seconds_count{status="error"} 9
k3s_etcd_snapshot_save_duration_seconds_count{status="success"} 1
k3s_etcd_snapshot_save_local_duration_seconds_count{status="success"} 1
k3s_etcd_snapshot_save_s3_duration_seconds_count{status="error"} 1

Note that there will be no s3 metrics if s3 is not enabled.

Testing

Linked Issues

• Expose prometheus metrics for etcd snapshots #11190

User-Facing Change

Further Comments

[codecov]

Codecov Report

Attention: Patch coverage is 47.91667% with 25 lines in your changes missing coverage. Please review.

Project coverage is 44.11%. Comparing base (2f787dd) to head (038c376).
Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/etcd/snapshot.go	13.79%	25 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11793      +/-   ##
==========================================
- Coverage   44.89%   44.11%   -0.78%     
==========================================
  Files         206      208       +2     
  Lines       19519    19548      +29     
==========================================
- Hits         8763     8624     -139     
- Misses       9485     9681     +196     
+ Partials     1271     1243      -28     

Flag	Coverage Δ
e2etests	35.55% <39.58%> (+1.08%)	⬆️
inttests	35.07% <41.66%> (+0.07%)	⬆️
unittests	16.48% <39.58%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.