Tablet throttler: read and use MySQL host metrics

[shlomi-noach]

Description

In #16850 we added the necessary proto/gRPC calls to get hsot metrics from the MySQL daemon. #16850 is part of the v21 release.

This PR complements #16850 by having the tablet throttler read and use said MySQL host metrics. These will henceforth be available to the user by the names:

• mysqld-loadavg (default threshold: 1.0)
• mysqld-datadir-used-ratio (range 0.0 == empty, to 1.0 == full ; default threshold: 0.98)

The MySQL host metrics stand out in that the throttler only reads them sporadically. It would be too expensive to call a sequence of two gRPC (one to tablet manager, then to mysql daemon) multiple times a second. Certainly for a metric such as mysqld-datadir-used-ratio, but also for mysqld-loadavg. The throttler only actually reads those metrics once per 10sec, and caches the results for subsequent checks.

Doc PR: vitessio/website#1878

Related Issue(s)

• Feature Request: MySQL load average & used disk space metrics in multi-metric throttler #16822
• RFC: tablet throttler multi-metrics #15624

Checklist

• "Backport to:" labels have been added if this change should be back-ported to release branches
• If this change is to be back-ported to previous releases, a justification is included in the PR description
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on CI?
• Documentation was added or is not required

Deployment Notes

[codecov]

Codecov Report

Attention: Patch coverage is 41.21622% with 87 lines in your changes missing coverage. Please review.

Project coverage is 67.38%. Comparing base (ffaeba0) to head (8adc7eb).
Report is 33 commits behind head on main.

Files with missing lines	Patch %	Lines
...t/tabletserver/throttle/base/self_metric_mysqld.go	32.85%	47 Missing ⚠️
.../tabletserver/throttle/base/self_metric_loadavg.go	0.00%	16 Missing ⚠️
go/vt/vttablet/tabletserver/throttle/throttler.go	48.27%	15 Missing ⚠️
go/osutil/loadavg_linux.go	66.66%	2 Missing ⚠️
...etserver/throttle/base/self_metric_custom_query.go	0.00%	2 Missing ⚠️
...blet/tabletserver/throttle/base/self_metric_lag.go	0.00%	2 Missing ⚠️
...erver/throttle/base/self_metric_threads_running.go	0.00%	2 Missing ⚠️
.../tabletserver/throttle/base/self_metric_default.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #16904      +/-   ##
==========================================
- Coverage   67.43%   67.38%   -0.06%     
==========================================
  Files        1571     1573       +2     
  Lines      252249   252980     +731     
==========================================
+ Hits       170098   170464     +366     
- Misses      82151    82516     +365     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mattlord]

I had some nits and questions. Can we also please add an e2e test (sorry if I'm missing it). The testing feels a bit light (and is reflected in the codecov results) and this seems like something we'd want to test through the RPC chain (to a fake and/or real mysqlctld).

[mattlord]

How do these changes relate to the PR? Is it that we're getting rid of the old loadavg OS metric now that we have the new mysql-loadavg metric and so we wanted to test another MySQL process metric here?

Is there any reason that we're not using any of the new MySQL host metrics in the e2e tests? Or am I missing something here?

[shlomi-noach]

How do these changes relate to the PR?

See #16904 (comment): it's about tests failing on MacOS machines.

Is there any reason that we're not using any of the new MySQL host metrics in the e2e tests? Or am I missing something here?

You're not missing anything here. Let me see what we can do for e2e tests.

[shlomi-noach]

Added endtoend tests that verify the full MysqlHostMetrics path don to mysqld, and checking response values. We check the existence of both new mysqld-* values, and we specifically check that mysqld-datadir-used-ratio is non-zero (as can be expected in local hosts and in CI). We do not check the value of mysqld-loadavg as that could be arbitrarily low and even appear to be zero, as well as because we do not implement it in MacOS.

[mattlord]

Having the type and variable exported makes this interface feel off to me. Since anyone can access the metric and this internal (?) variable as things are now.

[shlomi-noach]

Yet no one holds an instance of MysqldLoadAvgSelfMetric. The only instance in existence is:

var _ SelfMetric = registerSelfMetric(&MysqldLoadAvgSelfMetric{})

So yes, anyone can create their own new instance, but that cannot affect the existing instances.

[mattlord]

FWIW, we should be able to drain the ticker's channel on cancel if it's not empty. That would be ideal, unless I'm missing something here.

[shlomi-noach]

The race condition is a bit more nuanced:

		go func() {
			defer mysqlHostMetricsRateLimiter.Store(nil)
			defer rateLimiter.Stop()
			<-ctx.Done()
		}()

so when you hit cancel(), the rate limiter gets unblocked from <-ctx.Done(), but then it's still possible to get a new ticker event before defer rateLimiter.Stop() gets issued.

[shlomi-noach]

Docs: vitessio/website#1878

[mattlord]

Thanks! ❤️

[shlomi-noach]

Now supporting loadavg values on darwin. I've added go/osutil package that knows how to handle both systems.

This is used in both tablet's loadavg as well as in Mysqld host metrics.