Log level and log retention ought to be operator-configurable

[barryprice]

Steps to reproduce

1. Deploy a busy unit on a disk-limited flavor - say, 50GB of disk space for a VM running a 5GB database.
2. Throw enough traffic at the application driven by that database for a lot of logs to be written
3. Watch the disk fill up

Expected behavior

The operator should be able to prevent the disk from filling up by reducing either the log level, or the log retention, or a combination of both. Enabling compression for older logs may also be something to (re)consider, at least as a non-default option.

Actual behavior

If the unit is sufficiently busy (say, a content database for a popular data-driven website), the logs will quickly fill up the disk.

We've seen numbers up to the region of 1GB/hr of logging against a 5GB DB, so best part of 25GB of logging per day - with 7 days retention hard-coded, that's potentially 175GB or so of logs on disk at any time, about 35x the size of the database itself.

This requirement may not become clear to less attentive operators until up to a week after deployment time, or may creep up if the application using the database gradually sees more and more traffic over time.

Versions

Operating system: Ubuntu 22.04.4 LTS

Juju CLI: 2.9.50-ubuntu-amd64

Juju agent: 2.9.49

Charm revision: 240

LXD: N/A

Log output

Juju debug log: N/A

Additional context

N/A

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-5656.

This message was autogenerated

[hloeung]

In particular, the general logs:

ubuntu@juju-730dff-prod-admin-insights-ubuntu-db-27:~$ sudo ls -larth /var/snap/charmed-mysql/common/var/log/mysql/archive_general/ -hh
total 33G
-rw-r----- 1 snap_daemon root         33G Dec  8 02:48 general.log-20241206_2212

We've had to manually FLUSH GENERAL LOGS then zero it out to reclaim some space.