add support for etcd backups with s3 storage

[Kennybll]

K3S has built in support for uploading snapshots of the embedded etcd database to S3. This PR adds the settings to enable that to the configuration file.

If the s3 settings are not included it will default to not being enabled, to ensure backward compatibility.

I also added a check to make sure it is only enabled when using the etcd datastore mode.

It may also work with other S3 compatible providers. I cannot give a list, however I tested it with Backblaze B2 and it worked.

An example when uploading to AWS S3 in the us-east-1 region:

datastore:
  mode: etcd
  s3:
    enabled: true
    access_key: EXAMPLE_ACCESS_KEY
    secret_key: EXAMPLE_SECRET_KEY
    bucket: EXAMPLE_BUCKET

[vitobotta]

Wow, you made me VERY happy with this PR! It was on my list, so I appreciate you took the time to implement it.

I requested some changes if you don't mind, but these are mostly minor things and quick changes as you have done a great job really.

Thanks a lot!

[Kennybll]

I realize it was outside the scope of this PR, but after the request to refactor the structor of the cluster_config.yaml options I figured I would add some of the additional back up options.

The only backup option I could not figure out was schedule_cron. I could not get it to work correctly for the life of me. Maybe it's the way the string was getting parsed with all of the quotes. That can be saved for a future PR. Note to self, I was looking at creating a /etc/rancher/k3s/config.yaml file with the snapshot-cron value, instead of passing it as an arg to the k3s binary.

[vitobotta]

A few more comments, it's looking good!!! 👍

[vitobotta]

I realize it was outside the scope of this PR, but after the request to refactor the structor of the cluster_config.yaml options I figured I would add some of the additional back up options.

The only backup option I could not figure out was schedule_cron. I could not get it to work correctly for the life of me. Maybe it's the way the string was getting parsed with all of the quotes. That can be saved for a future PR. Note to self, I was looking at creating a /etc/rancher/k3s/config.yaml file with the snapshot-cron value, instead of passing it as an arg to the k3s binary.

Can you share more details on the issue you had with that setting? It should be a regular string if it's in cron format, isn't that the case?

[Kennybll]

The only backup option I could not figure out was schedule_cron. I could not get it to work correctly for the life of me. Maybe it's the way the string was getting parsed with all of the quotes. That can be saved for a future PR. Note to self, I was looking at creating a /etc/rancher/k3s/config.yaml file with the snapshot-cron value, instead of passing it as an arg to the k3s binary.

Can you share more details on the issue you had with that setting? It should be a regular string if it's in cron format, isn't that the case?

I tried every variation I could think of for a cli flag. None of them worked. I even tried manually setting it in master_install_script.sh in case it was the YAML parser being weird. If you are able to get it to work that would be awesome as it would be cleaner.
--etcd-snapshot-schedule-cron="* * * * *"
--etcd-snapshot-schedule-cron='* * * * *'
--etcd-snapshot-schedule-cron=* * * * *
--etcd-snapshot-schedule-cron * * * * *

Based on this comment k3s-io/k3s#5983 (comment), I was able to get it to work using a config file. So if snapshot-cron is set, the master_install_script.sh will create a file /etc/rancher/k3s/config.yaml.d/hetzner-k3s.yaml with the flag. If the option is removed, then it will delete that config file if it exists.

Whenever a cli flag changes the k3s service will automatically restart when rerunning the master_install_script.sh. However, if a config file changes, the service will not restart. I created an issue in the k3s repo k3s-io/k3s#11180. But in the meantime we would have to manually restart the k3s service, just in case there are changes in a config file, even if there are none.

[vitobotta]

I see, good idea to open an issue there. Did you also try normal escaping with \"?

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[Kennybll]

Yes, I tried escaping with \" and also \\\" just in case

[vitobotta]

Yes, I tried escaping with " and also \" just in case

I see. Thanks for the clarification