Week 47 routine

[kiwixbot]

Infra

• Check nodes free space

df -h / && df -h /data

• Nodes and worker system upgrades

apt update && apt upgrade

• Ensure all borg repositories are being updated
• Check Pod errors

k get pods -A -o wide|grep Error

• Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

• Upgrade k8s if applicable

Stats

• Ensure download.kiwix.org stats are being recorded
• Check whether matomo should be upgraded

Grafana

• Alert list is normal
• Zimfarm dashboard is normal (look after last week data)
• Mirrorbrain dashboard is normal
• There is no abnormal behaviors on cluster resources consumption

Projects

• UptimeRobot has no alert
• youzim.it backlog is reasonable
• No systematic failure in tasks.
• PRs awaiting your review

Security

• Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

@rgaudin your turn ;) (but I can take it if you need time)

[rgaudin]

Filesystem      Size  Used Avail Use% Mounted on
bastion        37G  7.6G   28G  22% /
stats        233G   91G  131G  41% /
services        456G  200G  233G  47% /
storage         33T   19T   13T  61% /

OK then you can do it. Can you please upgrade kiwix-serve as well in library.kiwix.org? There are two. There are comments about varnish ; let me know if you want to do it together. We'll see if it requires more documentation that those comments

[benoit74]

zimit

• 5x Timeout after 90000ms on seed page
• 2x NameResolutionError (1x onion website 🙈, 1x real issue in URL)
• 1x Connection refused (web.archive.org)
• 3x Description is too long (before we released Zimfarm fix ...)
• 1x puppeteer INTERNAL ERROR: missing context with id = 20
• 2x 403 Client Error
• 1x weird characters issue: Zimit WARC files of archives.nyphil.org are full of unrecognized characters openzim/zimit#253
• 1x Connection timeout (looks like IP whitelisting : www.textfiles.com)

[benoit74]

Oups, I almost forgot kiwix-serve update ...

[benoit74]

Upgrade kiwix-serve to 3.6.0 + renamed library-demo to library-data to try (🤣) to avoid confusions ; varnish restarted to reload its configuration, no need to purge cache indeed