Planned updates

[Grimler91]

In #1 (comment) @agnostic-apollo suggested some additional improvements. Lets track it in this issue so it is not forgotten:

Another thing I want implemented is a blacklisted mirrors list file that is hosted on fosshost and as fallback on github as well in case its inaccessible, and file possibly signed with our key. This should contain all the mirrors that must not be used or whose domains have been freed or url sub path changed, like bintray, packages.termux.org, kcubeterm.me, recent one, etc. This file should likely automatically be downloaded on termux app start and incrementally as well to $PREFIX/etc/apt/blacklisted-mirrors.list file. The mirrors in these files should be checked by termux-change-repo and excluded from showing and pkg should exclude it too from mirrors group and switch to default mirror if none remain or single mirror is blacklisted.

An additional default mirror site file should also be hosted, that is used by the app to replace any blacklisted mirror from existing apt sources list files so that if user uses apt instead of pkg, then blacklisted mirrors don't get used. The default mirror should also be used by pkg script as default.

With the above, the blacklisted mirrors will be independent of whether users updates termux-tools or the app or has old bootstrap and we hopefully won't get any (or lower) broken mirror issues anymore. Hopefully, fosshost and github will be accessible in all countries. There is of course an ethical issue of pinging either fossthost or github continuously that could be used to track termux users, so users should explicitly opt in with a prompt. The user should also be able to disable this in termux app settings.

We could also download the files in pkg script instead so that app doesn't need to and it doesn't ping in background and only when users wants to update, but then users that may be running apt directly would still remain affected, unless we change relevant apt binaries to a wrapper scripts or patch them to run a script at start, depending on command install/update/upgrade/search.

[Grimler91]

• have pkg and termux-change-repo download a blacklist (if a cached one is outdated). Verify gpg signature, or at least hashsum. Refuse to use mirrors in the blacklist
• make this options opt-in, add possibility to change the option in termux.properties or app settings (in app or termux-tools) to disable this check
• extend termux docs to explain how mirror handling/package management works
• convert termux.md.1 to a texdoc document, to get rid of pandoc dependency

[agnostic-apollo]

To solve termux/termux-packages#13977, in check_mirror, instead of just checking if Release file can be connected to, we can check if the Date field in the file (currently Date: Tue, 20 Dec 2022 07:20:29 UTC) is older than a day compared to our primary mirror(s), assuming they can be connected to. We could have a primary mirror per region as well in case hetzner/cloudflare is not accessible in specific reasons. All mirrors that haven't been synced within a day should be excluded from mirror selection. The termux-change-repo can mark such mirrors as outdated but still allow users to select them in case they want to use old package versions.

termux-tools/scripts/pkg.in

Lines 57 to 67 in 15fc3b6

	check_mirror() {
	local mirror="${1%/}"
	local timeout="${2-5}"
	timeout "$((timeout + 1))" curl \
	--head \
	--fail \
	--connect-timeout "$timeout" \
	--location \
	--user-agent "Termux-PKG/2.0 mirror-checker (termux-tools @PACKAGE_VERSION@) Termux (@TERMUX_APP_PACKAGE@; install-prefix:@TERMUX_PREFIX@)" \
	"$mirror/dists/stable/Release" >/dev/null 2>&1