tailscale: drop this package #25062
tailscale: drop this package #25062
Conversation
23.05 is using go 1.21 if I remember correctly which is out of support since a month. Go only support the latest version and the version before that.
This is extremely unlikely. Go has a very hard stability guarantee and doesn't do any breaking changes in the language until 2.0. Any such change is probably a bug and should be reported upstream.
You could totally backport it as a new package like many other distros do to support this use case.
OpenWrt should except packages that talk to online services like other distros do from that rule to be a base users can rely on to build their systems and without requiring custom repositories.
We should also take a look at other distros. Debian is pretty old and not the platform where new ideas and innovations are born.
I don't think it is clear at all and it is very much possible. Go is very easy on dependencies and unlike other ecosystems like C or C++ it is very doable to just backport the new compiler and build tailscale with it. There are very likely no other dependencies required to be changed.
🤚🏼 but I am not going to fight months over it being re-added. I could also just compile the binary statically and let it fall out of my deployment system and copy it over. It might be huge but I have a few MBs free on my router and it would be the least painful solution. |
|
@SuperSandro2000 , I am glad that you tried to convince us that tailscale should stay here, but only what I can see are words and not actually providing any help. As always pull requests are welcomed. :-)
Really? In the past, it happened multiple times.
Release branches should receive only security and bug fixes as you can see here: https://github.com/openwrt/packages/blob/72c8977b3b4b3b8c905bd53db7557cf6bc8963db/CONTRIBUTING.md#release-branches |
Golang's release notes seem to disagree with you. For example, recent Go 1.23 dropped
Debian stands for "stable", not "old". We are not another Arch Linux for routers.
While tailscale only supports the latest version. |
So far I didn't need to become to active since I am always upgrading to the latest release within a few weeks of release and being a few releases behind didn't bother me so far. In the past I build 1.62.1 one from the head branch to get newer features to fix my deployment. Right now I half an eye on the package and module in NixOS but that gets usually updated within a few days without my involvement, so I can usually stay in the background. I beta tested headscale 0.23.0 and provided several times feedback and did some PRs. I am using https://github.com/astro/nix-openwrt-imagebuilder to build my OpenWrt images and recently discovered that I couldn't build the last point release because tailscale was missing. Right now I built #24986 and will deploy that later to my router and I am pretty confident that it will continue to work. I will certainly in the future have an eye on new releases and create PRs if it becomes necessary.
Nebula also had a similar problem with newer go compilers. For the others I looked at them and it seems to be that all of them where related to dependency management which go is notoriously bad, not necessarily the language breaking.
I mean if the rules are to restrictive and actively hinder you on productive changes and fixing things then the rules should probably change. I would suggest to add an exception for packages that talk to online services and add an ambiguous clause if there are justified reasons like e.g. another packages requires a newer compiler version. Putting a new package next to the existing one which is only used at build time shouldn't cause any issues, or does it?
The stability guarantee covers the language itself not breaking, not necessarily covering the compiler steps but the change seemed to be handleable (https://github.com/openwrt/packages/pull/24992/files).
That's new to me. I am managing 30+ go projects at $work and the biggest breakage I can remember in the last year was when the pushed the wrong docker image to docker hub.
I kinda need to disagree. I actively used Debian for 3+ years and was pretty deep into it and with every update everything flowed through my hands like sands. I am using NixOS now since 2+ years and it solved like almost all my problems I had with Debian.
The software end consumers receive are backwards compatible but the compiler is not part of that story since usually end users are on x86_64 or aarch64 machines and can use the prebuild binaries. Since OpenWrt routers are usually on some more niche arches we need to compile the packages ourselves. It seems that the issues are not really tied to tailscale directly but more to the go ecosystem in general. It would be pretty sad if any go package in the future could be just removed on the discretion of a maintainer. I also actively use rclone which would be pretty sad to be dropped. |
We are talking about the code, not raw binaries. Tailscale provides prebuilt binaries for all go supported arches except loong64 and ppc64. As your opinion, you may just download them yourself.
Nothing to be sad as nobody really cares it. I cannot see anyone come up to resolve the issue, but just complaint.
Rclone is totally different story. The code has very good compatibility, the latest version still works well with Go 1.20. The developers are also active and ever helped us to build the package. Out of curious, what stops NixOS 24.05 updating tailscale to the latest version? |
Every single version of tailscale requires to have up-to-date version of Golang, despite that using our Golang version is supported in upstream. While updating Golang to the latest version, it might break other Go packages due to using unsupported version, yet. We can not also backport recent versions of Golang package to the stable branches as it potentially can break other packages and due to policy of stable branches, there should be backported only security fixes. Based on the upstream issue[^1] and also in Debian's bug report [^2], it is clear that we can not keep up with the upstream. Tailscale can be re-added to this repository, when someone from the community or the tailscale developers steps in and provide support and help while maintaining tailscale in OpenWrt. [^1]: tailscale/tailscale#7847 [^2]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=972439#19 Fixes: openwrt#24741 Signed-off-by: Josef Schlehofer <[email protected]>
|
Rebased, merge conflict resolved. |
But why do I have a distro with a package manager when I need more tooling to manage the entire story because the distro refused to track all programs I need?
The issue could be very easily resolved, if there would be a package for go 1.21 and 1.23 on stable but due to the restrictive rules that is not possible. We could also track a probably 1000 lines patch to remove the new features again but IMO that's not a good solution. We could also just remove the package from stable and keep it on edge only. It is a static binary and installing it on stable works. Eg. Debian has quite a few of those packages like Firefox.
Only really that someone creates a PR for it. Requiring a newer go compiler is a none issue and we've done that in the last. |
It's painful, but it's the only way to keep the package in stable with current rules I think. Maybe backport security updates only?
I'm okay with it.
Not sure if it's acceptable, there's no precedent. Needs discussion. (I guess no) |
|
Hi, Tailscale employee here. I assume the problem here is that it's against OpenWRT policies to include more than one version of Go in the tree? (many distros have such a "single version policy", for various reasons) Our release versions of Tailscale always depend on the latest versions of upstream stable Go, and that's because we can track and contribute to upstream Go actively. (A number of us used to be on the Go team; I was on the core Go team for about 10 years) Because we don't like waiting 6 months between Go releases, we also maintain our own Go fork (at https://github.com/tailscale/go) where we cherry-pick fixes and features we'd like immediately, without waiting for up to 6 months. But we know that would be a non-starter for most of the open source community so instead we only depend on that for our internal Go projects and our open source parts are supported with either One thing we could do is ship a Tailscale source tarball that includes our I doubt that'd be an acceptable answer here, but on the chance it is, we're willing to do that work if it helps. |
Cc @neheb any idea about this? |
|
I'm out of the loop. From looking at #24741 , it seems old versions of tailscale are unusable because of Security Update messages. The toolchain thing is interesting. I am however curious what the size impact of such a thing would be and if it would even matter to users. |
It should be zero disk or memory difference, as it produces a static binary with either toolchain. The main difference would be build time overhead (a couple minutes on a fast machine) and more source code. |
|
Chiming in as a user of tailscale on OpenWRT, while I can appreciate there are some difficulties to resolve in the build process, please do consider that there are active users of this package - the alternative would seem to be random other package sources that may not be trustworthy. |
Or implement another solution. |
|
Well, there is still plain wireguard, which you can use and it is even included in the Linux kernel. Tailscale is based on top of wireguard. |
We don't like pizza but we like pizza? Your general approach is commendable for tailscale - but this would be too much churn in this repo. It's master anyway, which paradoxically is what one would need to run to stay secure. I'm in favour of dropping, if this alleviates maintenance burden. I love Go, but here's the rub: the mono-binaries it produces are monstrously big for the majority of the small-memory/flash footprint devices that the binaries are intended to help. One or two and your router is full :/ BUT...... there's a REALLY OBVIOUS FIX @bradfitz: Tailscale hosts the repo for the binaries for current [22, 23, main] releases. Then there's zero churn here. Although users would need to manually add the repo. There are a few independent repos. Perhaps there you can refine a workflow and it would demonstrate what's necessary for other maintainers. |
|
Each branch of go is only supported for ~1 year, so we must update (https://go.dev/doc/devel/release#policy). In OpenWRT we are now upgrading the kernel constantly, so constantly risking regression affecting everything, but we can't accept risking regression just for golang packages ? Not everyone wants Debian style stability, and we must update for security, so why not just upgrade golang in stable branches ? |
|
Before talking about updating/backporting Go toolchain, I wonder if anyone wants to take over the go maintainer ship? @jefferyto seems no longer active here. |
So this sounds like the only "solution" for now to keep the tailscale package in stable, though it will take more spaces and build time.
The compatibility issues are already discussed above.
Isn't it the reason we only use LTS kernel? Or do you mean we "constantly" do major kernel upgrade on stable branches? Then we should expect kernel 6.6 or even 6.12 on openwrt-23.05. |
A bit overkill. Why don't Tailscale just host the repo? Like this one https://lanrat.github.io/openwrt-tailscale-repo/ |
|
@1715173329 I'm comparing golang major updates with kernel minor updates because both can come with regression (I recentely spent maybe a day tracking down why sriov-cni was broken after a kernel update and it was because of a security fix backported to all supported version https://lore.kernel.org/linux-cve-announce/2024053015-CVE-2024-36017-bf96@gregkh/T/) I looked at the 3 exemples from @BKPepe:
Restricting linkname will avoid breakage on golang upgrades in the future, so progress in the right direction for maintainers. The only real issue is not having anyone maintaining go, everything else is just normal maintenance work IMO. |
The golang minor updates also come with "regression", e.g. moby/moby#45935.
So there's a breakage now. There's always a reason, but just do not claim how great it is at backward compatibility. |
Probably, but as a user installing it from package manager is definitely more convenient. |
|
Package which adds tailscales repo? CentOS derivatives do this to install EPEL. |
|
(I don't have nearly as much free time as I did before but I still plan on coming back at some point 😅 I appreciate all the help from everyone with my packages so far, and if anyone would like to be co-maintainer I would be happy with that.) FWIW I agree with @BKPepe, we should drop this package and let upstream build and distribute their own packages. On updating Go compiler/packages in release branches in general: I'm not in favour of updates beyond bug/security fixes because this is a slippery slope. If this is allowed then users will ask for updates for other packages (they don't care about differences like Go vs C/C++/etc). A general rule or policy is easier to communicate/understand/enforce than one with exceptions. And (just to state/repeat the obvious) updating the release branches like this would require extra effort and we don't have enough maintainers/contributors already. Users who want the latest software can run main/snapshots or they can build their own packages. (I'm reasonably certain all of the necessary infrastructure to build packages and host a custom feed is available on GitHub, just modify the CI to save build artifacts to a gh-pages branch.) Things we can do (all of which also require extra effort):
Though I suppose users who don't want to lift a finger will also dislike having to enable an additional feed or build their own packages 🤷 On Tailscale in particular: Upstream has already stated their preference for hosting their own repositories in tailscale/tailscale#7847, specifically tailscale/tailscale#7847 (comment). Removing this package will help upstream focus their efforts on delivering the best experience for their users. |
|
@1715173329 linkname is a hack to access private functions, so if you use it you need to expect breakage at any time even with a minor version, as too many people are using it to access golang internals they will block it by default, with possibly to reenable it. Everything we upgrade might contain regression, golang is great at backward compatibility, but it's not perfect, upgrades do sometimes contain regression, and they sometimes need to break things for security, same as the kernel. |
Regardless of the reasons, breakage just objectively exists.
I'm afraid if you listen to me golang will explode 💥. |
Maintainer: @ja-pa
Every single version of tailscale requires to have up-to-date version of Golang, despite that using our Golang version is supported in upstream. While updating Golang to the latest version, it might break other Go packages due to using unsupported version, yet.
We can not also backport recent versions of Golang package to the stable branches as it potentially can break other packages and due to policy of stable branches, there should be backported only security fixes.
Based on the upstream issue1 and also in Debian's bug report 2, it is clear that we can not keep up with the upstream. Tailscale can be re-added to this repository, when someone from the community or the tailscale developers steps in and provide support and help while maintaining tailscale in OpenWrt.
Fixes: #24741
Footnotes
https://github.com/tailscale/tailscale/issues/7847 ↩
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=972439#19 ↩