telemetry in the Go toolchain #58409
Replies: 95 comments 394 replies
-
|
Should be off by default. There is no reason for a development chain to have any kind of telemetry on by default. One reason it should be off by default would be cases such as Jenkins farms, Bazel use, and other locations that arnt your regular development environment. |
Beta Was this translation helpful? Give feedback.
-
|
I think that lowering the barrier to opt-in and share data is a better solution than increasing the barrier to opt out. People who would decline telemetry at a prompt are people who don't want telemetry, making the life of those people harder is really bad whether it's intentional or not. |
Beta Was this translation helpful? Give feedback.
This comment was marked as spam.
This comment was marked as spam.
-
|
As soon as there is just one identifiable data collected, opt-out becomes completely illegal WRT GDPR, which applies to any entity processing a European user data regardless where it is collected. Add that the privacy shield is illegal now in Europe, you better watch out for the huge fine if that ships with opt-out to any European user. This is a legal issue, your personal opinion on the goods and bads of opt-in vs opt-out is irrelevant. |
Beta Was this translation helpful? Give feedback.
-
|
Transparent and "On by default" do not go together. Naming It transparent is like the 1984 moto "war is peace". “There’s no point in acting surprised about it. All the planning charts and demolition orders have been on display at your local planning department in Alpha Centauri for 50 of your Earth years, so you’ve had plenty of time to lodge any formal complaint and it’s far too late to start making a fuss about it now.... I’m sorry, but if you can’t be bothered to take an interest in local affairs, that’s your own lookout. "
|
Beta Was this translation helpful? Give feedback.
-
Are there any concrete percentages available from projects who did this before? You say that about 16,000 samples are needed. There are more than two million Go users. So you would need about 1% of users to opt-in. Is this really that difficult to achieve?
One does not have to understand the system well to answer a yes/no question in the installer or before starting a download from the Go website. This leaves those who install Go via the package manager of their Linux distribution. However, Linux users are more likely to be power users who understand how to set an environment variable to opt-in. And many (most?) Linux distributions would disable telemetry by default anyway.
It's in the hands of the Go project to not use it as a justification for that. Furthermore, it's a question of how attractive you make it to opt-in. More users will opt-in if you can convince them they will benefit. It's a communication and marketing task, just as you need to promote participation in the annual survey.
Those who opt-in probably don't care about the collection frequency as long as it doesn't bog down their systems. |
Beta Was this translation helpful? Give feedback.
-
|
@rsc I think this is an interesting idea. As an open-source developer (OpenTelemetry) I agree that we need an ethical way to collect usage telemetry about our own products. I will run this idea inside OpenTelemetry project to see what our community thinks. If you are interested in using an existing metric reporting library it may be helpful to take a look at OpenTelemetry-Go. It supports histograms and can send collected metrics to the backend using a standard OTLP protocol (Protobuf or JSON payloads over HTTP). |
Beta Was this translation helpful? Give feedback.
-
|
It would be interesting if there's an opportunity to collaborate here, although my understanding of OpenTelemetry is that it's architected around an event stream rather than the very coarse week-long counter aggregation in the design. Avoiding an event stream is one of the important ways transparent telemetry reduces its privacy impact. |
Beta Was this translation helpful? Give feedback.
-
I think you are right. We can likely make OpenTelemetry Go SDK do what you need but I think it is not worth the effort. Rolling out your own based on the design you posted will likely be simpler.
There probably is. I wonder if your proposal can be generalized to allow any open-source project (and not just to the Go toolchain) to ethically collect usage telemetry. OpenTelemetry may be a good home for this. |
Beta Was this translation helpful? Give feedback.
-
|
Even if not using the official OpenTelemetry, it would be nice to respect some the environment variables that people have configured for other tools that use OpenTelemetry, in terms of opt-in / opt-out. |
Beta Was this translation helpful? Give feedback.
-
|
That's quite a few environment variables. Which specifically are you suggesting it look for, and how should it interpret them? I'm not sure I understand why it would make sense to look for OpenTelemetry env variables without using OpenTelemetry. Do other systems do that? |
Beta Was this translation helpful? Give feedback.
-
|
Despite sharing the name "telemetry", I don't think this proposal intersects too much with how the OpenTelemetry SDKs or config (env vars) work at this point in time. |
Beta Was this translation helpful? Give feedback.
-
|
What would prevent users from sending garbage to the telemetry server? |
Beta Was this translation helpful? Give feedback.
-
I can imagine this bit being misunderstood or read out of context. This targeted telemetry would still be under the constraints of the design: only infrequent sampling of event counters with no sensitive information. I imagine that one example of targetting might be something like: many users on Windows say that their Go builds are slow, so we want to get more counters for build cache hits and misses from Windows users. Perhaps @rsc has other examples to help illustrate. |
Beta Was this translation helpful? Give feedback.
-
|
@svrnm and @lthibault correctly understand how distributed systems work; in particular, they understand that public data-collecting endpoints must accept garbage, and also that the public likes to put trash into such endpoints. @rsc: The point of the question is not to earn a good answer, but to remind you that the question represents a problem, a fundamental obstacle to your plan and desires. I recommend modifying your desires, rather than trying to administer computers which you do not own. Also, when you say "if a particular statistic is wrong", you are presuming that you can tell whether your data represents a biased sample, which implies that you already have a prior distribution and do not actually need this data collection; or you are trusting priors which do not actually correspond to any real-world data. The attack that sneaks through your plan is called "boiling the frog" or 'salami-slicing". An attacker would let you set up your system as planned and operate it for several months. During that time, the attacker notes the same trends that you note, including anything from likely IP ranges to expected OS and platform. Then, the attacker slowly starts creating fake reporters, using a variety of cloud services for IP diversity, and programming the reporters to reproduce your existing data distribution through random sampling. After the attacker is confident that they haven't been noticed, they start to systematically bias their reports:
I read through all four of your posts (the three on telemetry and the one on sampling) and somehow you never used the word "bias". However, even mere opt-out/opt-in creates sampling bias! Please, at the very least, make this consideration part of your design; you should show how your system can fail, not just how it can succeed. |
Beta Was this translation helpful? Give feedback.
-
Also, this isn't quite the scenario we were discussing. We were instead discussing the scenario in which users tried to subvert telemetry. In our example, Go is working just fine. The difference is important because it goes to show how opt-out telemetry inevitably produces a user-hostile game of cat-and-mouse. |
Beta Was this translation helpful? Give feedback.
-
|
@MostAwesomeDude Taking your argument to its logical conclusion, it seems like the most effective defense against such an attack would be some kind of proof-of-work system that would make it difficult for an attacker to overwhelm the honest participants. The most obvious approch would be to give up on sampling and collect from 100% of the systems, perhaps every day. In that system the "work" is the network traffic and reporting behavior, making it far more costly for attackers to overwhelm the reporting systems. Of course, that would be a very different system, and not one I want to build or use. My point is only that if you focus exclusively on attack mitigation strategies, you end up in a place that most people commenting here probably wouldn't like at all. |
Beta Was this translation helpful? Give feedback.
-
|
@rsc I don't think this is taking @MostAwesomeDude's argument to its logical conclusion at all. It is rather missing the point by proposing a technical solution that ratchets up the user-hostility another notch; users must now perform work unless they opt out! 😄 To be clear, I get that you are making this proposal in jest, but it goes to show that you've missed the point. |
Beta Was this translation helpful? Give feedback.
-
|
I don't think that an opt-out mechanism is compliant with GDPR. Was the proposal reviewed by any Compliance experts? Personally, I don't like the idea of automatic collection of telemetry pushed on me. Isn't Google tracking enough information about me already? |
Beta Was this translation helpful? Give feedback.
-
|
Also you are moving from an ad hominem to a false comparison naming once again my employer. It becomes hard to believe that this is a simple 'miscommunication'. I gave you facts, please respond on them without attacking or threatening me. |
Beta Was this translation helpful? Give feedback.
-
|
@padawan I was trying to provide a concrete example that you could go check on. Saying that consent is the only legal way to collect IP addresses under GDPR is plain wrong. Implying that the only exception is mandated traffic logging is also wrong. Legitimate interest can be a legal basis to collect IP addresses, as has been confirmed by the courts already for the case of IP collection for security reasons, or to fight piracy. @sulewicz I don't know if other legal bases could apply to this case. I'm not saying that they could. I was only saying that this legal determination is certainly not as trivial as has been suggested by multiple people on this thread, and that "GDPR means that you need consent!" is a gross simplification: it's at best not at all clear (I'm not aware of a court decision for a telemetry use case like this proposal), and at worst completely wrong. |
Beta Was this translation helpful? Give feedback.
-
|
@TedTed while you are right that IPs can be collected for legitimate interest without an explicit consent, For a case regarding IPs collected without the user consent see the Case of Google Fonts CDN where it was judged that using the CDN in any application requires user consent becuase it provides the IP of the user to the the CDN. |
Beta Was this translation helpful? Give feedback.
-
I'm not familiar with the GDPR article or DPA interpretation that makes such claims, can you tell more? Article 6.4 says that there are a number of points to take into consideration when making a legitimate interest determination, and this determination is far more subtle and nuanced than "is it part of the core functionality". Data minimization, risks to data subjects, use of technical safeguards, and data sensitivity are all taken into account in this determination. Recital 47 explicitly says that fraud detection or direct marketing can be used to regarded as legitimate interest, and those don't seem to me like "part of the core functionality". Of course neither is directly relevant to this telemetry proposal, but they show that this determination is complex and subtle. Again, I'm not saying "it would be perfectly fine to use legitimate interest for this use case". This would be wrong. I'm saying that it's complicated, that law is messy, and that absolute statements about compliance are not accurate nor helpful. Thanks for bringing up the Google Fonts ruling. I agree that it's very relevant, especially since the decision was about a website disclosing IP addresses to Google Fonts, regardless of what Google actually did with these IP addresses. Elsewhere in this thread, @ianlancetaylor says:
As the Google Fonts ruling suggests, the compliance story is likely not as simple as that — even without logging IP addresses, the system still relies on processing those in order to receive telemetry reports. |
Beta Was this translation helpful? Give feedback.
-
Yes but those are considered legitimate interest only when you "need to process personal data in order to carry out tasks related to your business activities". The compiler's and build tool's activities require nothing of the sort. Fraud detection can not apply because without phoning home there is no Fraud to protect From. I bring the very hard to interpret issue because here in Europe all consulting lawyers are very conservative about these things and even though Google may have the bandwidth to litigate such issues, the workplaces who use go and can't, for any reason, be 100% sure that will be opted out in every case, will need to disclose it and gather consent "just to be safe". |
Beta Was this translation helpful? Give feedback.
-
|
I don't feel like I have a stake in this debate, but I want to provide a "steel" man for the counter position. I think the principal issue for people who are averse to default-opt-in telemetry is that the policy of what data is sent over that connection is mutable. Even if the debates over which telemetry data to publish is public, once we establish the pattern of telemetry, the policy of what is sent through the pipes is subject to change. Put another way, the logic of "never telemetry" goes something like: if we never have a network call to begin with, we don't have to know the latest version of a policy that's wholly irrelevant to our day-to-day work, but essential to our personal privacy. I have full faith in the current humans of the Go team to write policy and compose processes that are fair and respect peoples' privacy. However, it's important to recognize that this move opens a hitherto nonexistent door for less-than-scrupulous data collection, and that future members of the Go team (or future versions of ourselves) might not hold privacy to be so sacrosanct. In summary, there's a trade-off to be had here. The proposed metrics to collect would be invaluable to ongoing development of the toolchain. And, to be clear, the technical solution proposed appears to be a sound basis for preserving privacy. The quandary is whether this value is worth creating space to (eventually/perhaps unwittingly) emit private information? I think one can argue either side rationally, but it has to be said that making the system unobtrusively opt-in would largely resolve the ethical concerns by introducing a mechanism for informed consent. Even if a default-opt-out solution provides 1/10th the value of opt-in telemetry, it may be worth considering for that reason alone. |
Beta Was this translation helpful? Give feedback.
-
I don't understand this point @rsc. It seems like you have re-articulated the main argument for "no telemetry, ever". I don't want to read into intent too much, but I am getting the distinct impression that the decision to ship telemetry has already been made, and that we are being asked to rubber stamp it. I sincerely hope that is not the case. |
Beta Was this translation helpful? Give feedback.
-
Yep! I'd say this is an accurate summary.
Unfortunately, the "no man's land" between "good" and "bad" telemetry is large and its boundaries are fuzzy. I believe the open source community would be able to effectively stand in the way of a direct, unequivocal step from "good" to "bad". However, I think the primary fear is that the OSS community is not an effective guard rail against a dozen or so tiny steps in a "bad" direction, especially when those steps are each correlated with improving the utility of the collected data. There will be a strong incentive for well-meaning people to propose steps in a "bad" direction. In a default-opt-out system, the OSS community would have a single, explicit line between consensual and non-consensual telemetry. They can gather their pitchforks around any proposal to change to it. As long as that consent line stands, the upper-bound on the integral of "harm to the non-consenting user over time" is zero. Zero is a powerful number! |
Beta Was this translation helpful? Give feedback.
-
When you publicly discuss work from a work platform in your working hours, you represent your company. |
Beta Was this translation helpful? Give feedback.
-
|
All the companies that collect data from me ask if I authorise them to do so, including Google (which is making efforts, e.g. recently they have started to respect GDPR in Maps, YT, they don't show up before asking me to first authorise them). If this simply opts (pun intended) for opt-in by default, then all my concerns are moot since this becomes legal according to the laws protecting a European user. |
Beta Was this translation helpful? Give feedback.
-
The forks didn't go anywhere because Muse removed the telemetry and left it with optional crash reports and optional update checking. It did not just continue as before; the default telemetry was removed.
I don't, but this isn't about whether or not I have warm feelings toward random Google employees. The current proposal happens to be about Go, but the introductory blog post being titled "Transparent Telemetry for Open Source Projects" instead of "Transparent Telemetry for Go" is probably not accidental. The pushback here is not specific to Go; a larger discussion is happening around whether the "open source community," vague definition notwithstanding, wants baked-in telemetry to be a part of the culture. It's obvious this stuff is going to get put into the Go programming language. It would look bad for Russ' proposal to get shot down, the Go governance team does not have a history of incorporating external feedback, and half of the Go maintainers are already too emotionally attached to the idea to even consider backtracking now. That's fine; it's Google's sandbox, we're all just playing in it. The question is whether this behavior convinces people that it's ok to bake telemetry into open-source projects. Previous attempts have been shouted down, but previous attempts were not bankrolled by a corporate behemoth. Google has sufficient influence to swing lots of opinions. Any objection has to be loud or it will not even be noticed. I personally don't want to live in the world where open source projects are rife with baked-in telemetry, as I regard it a technological shortcut: an attempt to bypass the proper community relationship building and trust-fostering that would enable necessary feedback to happen organically. "It's okay, we don't need to talk to anyone, we'll just have their computers report on them for us" is not a healthy community feedback channel, especially with an opt-out flag to ensure it's just a sampling bias factory. But to bring this back to Go, I don't think "I trust the Go devteam" is a long-term assurance. Google, as diverse a conglomeration of interests and motivations as it might be, is still a faceless megacorporation who could can the whole team at any time, for no reason whatsoever, or even for bad reasons. The fact that the Go dev team has taken zero steps to divest itself of utter reliance on Google dollars and Google services is proof only that the dev team trusts their managers. I do not share that trust, and I think the team would be well served to examine the long-term consequences of both that dependence and large-scale information-gathering exercises which submit the collected data to corporate ownership. It's not that hard to set up an independent body to hold the data in escrow. Hell, a public-benefit corporation dedicated to being a telemetry ombudsman might even be a good business to run. But that's not even being considered here; all the governance is just being taken for granted, and that indicates a lack of depth of consideration beyond the mere technical implementation of the tools. Content warning for this paragraph: offensive terminology reference Obviously nobody deliberately set about to create this situation. Someone at Google just decided to do a thing, and nobody ever revisited that decision. So, when someone at Google sets about to do this thing, they might want to put more thought into what the long-term consequences might be, whether it might be wise to revisit the decision some years down the road, what potential for abuse it might have, and so forth. It's not promising that people who are already asking about abuse vectors are being told that the proposed software is just way too clever to be gamed, even before the software has actually been implemented. In short, a lot of words to say "there is a lot more to worry about here than Go, but we should worry about Go too." |
Beta Was this translation helpful? Give feedback.
-
|
The controversy here is whether the proposed telemetry should default on or default off. But there's at least one other option: default advertise. When you have neither opted in nor opted out, the Go toolchain could remind you, "There's this telemetry option we'd like you to turn on." In fact here's another option: time-limited opt-in. "I agree to send telemetry until 31 Dec 2023." That should help some users over the natural resistance barrier to opting in. |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
It seems as if once a comment has been hidden, or marked as off topic or marked as spam, it is not possible to "vote or thumbs-up comments" so that perspective seems disingenuous, to understate it. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
The plan would be that this server (which does not exist yet) would be completely open source, running on Google Cloud, same as pkg.go.dev and go.dev. (When we did proxy.golang.org, we made the decision to use internal Google infrastructure for key parts, and that unfortunately means that the code is not possible to open source in a meaningful way. It's not worth revisiting that at the moment.) |
Beta Was this translation helpful? Give feedback.
-
|
If you are talking about the specific ideals of having an open-source server, yes, telemetry would use an open-source server. |
Beta Was this translation helpful? Give feedback.
-
|
for people that don't trust this, how could they verify that the server isn't running a modified build? (or something completely different?) |
Beta Was this translation helpful? Give feedback.
-
|
There is no way to verify what the server is running. That is why so much of the filtering and aggregation is done on the client, which you can verify. Literally the only thing the server needs to do is republish the uploaded record (which you can check) and forget the IP address that uploaded it (which you can't). Literally the only thing you must trust is that the server runs the code we say it does, so that one time per year that the server gets an upload from your system, it discards your IP address. If you don't trust that, it is of course easy to set GOTELEMETRY=off. |
Beta Was this translation helpful? Give feedback.
-
|
Edit: I missed the use cases link at the bottom. (I feel one or two examples should have been included in the original post, and left the link as an addendum. the other two blog posts are otherwise sufficiently summarized)
|
Beta Was this translation helpful? Give feedback.
-
|
Did you read the list of use cases? |
Beta Was this translation helpful? Give feedback.
-
|
This was answered in the links at the bottom of the initial post, most notably the list of use cases. |
Beta Was this translation helpful? Give feedback.
-
|
The introductory blog post linked in the OP gives multiple concrete examples. To make a sweeping generalisation, the examples basically boil down to "things that the users of Go tools either didn't realise were bugs, or relied on as features", neither of which will motivate the users to open bug reports unprompted (and in the case of bugs, there isn't any way to prompt the user because the bugs aren't intentional). |
Beta Was this translation helpful? Give feedback.
-
|
I'm particularly interested in the numbers for cgo usage. I try to avoid cgo wherever I can, even going so far as to attempt to rewrite wlroots in as pure Go as I could get it, but its usage is just inevitable in an annoying number of cases. I think the discouragement from using it and the awkwardness of getting the garbage collector to play nicely with C has contributed a lot to the lack of, for example, good GUI support in Go, among other things. Also, if you're going to consider phasing out Fortran support, it might be interesting to see Rust get supported by cgo. Being able to build a mixed Go/Rust system using just |
Beta Was this translation helpful? Give feedback.
-
|
I don't believe we'd remove Fortran support outright but it would be interesting to know whether it merits any new attention at all. |
Beta Was this translation helpful? Give feedback.
-
|
I have resisted Go for years because it comes from Google. I've started to work with it a little bit, because I've been convinced over time by friends who love building with Go that it really isn't that Googley, and it's just Google funds this thing the Go creators wanted to do their way. The GOPROXY issues were a big scare point, but hey, you can shut that off, I guess. Now I feel just a little bit sold out by my friends who told me Go was okay to invest time in. Now you guys want to introduce telemetry into your programming language? This is how you drive off any person who even considered giving your project a chance despite the warning signs. Please don't do this, and please issue a public apology for even proposing it. Please leave a blast radius around this idea wide enough that nobody even suggests trying to do this again. Trust in Google's behavior is at an all time low, and moves like this are a choice to shove what's left of it off the edge of a cliff. Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
@bronger GOPROXY does not exist to collect data about you, nor does it actually collect data about you. It provides important functionality. I wrote above in #58409 (reply in thread):
You are welcome to turn it off, but understand what you are giving up. Also note that if you set GOPROXY=direct, the go command still uses the checksum database to protect against supply chain attacks. If you really want the go command not to use servers, you also need to set GOSUMDB=off. |
Beta Was this translation helpful? Give feedback.
-
|
As someone who specifically set If the Go team cannot be convinced to make these features opt-in, I would at least suggest creating a single variable that will shut all of these features off, with a commitment that this will apply to any new features that talk to Google's systems as well. Users should not have to read the release notes to determine whether they need to change how they go about opting out. |
Beta Was this translation helpful? Give feedback.
-
|
The use of the proxy and the checksum database were introduced in the same Go release, and we've been clear to mention both of them from the start, including in the release notes. |
Beta Was this translation helpful? Give feedback.
-
That ... doesn't address the point 😕 |
Beta Was this translation helpful? Give feedback.
-
|
The user was saying these environment variables proliferated, as if they appeared over a period of time without properly announcing them to Go users. There are only two such variables, and they were added and announced at the same time. If a user only knew about one but not the other, it's possible that they heard about GOPROXY from another website or post which was incomplete, not from Go's own release notes, announcements, or documentation. |
Beta Was this translation helpful? Give feedback.
-
|
Quick question. You claim in the article that even though the IP is exposed because of the TCP connection, it is not logged in the reports. What happens if someone decides to spam the telemetry service with bogus data? Will the data be kept? How will you identify which data came from the spammer's address so that it can be removed? |
Beta Was this translation helpful? Give feedback.
-
|
It remains to be seen what will be necessary to combat potential abuse. I expect that similar to https://proxy.golang.org/privacy there would be the possibility of separate service logs used for running the system that are kept for not more than 30 days. That's fairly standard for running a system. That information would not make its way into the analysis side of the system. |
Beta Was this translation helpful? Give feedback.
-
|
Does this not negate the claim of "This IP address would not be associated with the uploaded reports in any way."? Even if the IP address and/or association to the IP address to the other service logs are only kept for 30 days (as you claim, I'm guessing laws could impact this retention period, but IANAL), they are still being kept, even if they are "separate" from the reports, they can, and probably will still be used to associate reports to IPs. Because of this, I don't think it is fair to make the above claim. |
Beta Was this translation helpful? Give feedback.
-
|
The system does not exist yet; only a design does. Certainly there will be temporary traffic logs that don't record the association, in order to block DoS attacks and the like. The question is whether we would be forced to add the association to those temporary logs to deal with abuse. My first choice is to not record the association at all. I remain skeptical that IP information for abuse prevention, in part because someone could just grab a bunch of different IP addresses from a cloud service. Probably it will work better to look at the uploaded reports themselves, which will probably all look alike when viewed the right way. I also expect that fake data will make it into uploaded reports in small quantities and we can flag it retroactively. That said, I don't have a crystal ball. If bad actors make it necessary to record the association on the side temporarily in order to deal with abuse, then the design may need to be adjusted to permit that. Personally I doubt it will come to that, but I can't predict the future. I can say that even in the unfortunate event that the service logs do need to retain the association to help with abuse, that information will not be stored with the permanent logs nor made available to any part of the analysis side of the server. The proxy.golang.org/privacy policy says:
I expect we'd be able to make the same statement for the telemetry server. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the answers! I'm guessing the community will closely monitor the progress of this design, especially with the privacy implications. When the design necessitates storing of personally identifiable information, and their relation to reports (and it probably will), I hope you take into consideration at least making reporting telemetry opt-in. Sspecially since, IMHO, the problems it solves do not appear to be as impactful as the problems the proxy solved for it to be enabled by default. If the decision is made to enable telemetry by default, I hope the user is informed of this when using tooling that makes reports, even if the personally identifiable data is not associated to reports, or logged yet, as long as it can be. Having not kept with release notes on every toolchain update, it was a very bad surprise to find out that the proxy was enabled by default, I hope this does not happen again with telemetry. |
Beta Was this translation helpful? Give feedback.
This comment was marked as spam.
This comment was marked as spam.
-
|
No telemetry in my toolchains, hard pass. Edit: I'm making a bet here now this will get shoved through despite concerns and people not finding these "mitigations" acceptable. This isn't something that belongs in major developer tooling like the Go compiler/toolchain and is an abuse of power from a company that tends to take privacy lightly. It's not about mitigating concerns, it's about why we have them and why this is a toxic idea. My advice is to steer clear of this, Google swung and missed and let Go of the bat here. |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
gccgo is also an option. |
Beta Was this translation helpful? Give feedback.
-
|
While I am always in favor of more gccgo users, the telemetry being discussed here would be managed by the go tool, and gccgo's versions of the go tool is largely the same as the gc version. So using gccgo would not avoid telemetry unless you also avoid the go tool. And you can avoid the go tool using the gc compiler just as you can with gccgo; that's how, for example, Bazel works. That said, it's true that using gccgo would reduce the amount of telemetry collected specifically by the compiler. And, of course, if you are concerned enough about telemetry to use gccgo, I think that a simpler approach would be setting the environment variable |
Beta Was this translation helpful? Give feedback.
This comment was marked as spam.
This comment was marked as spam.
-
|
There's already quite a lot of useful data stored at https://pkg.go.dev/ and also able to be crawled from source-available (not just open-source) code repositories like GitHub, etc What additional data points would the Go team find useful that isn't already available from the above? |
Beta Was this translation helpful? Give feedback.
-
|
@natefinch many of the things you just mentioned could be better understood by analysis of CI definitions e.g. GitHub Actions, GitLab CI, etc in open-source repositories It would be terrific actually for these to be crawled and checked prior to the deprecation and removal of CLI features, please do this so that my builds never break unexpectedly :) |
Beta Was this translation helpful? Give feedback.
-
|
Dockerfile files are another good source of |
Beta Was this translation helpful? Give feedback.
-
|
Ultimately, what I'm getting at here is there's a wealth of information already available in forms where submission for public use is already explicit (e.g. pushing code to a source-available / open-source repository), and it would be good to exhaust this approach first |
Beta Was this translation helpful? Give feedback.
-
|
I don't believe there is much evidence that open-source usage looks anything like closed-source usage. Most open-source Go usage is in libraries and small commands. Most closed-source production usage is whole programs, some of them quite large. There are exceptions on both sides, but it seems quite wrong to assume that things like tool usage and performance look the same in open-source as in closed-source production usage. There's only one way to find out, of course. |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as spam.
This comment was marked as spam.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
This is mostly a technical proposal offered without enough consideration for one of the main preliminary concerns, which is (social) trust. And as I'll argue at the end of this comment, among others, more expert consensus is necessary to get there first. You can't admit that for years you didn't believe telemetry was necessary and that we all tend to have a "visceral negative reaction" to telemetry and then not confront the subject anywhere near the title of this discussion. Or not discussing the potential dangers nor attempting to explicitly address the concerns of the many people who have doubts. If someone wants to say: "you should have read the blogpost, most issues you mention are addressed through the comments of this long discussion"; yes, that's exactly why this is poor communication. Pursuing something that you believe can help improve the way things are done no matter how contrary it may to the public sentiment can be brave and noble. Failing to go out of your way to address that public sentiment while you do it is a communication and leadership failure (let's not get too dramatic either, though). The list of use cases and the blogpost make a strong case for the pros of telemetry. If the cons and potential dangers and risks were discussed in as much depth and prominently acknowledged, this could start to be a different discussion. Now, on the topic of opt-in by default, there are obvious moral concerns being brushed away too easily:
To me, a major issue that I think hasn't been discussed here is that the differences between transparent telemetry and telemetry are not yet recognized or clear to developers, and therefore opt-in by default can't be possibly acceptable. But even if you happen to be right and opt-in by default is perfectly morally acceptable for transparent telemetry and given some time privacy experts would agree with you or whatever, we are not at that point yet. I honestly believe that a few blogposts by a single individual, no matter how good your reputation may be, don't have enough weight to say that opt-in by default is ok (specially in a topic related to privacy). After reading the more detailed proposal, I'm still very unconvinced. I think there needs to be more and more qualified and broader consensus and feedback, both from experts and developers, before transparent telemetry can legitimately materialize in the suggested form for a project of the scale of Go. I don't think this discussion approaches all these concerns from the right angle at the right place. |
Beta Was this translation helpful? Give feedback.
-
|
Actually, opt in by default is neither morally nor in many cases legally defendable for private data, even if it is "only" statistics. In fact in many countries you have to provide the option to delete the data collected on demand. |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment has been minimized.
This comment has been minimized.
-
|
After seeing what happen with Tailscale IOS, I do see the need we should have some form of telemetry, I would also recommend try discussing this with Developers of CockroachDB or Traefik to get their feedback |
Beta Was this translation helpful? Give feedback.
-
What if it’s just not IOS what if it every single “GOOS” and “GOARCH”, what if it’s a certain implementation in the runtime, it’s not like we operate in a scale like cloudflare where instances of 1 in a billion such issue can pop up, what if we were to implement a feature that’s undermining to a certain architecture, the Go compiler is using its own tools there’s no llvm, no gcc there’s no 3rd party tool helping the backbone of GC, |
Beta Was this translation helpful? Give feedback.
-
|
@beoran the tool you linked to is marked as "Deprecated", so I doubt anyone would reach for it when writing production software. More importantly, the problem in this case is that Apple has an internal non-public app verification process, and there is no (efficient) way for anyone outside Apple to enumerate all of the checks they are currently applying, or what checks they will add or remove in future. Developers will do what they can, but it's unrealistic to expect all these issues to be caught in CI (and I expect Apple would ban apps that tried to submit every single CI app build for testing). |
Beta Was this translation helpful? Give feedback.
-
|
@Patrickmitech I don't think telemetry would have helped in this case either. The problem was not visible in the compilation process; the API is there because C, and it being private is "socially" enforced by Apple. |
Beta Was this translation helpful? Give feedback.
-
|
For what it's worth, if this is only about #58323, that's something that's handled perfectly well with a bug report, because it's broken enough that users will report it. No telemetry needed. Plenty of other misbehaviors aren't necessarily broken enough for that, but this one is. |
Beta Was this translation helpful? Give feedback.
-
|
Another thing about Tailscale and telemetry is, I was impressed by security bulletins they have dispatched where they're able to gauge the blast radius on bugs that I imagine were not pleasant to uncover. That's huge. The normative concerns around telemetry are legitimate and in ways might supersede technical merits of any particular solutions, but I think this works as an example of something where privacy and security is well served by telemetry. |
Beta Was this translation helpful? Give feedback.
-
|
Enabled by default means you believe that what I do on my computer is your business unless I choose otherwise. This should be self-evidently "wrong", because what I do on my computer is absolutely not the Go team's business unless and until I choose to make it your business. It does not matter whether knowing "what I do" makes your job easier, or is somehow "useful" to you. It does not matter if you find an implementation that is somehow technically "legal". It does not matter if the proposed implementation of telemetry itself is sound and reasonable. None of that changes the simple fact that what I do on my computer is not, by default, ever any of your business. That this is even under consideration raises a number of questions regarding Go's leadership and future direction. |
Beta Was this translation helpful? Give feedback.
-
|
This may be a naive response why does a “button” called |
Beta Was this translation helpful? Give feedback.
-
Because the discussion is not only about the people that are already here. It's about all the people who are going to install in the future that don't know about the telemetry and that their personal data is being collected by someone without their consent. |
Beta Was this translation helpful? Give feedback.
-
|
This whole thing coud have been averted if the “elephant in the room” pre-requisite had been acknowledged: opt-in is not an option and is the default in many countries. If that acknowledgment had come from the beginning, and guarantees given by the Go team on respecting things like GDPR, then it would be exactly what everybody else is doing (because laws forces them to) and the technical discussion could have continued on its own merits. Instead, comments speaking precisely about that early on are hidden or wrongly treated as off-topic for the sake of “readability”. This is a terrible signal for people looking at Go for future developments. |
Beta Was this translation helpful? Give feedback.
-
"all comments" is utterly false. Literally the very first comment that people see when they load the page (with the default "Oldest" sort) is about opt-in vs opt-out. So is this one that we are both replying to. Many pointless, redundant snarky comments on this topic are hidden in order to focus the productive discussion on the most useful and well-reasoned commentary on this topic. Please stop lying about this. (Or, perhaps more charitably, if you are unfamiliar with GitHub, perhaps you are getting confused by GitHub not being able to render more than a few top-level threads and instead drawing a "paper cut" with an "N hidden items" link. If so, that's not about the content.) |
Beta Was this translation helpful? Give feedback.
-
|
You're right, not all. But many that were issued early on. The others came after those, which were hidden, so repetitions etc. |
Beta Was this translation helpful? Give feedback.
-
|
Coming from a very privacy aware country and background the problem for me at least is the collecting itself. Also taking Google's reputation on handling private data into consideration I think there is very little to be said to convince people. Voluntary participation in surveys etc is much better way overall and I think this telemetry idea should be abandoned. |
Beta Was this translation helpful? Give feedback.
-
Go surveys could begin with the sentence "Telemetry data is extremely helpful, so please consider running |
Beta Was this translation helpful? Give feedback.
-
|
I'm very privacy aware and I don't think they have to abandon the idea. They simply have to commit to respect privacy by default, and collect whatever from whoever has been properly warned and has accepted that beforehand. |
Beta Was this translation helpful? Give feedback.
-
Fair enough. But do you think "on by default" covers the constraints? The reason why I suggested to abandon the idea is that what value would "opt-in" model bring? Ofc there might be angles and solutions I didn't consider. |
Beta Was this translation helpful? Give feedback.
-
Why was this comment minimized? It seems pretty darn legit to me, I don't think this line of thinking has been mentioned elsewhere in the discussion, and it's not really pithy. Suppress this line of thinking and I will personally feel the need to move even more of my everyday computing into sandboxes that prevent unwanted networking. Also, need I remind people of https://harmful.cat-v.org/cat-v/ |
Beta Was this translation helpful? Give feedback.
-
The issue with hidden comments is that people have to spend extra time opening them up to see if they really are spam or harassment, or just things the moderators don't agree with, they are almost always trigger happy. I have found some hidden comments that can't really be explained by "repeated" or "adding nothing":
I'm sure there are more. |
Beta Was this translation helpful? Give feedback.
-
|
@noonien I saw both of those comments, and I saw the earlier comments with essentially the same feedback as those. The first of them offers the most links of any of the "Google habitually ignores privacy laws" comments to support the claim, but the number was not otherwise zero. The second offers a particular spelling for the environment variable with the effect suggested (I think only once) before. I am at work now, but if you remind me in about five hours, I'll be happy to find those comments for you. |
Beta Was this translation helpful? Give feedback.
-
|
Please do. But understand, hiding "repeated arguments" doesn't make the initial argument more visible, and you not finding the comments you're referring only proves this point. This is why I asked what the reasoning behind doing it is. Hiding comments only compounds the issue that the GitHub UI already does this by default. |
Beta Was this translation helpful? Give feedback.
-
@noonien We're on the same side of this issue, but I think it's also important to recognize that there will be some false-positives in any moderation process. Perfection cannot be the standard. I for one find the dialogue tense, but net constructive. I am having good results by taking the time to articulate my thoughts clearly (as are you), and I hope you will join me in focusing on that 🙂. |
Beta Was this translation helpful? Give feedback.
-
I understand and completely agree, however, this is the exact reason why hiding comments because of "repeating" does not seem like a very good rule. Comments that are vulgar, or completely off-topic should 100% be hidden/removed though. Perfection should not be the goal for moderating, but transparency and consistency. The comments I linked are neither vulgar nor off-topic (IMHO). No reason was given for them being hidden, nor were they un-hidden after they were pointed out.
Agree, and I'm doing my best 😄 |
Beta Was this translation helpful? Give feedback.
This comment was marked as spam.
This comment was marked as spam.
-
|
there seems to be quite a detailed design posted on the blog for something that's supposed to be "not a proposal" and "something that might become a proposal"... |
Beta Was this translation helpful? Give feedback.
-
|
That is standard procedure for the Go project. |
Beta Was this translation helpful? Give feedback.
-
|
I quite appreciate it: better to have a non-proposal the subject of a "spirited debate" than a formal one. For such draft proposals, my late father used to say Suggestions, corrections and well-flung brickbats are cordially invited (:-)) |
Beta Was this translation helpful? Give feedback.
-
|
@victorb64 I think I've been one of the more vocal opponents of this idea, but I also think this is a prime example of where we should assume good faith on the part of the Go team. When you think about it, it's "damned if you do; damned if you don't". If they had not posted a detailed design and accompanying blog-post, we would be berating them for a lack of transparency. I for one choose to view this charitably, and I hope you will join me. |
Beta Was this translation helpful? Give feedback.
This comment was marked as spam.
This comment was marked as spam.
-
|
@udf2457 This issue tracker is governed by the Go Community Code of Conduct. Disagreement is fine but please be polite. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
I've made a list of pros and cons that I've found in this discussion so far: Pros:
Cons:
I am obviously biased towards opt-in telemetry, and I might have missed arguments, so please add to the list if you can think of anything else. |
Beta Was this translation helpful? Give feedback.
and others
-
Feb 10 5pm US Eastern: Thanks for the lively discussion everyone. It has given me plenty to think about. We are starting to go in circles, and actually new comments are increasingly rare, so I've locked this to give everyone their weekend back. Thanks again.
If anyone feels like there's something important that wasn't said in these 506 comments, please feel free to email me at [email protected].
Feb 26 The current plan is to use some form of opt-in to collect telemetry.
How do software developers understand which parts of their software are being used and whether they are performing as expected? The modern answer is telemetry, which means software sending data to answer those questions back to a collection server.
I believe that open-source software projects need to explore new telemetry designs that help developers get the information they need to work efficiently and effectively, without collecting invasive traces of detailed user activity.
I have written a short series of blog posts about one such design, which I call transparent telemetry, because it collects as little as possible (kilobytes per year from each installation) and then publishes every bit that it collects, for public inspection and analysis.
I’d like to explore using transparent telemetry, or a system like it, in the Go toolchain, which I hope will help Go project developers and users alike. To be clear, I am only suggesting that the instrumentation be added to the Go command-line tools written and distributed by the Go team, such as the
gocommand, the Go compiler,gopls, andgovulncheck. I am not suggesting that instrumentation be added by the Go compiler to all Go programs in the world: that’s clearly inappropriate.Transparent telemetry has the following key properties:
The decisions about what metrics to collect are made in an open, public process.
The collection configuration is automatically generated from the actively tracked metrics: no data is collected that isn’t needed for the metrics.
The collection configuration is served using a tamper-evident transparent log, making it very difficult to serve different collection configurations to different systems.
The collection configuration is a cacheable, proxied Go module, so any privacy-enhancing local Go proxy already in use for ordinary modules will automatically be used for collection configuration. To further ameliorate concerns about tracking systems by the downloading of the collection configuration, each installation only bothers downloading the configuration each week with probability 10%, so that each installation only asks for the configuration about five times per year.
Uploaded reports only include total event counts over a full week, not any kind of time-ordered event trace.
Uploaded reports do not include user IDs, machine IDs, or any other kind of ID.
Uploaded reports only contain strings that are already known to the collection server: counter names, program names, and version strings repeated from the collection configuration, along with the names of functions in specific, unmodified Go toolchain programs for stack traces. The only types of non-string data in the reports are event counts, dates, and line numbers.
IP addresses exposed by the HTTP session that uploads the report are not recorded with the reports.
Thanks to sampling, only a constant number of uploaded reports are needed to achieve a specific accuracy target, no matter how many installations exist. Specifically, only about 16,000 reports are needed for 1% accuracy at a 99% confidence level. This means that as new systems are added to the system, each system reports less often. With a conservative estimate of two million Go installations, about 16,000 reporting each week corresponds to an overall reporting rate of well under 2% per week, meaning each installation would upload a report on average less than once per year.
The aggregate computed metrics are made public in graphical and tabular form.
The full raw data as collected is made public, so that project maintainers have no proprietary advantage or insights in their role as the direct data collector.
The system is on by default, but opting out is easy, effective, and persistent.
Please read the introductory blog post for more background before commenting here. You may also want to check out the detailed design and list of use cases.
Beta Was this translation helpful? Give feedback.
All reactions