OONI Probe CLI v3.17.0-alpha.1

Overview

This release should be the solid starting point for the 3.17.x train of releases. We will test it next week. If everything is fine, we can then move forward with creating v3.17.0 and with updating all the apps.

What's Changed

✨✨✨ We rewrote the build procedure in Go. We did this for two reasons. The first reason is unit testing. We can now be confident that the build procedure invokes the correct commands. The second reason is that we wanted to compile tor and its dependencies for Android using hardening flags. We implemented these changes in these pull requests:

• feat: simplify writing tests for code using shellx by @bassosimone in #1042
• refactor: factor common code to check whether file exists by @bassosimone in #1045
• feat(shellx): support incremental command construction by @bassosimone in #1046
• feat(runtimex): add the TryN functions by @bassosimone in #1047
• feat(must): convenience package with "must" functions by @bassosimone in #1048
• feat: rewrite darwin build rules in Go by @bassosimone in #1049
• feat: rewrite windows build rules in Go by @bassosimone in #1050
• feat: rewrite generic build rules in Go by @bassosimone in #1051
• feat: rewrite linux build rules in Go by @bassosimone in #1044
• feat: mechanism compile and embed tor by @bassosimone in #1052
• feat: rewrite the android build in Go by @bassosimone in #1053
• feat: convert ios build to use Go by @bassosimone in #1054
• feat: build for Android using Go by @bassosimone in #1055
• cleanup: ghgen should be in internal/cmd by @bassosimone in #1056
• refactor(buildtool): unify C deps with other deps by @bassosimone in #1057
• refactor(buildtool): merge cdeps and android build envs by @bassosimone in #1058
• refactor(buildtool): use naming reminding of bash variables by @bassosimone in #1059
• refactor(buildtool): improve how we set environment variables by @bassosimone in #1060
• feat: cross-compile tor for android using the Go builder by @bassosimone in #1061
• feat(buildtool): add unit tests for android builds by @bassosimone in #1062
• feat: use internal/libtor for android builds by @bassosimone in #1063
• chore: update to ooni/go-libtor v1.1.7 by @bassosimone in #1064
• doc(buildtool): explain ANDROID_NDK_HOME for android gomobile by @bassosimone in #1066

✨✨✨ We continued to improve and modernize the code that communicates with the OONI backend. In particular, in this release, we added new code that allows us to perform A/B testing of the Web Connectivity implementation. This change will allow us to start testing the new Web Connectivity implementation. We will take advantage of this functionality to start implementing and testing throttling measurements inside Web Connectivity, which we will work on soon. In addition, the changes we implemented simplify providing more complex inputs to experiments. (The fact that we can tell the probe which version of Web Connectivity to run is indeed a special case of providing more complex input.) We did this work in:

• refactor: rename check-in nettests results for clarity by @bassosimone in #1036
• refactor(session): observe full check-in response by @bassosimone in #1037
• fix(httpapi): don't return nil, nil with null JSON input by @bassosimone in #1038
• feat: enable A/B testing using check-in API by @bassosimone in #1039

✨✨✨ We added support for exposing build information. This gives us extra confidence about the specific version of the measurement engine being used for collecting a measurement, which is great for data quality assessments. We did this work in:

• feat: expose build information by @bassosimone in #1030

🐛🐛🐛 We fixed a bug where Web Connectivity could be tricked to fetch from localhost:

• fix([email protected]): stop fetching from localhost by @bassosimone in #1027

🚧 🚧 🚧 We completed a long standing refactoring by moving internal/engine packages to internal. It's good to do this right before making a new release train (3.17.x) because it simplifies backporting patches if needed:

• refactor: prepare for renaming all experiments by @bassosimone in #1032
• refactor: continue moving packages outside of internal/engine by @bassosimone in #1031
• refactor: move all experiments into internal/experiment by @bassosimone in #1033
• cleanup: finish moving packages away from engine by @bassosimone in #1034
• chore: update readme files and perform minor cleanups by @bassosimone in #1035

🚧 🚧 🚧 We did the usual grunt work associated with preparing a release: updating dependencies and cleaning up stuff. We specifically did this work in the following pull requests:

• cleanup: use sync/atomic instead of atomicx by @bassosimone in #1028
• chore: use go1.19.5 by @bassosimone in #1029
• feat(engine): make netxlite.utlsConn public by @kelmenhorst in #1018
• chore: update dependencies by @bassosimone in #1041
• fix: attempt to make all workflows green by @bassosimone in #1065

Full Changelog: v3.17.0-alpha...v3.17.0-alpha.1

OONI Probe CLI v3.17.0-alpha

Changes since v3.16.0-alpha.3

• ✨ ooniprobe: add the --proxy command line flag which works like miniooni --proxy
• ✨ miniooni and ooniprobe: add support for the --proxy torsf:///, which uses snowflake over tor
• ✨ miniooni: add --snowflake-rendezvous CLI flag to specify the snowflake rendezvous policy
• ✨ miniooni: read OONI Run v2 descriptors form the filesystem with miniooni oonirun -f FILE
• ✨ oohelperd: add support for measuring HTTP/3
• ✨ portfiltering: new experiment for measuring port blocking
• ✨ echcheck: new experiment for measuring Encrypted Client Hello blocking
• ✨ tlsmiddlebox: new experiment implementing iterative network tracing for TLS
• ✨ [forwardport] webconnectivity: update LTE implementation to v0.5.19
• ✨ geoipx: use ASN+Country database generated using ooni/historical-geoip
• ✨ httpapi: new API allowing to fallback when multiple endpoints are available
• ✨ measurexlite: add refraction-networking/utls support
• ✨ probeservices: support compressed check-in API responses
• ❗ [API BREAK] oonimkall: drop deprecated APIs
• ❗ [API BREAK] oonimkall: rename CheckInConfigWebConnectivity's Add method to AddCategory
• ❗ [ABI BREAK] model: use udp rather than quic as the protocol (see #946 for details)
• 🐛 psiphon: make sure we include a config when building for Linux
• 🐛 [forwardport] signal: update the embedded signal CA
• 🐛 webconnectivity: increase robustness by trying to use all available TH before giving up
• 🐛 whatsapp: stop measuring http://web.whatsapp.com/ to avoid false positives
• 🐛 geolocate: make sure we use the session resolver for consistency
• 🐛 measurexlite: emit the resolve_start and resolve_done events
• 🐛 model: mark optional fields as optional and sync up with ooni/spec's data formats
• 🐛 [forwardport] model: improve measurements scrubbing
• 🐛 netxlite: make sure we wrap DNS decoding errors
• 🐛 scrubber: merge improvements from the snowflake codebase
• 🚧 [forwardport] telegram: stop measuring http://web.telegram.org/ to avoid potential false positives
• 🚧 .github/workflows: automatically generate several github workflows files
• 🚧 all: use go1.19.4 and update dependencies
• 🚧 QA: simplify quality assurance scripts and focus on web connectivity only
• 🚧 Makefile: add generic rules for building ooniprobe and miniooni for the current GOOS/GOARCH
• 🚧 MONOREPO: merge monorepo-like scripts to facilitate integration testing
• 🚧 ooniprobe: make database code abstract to facilitate subsequent refactoring
• 🚧 internal: change way in which we provide arguments to OONI experiments
• 🚧 probeservices: start using the httpapi package
• 🚧 probeservices: remove support for deprecated APIs
• 🚧 httpx: deprecate in favour of httpapi
• 🚧 model: reintroduce netxlite's underlying network functionality

Entries marked as [forwardport] lift patches from the latest stable releases in the v3.16.x series.

We don't need to bump the major version number because API changes in pkg/oonimkall do not influence the major version number.

New Contributors

• @arky made their first contribution in #967
• @ooninoob made their first contribution in #987
• @d1vyank made their first contribution in #970

Full Changelog: v3.16.0-alpha.3...v3.17.0-alpha

OONI Probe CLI v3.16.7

This patch release includes the following features and fixes:

• [backport] fix(whatsapp): stop measuring http://web.whatsapp.com/ (#998) 4329b89
• [backport] fix(telegram): stop measuring http://web.telegram.org/ (#999) 7a03da8
• [backport] doc(whatsapp): explain the tricky >= 2 check (#1000) e3bf526
• [backport] fix(whatsapp): allow registration server to HTTP-fail (#1001) 1c99427

OONI Probe CLI v3.16.6

This patch release includes the following features and fixes:

• [backport] refactor([email protected]): improve logging clarity (#964): ac423f6
• [backport] doc([email protected]): improve readme: d5b3ab9
• [backport] doc([email protected]): link to analysiscore.go: dc9ff91
• [backport] fix([email protected]): limit number of redirects (#965): 9bf6e38
• [backport] feat(webconnectivity): try all the available THs (#980): 25d8c86
• fix(QA): reckon with TH fallback policies: 1de058d

OONI Probe CLI v3.16.5

This release contains the following fixes:

• 🐛 Fix the signal root CA

Full Changelog: v3.16.4...v3.16.5

OONI Probe CLI v3.16.4

This release contains the following fixes:

• [backport] fix(go-build-alpine): honour OONI_PSIPHON_TAGS

• [backport] fix: stop using deprecated ams-pg.ooni.org domain name

• [backport] Add new root CA to the signal test

Full Changelog: v3.16.3...v3.16.4

OONI Probe CLI v3.16.3

This patch release contains minor fixes as well as improvements for the experimental [email protected].

Please, see v3.16.0's release notes for more details on what changed since v3.15.x.

Full Changelog: v3.16.2...v3.16.3

OONI Probe CLI v3.16.2

This release contains changes from the master branch that it felt okay to backport to release/3.16. Because the master branch does not include refactoring yet, it feels safe to backport fixes and improvements, while skipping entirely new features.

Please, see v3.16.0's release notes for more details on what changed since v3.15.x.

Full Changelog: v3.16.1...v3.16.2

OONI Probe CLI v3.16.1

Overview

This patch release fixes a bug in [email protected].

Please, see v3.16.0's release notes for more details on what changed since v3.15.x.

What's changed

• [backport] fix([email protected]): DoH failure shouldn't set flags (#948): 68cfc70

Full Changelog: v3.16.0...v3.16.1

OONI Probe CLI v3.16.0

Highlights

• web connectivity's long-term evolution #882 #870: we have implemented an improved version of Web Connectivity that improves TLS and DNS censorship detection while retaining backward compatibility. In particular, we are now able to detect duplicate DNS over UDP responses when some middlebox replies to DNS queries but also allows replies from legitimate servers to reach users (which is what happens, for example, in China). This new version of Web Connectivity is only available to miniooni users, who can invoke it using miniooni [email protected]. We look forward to collect more data and perform A/B comparison with the current stable version. We aim to make this new version of Web Connectivity the default in the next release of ooniprobe. To help us collect more data, if you are a miniooni user, please run miniooni [email protected] periodically, check the resulting JSON, and reach out to us in case you see unexpected results. Thank you!

• call getaddrinfo directly #764 #908: previous releases of ooniprobe and miniooni used the Go resolver to lookup domain names, but this strategy was problematic because it hid the real system resolver error. A particularly problematic case was Android (see ooni/probe#2029 for more info). To workaround this issue, we ensured that we always link with the C library for officially built binaries and we implemented saving the raw error from libc's getaddrinfo in such a case. This change makes the resulting measurement more transparent and enables third parties to more easily vet our results.

• oohelperd improvements #890 #886: this release includes improvements in the Web Connectivity test helper that allow Web Connectivity's long term evolution's code to perform better TLS blocking detection decisions. In addition to this change, the improved test helper also includes in its response additional data useful to determine whether IP addresses resolved by any version of Web Connectivity (including the stable version) are legitimate or invalid for the requested domain.

• build ooniprobe for armv6 #904: we're now building ooniprobe for armv6, as requested by the community.

• implement rolling builds #910 #920: we created a new (fake) release called rolling. Every night a build task builds the master branch and publishes binaries in such a release. Additionally, we have a way to manually force specific builds to publish packages to be tested in such a release. If you want to run very recent binaries with all the latest changes, you should download from this release. This set of changes is an initial response to community requests regarding facilitating using ooniprobe and miniooni for research purposes in unattended, censored environments.

• make miniooni CLI more user friendly #913 #932: we modified miniooni's CLI to make miniooni [experiment] [options] work exactly as miniooni [options] [experiment]. Previously, only the latter was supported. Additionally, we revamped the output emitted with --help to provide experiment-specific contextual information, including a list of all supported options. These changes should hopefully simplify using miniooni for research purposes. Additionally, now you can pass --emoji as a command line flag to get emoji output (which simplifies quickly eyeballing logs).

• build miniooni and ooniprobe for android #907: we're now building miniooni and ooniprobe for Android, such that they could be run from Termux's CLI, as requested by the community.

• oonirun v2 #842 #843 #844 #916: the miniooni included in this release includes a preview of OONI Run v2 links which allows anyone to execute arbitrary OONI Run v2 descriptors by using the miniooni oonirun -i URL command. Please, read the provisional design document for this OONI Run v2 preview for more information: https://github.com/ooni/probe-cli/blob/v3.16.0/docs/design/dd-004-minioonirunv2.md.

• use ooni/oocrypto instead of ooni/go #751 #872: anticipating future changes in which we will make our TLS fingerprint more similar to Chrome's own fingerprint, this release switches to our fork of Go's crypto/tls library.

• dnscheck: lower the default timeouts #917: we lowered the default timeout of dnscheck, following a community request.

• generally expose more low-level fields: we improved the data model to include extra information useful to treat each kind of event separately (see ooni/spec#261). This change is the initial step to implement a community request to provide OONI measurements in tabular data, where each table is a specific kind of event (e.g. DNS lookup, TLS handshake).

• step-by-step design document #814: we designed and implemented a new strategy for organizing experiment's code that simplify collecting better data and ultimately enables providing self-sufficient events in tabular format.

What's Changed in detail

• chore: we're now hacking on v3.16.0-alpha by @bassosimone in #749
• chore: upgrade to go1.18.2 by @bassosimone in #750
• refactor: use ooni/oocrypto instead of ooni/go by @bassosimone in #751
• fix(oonimkall): export CheckInConfig.RunType by @bassosimone in #752
• fix({simplequic,tls}ping): default SNI to URL's hostname by @bassosimone in #753
• refactor: only use shaping dialer for ndt7 and dash by @bassosimone in #754
• cleanup: mark more integration tests as !short mode by @bassosimone in #755
• cleanup(geolocate): use netxlite rather than netx by @bassosimone in #756
• cleanup: minor Readme.md tweaks and changes by @bassosimone in #757
• cleanup: merge httpheader and httpfailure into model by @bassosimone in #758
• cleanup: move legacy from internal/engine to internal by @bassosimone in #759
• Spring cleanup: remove unused/unneded code by @bassosimone in #761
• refactor: DNSTransport I/Os DNS messages by @bassosimone in #760
• feat(netxlite): observe additional DNS-over-UDP responses by @bassosimone in #762
• fix(dnsoverudp): allow to cancel async round trip immediately by @bassosimone in #763
• netxlite: call getaddrinfo and handle platform-specific oddities by @bassosimone in #764
• getaddrinfo: fix CGO_ENABLED=0 and record resolver type by @bassosimone in #765
• netxlite: do not call netgo the CGO_ENABLED=0 resolver by @bassosimone in #766
• refactor(session.go): replace engine/netx with netxlite by @bassosimone in #767
• refactor(ndt7): use netxlite rather than netx by @bassosimone in #768
• refactor: continue to simplify engine/netx by @bassosimone in #769
• refactor(netxlite): allow easy dialer chain customization by @bassosimone in #770
• refactor(netxlite): allow easy QUIC dialer chain customization by @bassosimone in #771
• refactor(netx): merge archival, trace, and the savers by @bassosimone in #772
• refactor(tracex): start applying recent code conventions by @bassosimone in #773
• refactor(netxlite): better integration with tracex by @bassosimone in #774
• refactor(netxlite): introduce the getaddrinfo transport by @bassosimone in #775
• feat(netxlite): implement DNSTransport wrapping by @bassosimone in #776
• refactor(tracex): do not depend on strings for event names by @bassosimone in #777
• tracex: prepare HTTP code for future refactoring by @bassosimone in #778
• fix: pin to gopkg.in/yaml.v3 v3.0.1 by @bassosimone in #779
• fix(tracex): generate archival from single transaction-done event by @bassosimone in #780
• refactor(tracex): convert to unit testing by @bassosimone in #781
• refactor: move tracex outside of engine/netx by @bassosimone in #782
• chore: upgrade oocrypto, oohttp, probe-assets by @bassosimone in #783
• hotfix: disable oocrypto until we investigate ciphers selection by @bassosimone in #784
• [forwardport] fix(dns...