OONI Probe CLI v3.16.0
Highlights
-
web connectivity's long-term evolution #882 #870: we have implemented an improved version of Web Connectivity that improves TLS and DNS censorship detection while retaining backward compatibility. In particular, we are now able to detect duplicate DNS over UDP responses when some middlebox replies to DNS queries but also allows replies from legitimate servers to reach users (which is what happens, for example, in China). This new version of Web Connectivity is only available to
miniooni
users, who can invoke it usingminiooni [email protected]
. We look forward to collect more data and perform A/B comparison with the current stable version. We aim to make this new version of Web Connectivity the default in the next release ofooniprobe
. To help us collect more data, if you are aminiooni
user, please runminiooni [email protected]
periodically, check the resulting JSON, and reach out to us in case you see unexpected results. Thank you! -
call getaddrinfo directly #764 #908: previous releases of
ooniprobe
andminiooni
used the Go resolver to lookup domain names, but this strategy was problematic because it hid the real system resolver error. A particularly problematic case was Android (see ooni/probe#2029 for more info). To workaround this issue, we ensured that we always link with the C library for officially built binaries and we implemented saving the raw error from libc'sgetaddrinfo
in such a case. This change makes the resulting measurement more transparent and enables third parties to more easily vet our results. -
oohelperd improvements #890 #886: this release includes improvements in the Web Connectivity test helper that allow Web Connectivity's long term evolution's code to perform better TLS blocking detection decisions. In addition to this change, the improved test helper also includes in its response additional data useful to determine whether IP addresses resolved by any version of Web Connectivity (including the stable version) are legitimate or invalid for the requested domain.
-
build ooniprobe for armv6 #904: we're now building ooniprobe for armv6, as requested by the community.
-
implement rolling builds #910 #920: we created a new (fake) release called rolling. Every night a build task builds the
master
branch and publishes binaries in such a release. Additionally, we have a way to manually force specific builds to publish packages to be tested in such a release. If you want to run very recent binaries with all the latest changes, you should download from this release. This set of changes is an initial response to community requests regarding facilitating usingooniprobe
andminiooni
for research purposes in unattended, censored environments. -
make miniooni CLI more user friendly #913 #932: we modified
miniooni
's CLI to makeminiooni [experiment] [options]
work exactly asminiooni [options] [experiment]
. Previously, only the latter was supported. Additionally, we revamped the output emitted with--help
to provide experiment-specific contextual information, including a list of all supported options. These changes should hopefully simplify usingminiooni
for research purposes. Additionally, now you can pass--emoji
as a command line flag to get emoji output (which simplifies quickly eyeballing logs). -
build miniooni and ooniprobe for android #907: we're now building
miniooni
andooniprobe
for Android, such that they could be run from Termux's CLI, as requested by the community. -
oonirun v2 #842 #843 #844 #916: the
miniooni
included in this release includes a preview of OONI Run v2 links which allows anyone to execute arbitrary OONI Run v2 descriptors by using theminiooni oonirun -i URL
command. Please, read the provisional design document for this OONI Run v2 preview for more information: https://github.com/ooni/probe-cli/blob/v3.16.0/docs/design/dd-004-minioonirunv2.md. -
use ooni/oocrypto instead of ooni/go #751 #872: anticipating future changes in which we will make our TLS fingerprint more similar to Chrome's own fingerprint, this release switches to our fork of Go's crypto/tls library.
-
dnscheck: lower the default timeouts #917: we lowered the default timeout of
dnscheck
, following a community request. -
generally expose more low-level fields: we improved the data model to include extra information useful to treat each kind of event separately (see ooni/spec#261). This change is the initial step to implement a community request to provide OONI measurements in tabular data, where each table is a specific kind of event (e.g. DNS lookup, TLS handshake).
-
step-by-step design document #814: we designed and implemented a new strategy for organizing experiment's code that simplify collecting better data and ultimately enables providing self-sufficient events in tabular format.
What's Changed in detail
- chore: we're now hacking on v3.16.0-alpha by @bassosimone in #749
- chore: upgrade to go1.18.2 by @bassosimone in #750
- refactor: use ooni/oocrypto instead of ooni/go by @bassosimone in #751
- fix(oonimkall): export CheckInConfig.RunType by @bassosimone in #752
- fix({simplequic,tls}ping): default SNI to URL's hostname by @bassosimone in #753
- refactor: only use shaping dialer for ndt7 and dash by @bassosimone in #754
- cleanup: mark more integration tests as !short mode by @bassosimone in #755
- cleanup(geolocate): use netxlite rather than netx by @bassosimone in #756
- cleanup: minor Readme.md tweaks and changes by @bassosimone in #757
- cleanup: merge httpheader and httpfailure into model by @bassosimone in #758
- cleanup: move legacy from internal/engine to internal by @bassosimone in #759
- Spring cleanup: remove unused/unneded code by @bassosimone in #761
- refactor: DNSTransport I/Os DNS messages by @bassosimone in #760
- feat(netxlite): observe additional DNS-over-UDP responses by @bassosimone in #762
- fix(dnsoverudp): allow to cancel async round trip immediately by @bassosimone in #763
- netxlite: call getaddrinfo and handle platform-specific oddities by @bassosimone in #764
- getaddrinfo: fix CGO_ENABLED=0 and record resolver type by @bassosimone in #765
- netxlite: do not call netgo the CGO_ENABLED=0 resolver by @bassosimone in #766
- refactor(session.go): replace engine/netx with netxlite by @bassosimone in #767
- refactor(ndt7): use netxlite rather than netx by @bassosimone in #768
- refactor: continue to simplify engine/netx by @bassosimone in #769
- refactor(netxlite): allow easy dialer chain customization by @bassosimone in #770
- refactor(netxlite): allow easy QUIC dialer chain customization by @bassosimone in #771
- refactor(netx): merge archival, trace, and the savers by @bassosimone in #772
- refactor(tracex): start applying recent code conventions by @bassosimone in #773
- refactor(netxlite): better integration with tracex by @bassosimone in #774
- refactor(netxlite): introduce the getaddrinfo transport by @bassosimone in #775
- feat(netxlite): implement DNSTransport wrapping by @bassosimone in #776
- refactor(tracex): do not depend on strings for event names by @bassosimone in #777
- tracex: prepare HTTP code for future refactoring by @bassosimone in #778
- fix: pin to gopkg.in/yaml.v3 v3.0.1 by @bassosimone in #779
- fix(tracex): generate archival from single transaction-done event by @bassosimone in #780
- refactor(tracex): convert to unit testing by @bassosimone in #781
- refactor: move tracex outside of engine/netx by @bassosimone in #782
- chore: upgrade oocrypto, oohttp, probe-assets by @bassosimone in #783
- hotfix: disable oocrypto until we investigate ciphers selection by @bassosimone in #784
- [forwardport] fix(dnscheck): stop emitting progress by @bassosimone in #785
- refactor(tracex): internally represent errors as strings by @bassosimone in #786
- refactor(tracex): internally store just the raw certificate by @bassosimone in #787
- cleanup(netx): remove subpackages and unnecessary code by @bassosimone in #788
- cleanup(netx): another batch of small/simple cleanups by @bassosimone in #789
- refactor(netx): use netxlite to build TLSDialer by @bassosimone in #790
- cleanup(netx): remove redundant config options by @bassosimone in #791
- cleanup(jafar): do not depend on netx and urlgetter by @bassosimone in #792
- chore: add gosec step by @ainghazal in #793
- chore: improve testing and increase coverage by @bassosimone in #794
- fix(netx): ensure we create ~same HTTP3 and HTTP2 transports by @bassosimone in #795
- cleanup(netxlite): drop the DefaultDialer legacy name by @bassosimone in #796
- cleanup(netx): stop using most netxlite resolver internals by @bassosimone in #797
- refactor(netx): move construction logic outside package by @bassosimone in #798
- cleanup: move caching resolvers from netx to netxlite by @bassosimone in #799
- refactor(netx): reorganize by topic by @bassosimone in #800
- cleanup: netx does not use netxlite legacy names by @bassosimone in #801
- doc(netx): reference issue mentioning future improvements by @bassosimone in #802
- [forwardport] fix(dnscheck): remove apparently-broken static input by @bassosimone in #804
- [forwardport] fix(stunreachability): don't emit spurious progress events by @bassosimone in #803
- refactor(oohelper): use netxlite rather than netx by @bassosimone in #805
- refactor(oohelperd): use netxlite rather than netx by @bassosimone in #806
- refactor: start refactoring session resolver by @bassosimone in #807
- hotfix(sessionresolver): prevent data race inside http3 by @bassosimone in #809
- cleanup: use NewHTTPTransportWithResolver more often by @bassosimone in #808
- refactor(sessionresolver): minor changes in files and types naming by @bassosimone in #810
- refactor(sessionresolver): replace dnsclientmaker with a function by @bassosimone in #811
- refactor(netxlite): more abstract proxy-enabled dialer construction by @bassosimone in #812
- refactor(netxlite): expose useful HTTPTransport/DNSTransport factories by @bassosimone in #813
- doc: publish the step-by-step design document by @bassosimone in #814
- chore: use go1.18.3 by @bassosimone in #816
- chore: avoid duplicating the code of conduct by @bassosimone in #818
- doc(step-by-step): readability improvements by @ainghazal in #820
- fix(oonimkall): only set annotations on success by @bassosimone in #821
- fix(workflows): elevate GITHUB_TOKEN permissions when needed by @bassosimone in #822
- feat: tlsping and tcpping using step-by-step by @bassosimone in #815
- doc(step-by-step): further improvements on design doc by @ainghazal in #830
- fix(oohelperd): use throw-away HTTPClient, Dialer, Resolver by @bassosimone in #833
- refactor(oohelperd): flatten package hierarchy by @bassosimone in #834
- refactor(oohelperd): improve tests implementation by @bassosimone in #835
- refactor(engine): *http.Client -> model.HTTPClient by @bassosimone in #836
- refactor(engine): set options from any value by @bassosimone in #837
- measurexlite: fix flaky tls_test.go by @bassosimone in #839
- refactor(engine): more abstract Experiment{,Builder} by @bassosimone in #838
- cleanup(miniooni): remove --limit option by @bassosimone in #840
- fix(engine): repair broken integration test by @bassosimone in #841
- feat: start sketching out the oonirun package by @bassosimone in #842
- feat(oonirun): implement OONIRun v1 by @bassosimone in #843
- feat(oonirun): add support for OONIRun v2 links by @bassosimone in #844
- feat(miniooni): introduce the --repeat-every command line flag by @bassosimone in #819
- feat: dnsping using step-by-step by @DecFox in #831
- feat: collect system resolver results using context by @DecFox in #856
- feat: refactor dns implementation in measurexlite by @DecFox in #857
- feat: add support for system resolver in measurexlite by @DecFox in #858
- refactor(simplequicping): using step-by-step by @DecFox in #852
- refactor(engine): allow scripts to register experiments by @bassosimone in #860
- cleanup: define required Go version just once by @bassosimone in #861
- fix(registry): ensure we can run telegram and webconnectivity by @bassosimone in #862
- refactor(mk): move build rules into separate scripts by @bassosimone in #855
- cleanup: chmod -x mk && mv mk Makefile by @bassosimone in #863
- feat: avoid safe options to be serialized into the measurement by @ainghazal in #859
- fix(Makefile): re-enable Android builds by @bassosimone in #865
- fix(MOBILE/android): make scripts monorepo compatible by @bassosimone in #866
- refactor(measurexlite): make buffered channels private by @DecFox in #864
- refactor: allow automatically wrap net/quic conn by @DecFox in #867
- chore: upgrade deps and attempt to enable using go1.19 by @bassosimone in #869
- fix: disable psiphon when building with go1.19 by @bassosimone in #871
- chore: use {go,oohttp,oocrypto} v1.18.5 by @bassosimone in #872
- feat: context-based tracing to record delayed DNS responses by @DecFox in #870
- cleanup: remove UnderlyingNetworkLibrary and TProxy by @bassosimone in #874
- feat(netxlite): support extracting the CNAME by @bassosimone in #875
- feat(dnsovergetaddrinfo): collect the CNAME by @bassosimone in #876
- feat: save CNAME into archival data format by @bassosimone in #877
- feat(dns): expose more low-level fields by @bassosimone in #873
- feat(measurexlite): add T0 and TransactionID by @bassosimone in #879
- feat: record delayed DNS responses in dnsping by @DecFox in #878
- feat(measurexlite): generate HTTP traces by @bassosimone in #881
- feat(webconnectivity): long-term-evolution prototype by @bassosimone in #882
- feat([email protected]): record late DNS replies by @bassosimone in #883
- feat: clearly indicate which resolver we're using by @bassosimone in #885
- fix(oohelperd): by default listen on localhost by @bassosimone in #887
- fix(oohelperd): enforce timeout for each measurement step by @bassosimone in #888
- refactor(oohelperd): make performing additional measurements easier by @bassosimone in #889
- feat(oohelperd): follow (and record) TH and probe endpoints by @bassosimone in #890
- feat(oohelperd): measure TLS for :443 endpoints by @bassosimone in #886
- refactor: spin geoipx off geolocate by @bassosimone in #893
- refactor: move TH structs and definitions to model by @bassosimone in #894
- refactor: move WebGetTitle inside measurexlite by @bassosimone in #895
- refactor: make measurex depend on measurexlite by @bassosimone in #892
- feat(oohelperd): log messages at info level by @bassosimone in #896
- feat(oohelperd): add prometheus metrics by @bassosimone in #897
- fix(oohelperd): refuse to measure bogons by @bassosimone in #898
- chore: we're now hacking on v3.16.0-alpha.1 by @bassosimone in #899
- fix: repair releasing miniooni and ooniprobe-windows by @bassosimone in #900
- chore: we're now hacking on v3.16.0-alpha.2 by @bassosimone in #901
- fix: repair broken linux-static build by @bassosimone in #902
- fix(oohelperd): metrics improvements after design review by @bassosimone in #903
- feat: build ooniprobe for armv6 by @bassosimone in #904
- fix: repair the Android build by @bassosimone in #905
- fix(oohelperd): measurement -> wctask by @bassosimone in #906
- feat: build miniooni and ooniprobe for android by @bassosimone in #907
- feat: stop cross compiling miniooni by @bassosimone in #908
- fix: attempt to make android builds faster by @bassosimone in #909
- feat: implement rolling builds by @bassosimone in #910
- fix: attempt to make linux builds faster by @bassosimone in #911
- refactor(miniooni): divide et impera by @bassosimone in #912
- feat(miniooni): make CLI much more user friendly by @bassosimone in #913
- fix(miniooni): handle panics with --repeat-every by @bassosimone in #914
- feat(oonirun): improve tests by @bassosimone in #915
- doc: document the minioonirunv2 functionality by @bassosimone in #916
- fix(dnscheck): lower the default timeouts by @bassosimone in #917
- dnscheck: bump patch version number by @bassosimone in #919
- feat(rolling): build fresh binaries every night by @bassosimone in #920
- fix(dnscheck): record whether residual is enabled by @bassosimone in #922
- fix(dnscheck): trim the static input list by @bassosimone in #923
- cleanup: doh.powerdns.org is not working anymore by @bassosimone in #924
- fix(tracex): use HTTP transaction end time for
t
by @bassosimone in #925 - fix(probeservices): use api.ooni.io by @bassosimone in #926
- feat(ghpublish): choose (pre)release depending on tag by @bassosimone in #928
- chore: run go generate ./... by @bassosimone in #929
- chore: upgrade to db-ip.com 2022-09 release by @bassosimone in #930
- chore: upgrade user-agent used for measuring by @bassosimone in #931
- feat(miniooni): optionally log using emojis by @bassosimone in #932
- feat([email protected]): use TLS info from TH by @bassosimone in #933
- fix(measurexlite): expose TCP connect event by @bassosimone in #934
- fix([email protected]): fetch HTTP only using system-resolver addrs by @bassosimone in #935
- chore: update to ooni/go-libtor v1.1.6 by @bassosimone in #936
- chore: use go1.18.6 by @bassosimone in #937
- QA: refactor, cleanup, fix, and simplify by @bassosimone in #938
Full Changelog: v3.15.3...v3.16.0