Skip to content
This repository has been archived by the owner on Apr 14, 2022. It is now read-only.

Merge from master (February 28th, 2019) #107

Merged

Conversation

pquentin
Copy link
Member

This pull request is mostly about TLS 1.3 support.

TLS 1.3 works without this PR, but this changes tests and adds support in two SSL backends: pyOpenSSL and SecureTransport. We don't support them yet, and did not modify them much, so the merges were mostly automatic.

I also bumped the macOS Python releases we use so that Python 3.7 is tested with OpenSSL 1.1.1 (and TLS 1.3) support.

I don't understand the post-handshake authentication (PHA) comment in the pyOpenSSL backend, but I imagine it was added because of psf/requests#4961 (comment). In any case, I'm not sure why the comment mentions PHA: at the time of the commit, PHA was not enabled in urllib3, and not yet supported by pyOpenSSL anyway: pyca/pyopenssl#793.

I fixed a bug that's also in the urllib3 master branch (see urllib3/urllib3#1664). It was hard to notice because OpenSSL 1.1.1 / TLS 1.3 using Python's ssl module was not tested in CI until now. It took me a long time to fix for two reasons: 1/ I did not have OpenSSL 1.1.1 support in my Python build, so I was testing on CI, which takes a long time (I ended up compiling Python from source) 2/ the fixed test was using an unexpected certificate which led me to think that the bug was #106

sethmlarson and others added 4 commits February 27, 2019 13:54
* Add tests for specific TLS/SSL versions

* Add change and update bindings

* SSLSocket.version() not available sometimes

* Add support for kTLSProtocolMaxSupported

* Try setProtocolVersionMax again if error

* Get ctypes.c_uint.value for SSLSocket.version()

* Opt-in TLS 1.3 on macOS 10.13

* Update tornado to 5.1.1

* Add documentation updates for TLSv1.3

* Add wbond/oscrypto license to contrib/securetransport

* Remove all TLS 1.3 ciphersuites from DEFAULT_CIPHERS

* Experiment showing cipher list per protocol

* Update test_https.py

* Update test_https.py

* Update test_https.py

* Update changelog wording to exclude pyOpenSSL

* minor rewording

* Add support for IPv6 in subjectAltName

* Don't use OP_ALL

* Update CHANGES.rst

* No PROTOCOL_TLSv1_3

* Remove DSS, rearrange SecureTransport ciphers

* Use ECDSA before RSA with ECDHE

* ReviReorder ciphers

* ECDHE

* Update test_https.py

* Turns out we don't need version detection

* Reorder per Hyneks post and favor ephemeral

* Refactor HTTPS unit tests

* Fix up tests

* Test locking pytest-httpbin

* Update requests.sh

* remove whitespace
@njsmith
Copy link
Member

njsmith commented Aug 17, 2019

Travis had three failures, and they all involve timeouts, so I suspect they're flaky tests:

One on pypy3.6-7.1.1:

________________ TestConnectionPoolTimeouts.test_timeout_float _________________

Traceback (most recent call last):

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/_sync/connectionpool.py", line 412, in _make_request

    response = conn.send_request(request, read_timeout=read_timeout)

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/_sync/connection.py", line 411, in send_request

    request, self._state_machine, self._sock, read_timeout

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/_sync/connection.py", line 285, in _start_http_request

    produce_bytes, consume_bytes, read_timeout)

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/_backends/sync_backend.py", line 127, in send_and_receive_for_a_while

    self._wait(want_read, want_write, read_timeout)

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/_backends/sync_backend.py", line 50, in _wait

    raise socket.timeout()  # XX use a backend-agnostic exception

_socket.timeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/travis/build/python-trio/urllib3/test/with_dummyserver/test_connectionpool.py", line 64, in test_timeout_float

    pool.request('GET', '/')

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/_sync/request.py", line 70, in request

    **urlopen_kw)

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/_sync/request.py", line 91, in request_encode_url

    return self.urlopen(method, url, **extra_kw)

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/_sync/connectionpool.py", line 604, in urlopen

    _stacktrace=sys.exc_info()[2])

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/util/retry.py", line 344, in increment

    raise six.reraise(type(error), error, _stacktrace)

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/packages/six.py", line 686, in reraise

    raise value

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/_sync/connectionpool.py", line 572, in urlopen

    headers=headers)

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/_sync/connectionpool.py", line 414, in _make_request

    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)

  File "/home/travis/build/python-trio/urllib3/.tox/pypy3/site-packages/urllib3/_sync/connectionpool.py", line 334, in _raise_timeout

    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)

urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=44537): Read timed out. (read timeout=0.001)

And two on the macOS 3.7 build:

_______________ TestSocketClosing.test_delayed_body_read_timeout _______________

Traceback (most recent call last):

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connectionpool.py", line 412, in _make_request

    response = conn.send_request(request, read_timeout=read_timeout)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connection.py", line 411, in send_request

    request, self._state_machine, self._sock, read_timeout

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connection.py", line 285, in _start_http_request

    produce_bytes, consume_bytes, read_timeout)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_backends/sync_backend.py", line 127, in send_and_receive_for_a_while

    self._wait(want_read, want_write, read_timeout)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_backends/sync_backend.py", line 50, in _wait

    raise socket.timeout()  # XX use a backend-agnostic exception

socket.timeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connectionpool.py", line 572, in urlopen

    headers=headers)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connectionpool.py", line 414, in _make_request

    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connectionpool.py", line 334, in _raise_timeout

    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)

urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=50120): Read timed out. (read timeout=0.01)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/travis/build/python-trio/urllib3/test/with_dummyserver/test_socketlevel.py", line 491, in test_delayed_body_read_timeout

    timeout=Timeout(connect=1, read=0.01))

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connectionpool.py", line 604, in urlopen

    _stacktrace=sys.exc_info()[2])

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment

    raise MaxRetryError(_pool, url, error or ResponseError(cause))

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=50120): Max retries exceeded with url: / (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=50120): Read timed out. (read timeout=0.01)"))

_____________ TestSocketClosing.test_timeout_errors_cause_retries ______________

Traceback (most recent call last):

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connectionpool.py", line 412, in _make_request

    response = conn.send_request(request, read_timeout=read_timeout)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connection.py", line 411, in send_request

    request, self._state_machine, self._sock, read_timeout

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connection.py", line 285, in _start_http_request

    produce_bytes, consume_bytes, read_timeout)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_backends/sync_backend.py", line 127, in send_and_receive_for_a_while

    self._wait(want_read, want_write, read_timeout)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_backends/sync_backend.py", line 50, in _wait

    raise socket.timeout()  # XX use a backend-agnostic exception

socket.timeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connectionpool.py", line 572, in urlopen

    headers=headers)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connectionpool.py", line 414, in _make_request

    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connectionpool.py", line 334, in _raise_timeout

    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)

urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=50145): Read timed out. (read timeout=0.01)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/travis/build/python-trio/urllib3/test/with_dummyserver/test_socketlevel.py", line 462, in test_timeout_errors_cause_retries

    response = pool.request('GET', '/', retries=1)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/request.py", line 70, in request

    **urlopen_kw)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/request.py", line 91, in request_encode_url

    return self.urlopen(method, url, **extra_kw)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connectionpool.py", line 632, in urlopen

    body_pos=body_pos, **response_kw)

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/_sync/connectionpool.py", line 604, in urlopen

    _stacktrace=sys.exc_info()[2])

  File "/Users/travis/build/python-trio/urllib3/.tox/py37/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment

    raise MaxRetryError(_pool, url, error or ResponseError(cause))

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=50145): Max retries exceeded with url: / (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=50145): Read timed out. (read timeout=0.01)"))

------------------------------ Captured log call -------------------------------

connectionpool.py          628 WARNING  Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=50145): Read timed out. (read timeout=0.01)")': /

I'll click the restart button and see if it fixes anything.

@codecov
Copy link

codecov bot commented Aug 17, 2019

Codecov Report

Merging #107 into bleach-spike will increase coverage by <.01%.
The diff coverage is 100%.

@@               Coverage Diff                @@
##           bleach-spike     #107      +/-   ##
================================================
+ Coverage         99.46%   99.46%   +<.01%     
================================================
  Files                29       29              
  Lines              1852     1854       +2     
================================================
+ Hits               1842     1844       +2     
  Misses               10       10
Impacted Files Coverage Δ
src/urllib3/util/__init__.py 100% <ø> (ø) ⬆️
src/urllib3/_backends/sync_backend.py 98.83% <100%> (+0.02%) ⬆️
src/urllib3/util/ssl_.py 100% <100%> (ø) ⬆️

@njsmith
Copy link
Member

njsmith commented Aug 17, 2019

Ta-da, now the CI is green. Well, at least that gets us through this PR.

Well done figuring out that obscure bug!

@njsmith njsmith merged commit c234016 into python-trio:bleach-spike Aug 17, 2019
@pquentin pquentin deleted the merge-from-master-2019-02-28 branch August 19, 2019 09:24
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants