[TlsInterception] Breaks ModifyChunkResponsePlugin in v2.3.1

[Telemattic]

Try to use the ModifyChunkResponsePlugin together with TLS Intercept: while the parts work separately, they don't work together.

To Reproduce
Steps to reproduce the behavior:

1. Make certificates, as specified for TLS intercept: make ca-certificates
2. Verify that the proxy works for pure TLS intercept:
3. -- Run proxy.py as python3 -m proxy --ca-key-file ca-key.pem --ca-cert-file ca-cert.pem --ca-signing-key-file ca-signing-key.pem --hostname 127.0.0.1
4. -- Fetch a data drip: curl -v -x localhost:8899 --cacert ca-cert.pem "https://httpbin.org/drip?duration=2&numbytes=29&code=200&delay=2"
5. -- Successfully get the data
6. Verify that the proxy works for just ModifyChunkResponsePlugin, without TLS intercept:
7. -- Run proxy.py as python3 -m proxy --plugins proxy.plugin.ModifyChunkResponsePlugin --hostname 127.0.0.1
8. -- Fetch a data drip: curl -v -x localhost:8899 "http://httpbin.org/drip?duration=2&numbytes=29&code=200&delay=2"
9. -- Successfully get the data
10. Combine TLS intercept and ModifyChunkResponsePlugin:
11. -- Run proxy.py as python3 -m proxy --ca-key-file ca-key.pem --ca-cert-file ca-cert.pem --ca-signing-key-file ca-signing-key.pem --plugin proxy.plugin.ModifyChunkResponsePlugin --hostname 127.0.0.1
12. -- Fetch a data drip: curl -v -x localhost:8899 --cacert ca-cert.pem "https://httpbin.org/drip?duration=2&numbytes=29&code=200&delay=2"
13. -- Don't get the data

Expected behavior
TLS intercept and ModifyChunkResponsePlugin should compose

Version information

• OS: ubuntu
• Browser: curl/7.68.0
• Device: linux desktop
• proxy.py Version 2.3.1

Additional context
Note I was being clever above and specifying that drip should return 29 bytes, which is the same as the length of the modified content, this sidesteps issue #1039

I found a fix for this problem, although whether this is fixing the root cause or merely a symptom I couldn't say. I changed TCPConnection.flush to not do a send on a zero length buffers, instead it just pops them. For whatever reason doing the send of a zero length buffer raises an OSError somewhere in the SSL code.

I've attached the logfiles for the repro starting at step 10 (--log-file proxy.txt added to the proxy, 2> curl.txt added to the curl)

proxy.txt
curl.txt

[Telemattic]

Here's a patch that appears to fix the issue. I made TCPConnection.flush() not do a send of zero length buffers
patch.txt

[abhinavsingh]

Thank you reporting it. I remember encountering similar issue when adding integration tests for web server plugin, likely you already discovered #1028 . Looks like this bug can also effects proxy plugins. I have updated the bug to highlight the same.

Btw if you are using https://httpbin.org/drip?duration=2&numbytes=29&code=200&delay=2 for chunked response plugin, it ain't gonna work. Response is not chunked. E.g. see below:

$ curl -v https://httpbin.org/drip\?duration\=2\&numbytes\=29\&code\=200\&delay\=2            ─╯
*   Trying 34.234.159.18:443...
* Connected to httpbin.org (34.234.159.18) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=httpbin.org
*  start date: Nov 21 00:00:00 2021 GMT
*  expire date: Dec 19 23:59:59 2022 GMT
*  subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc54c80d400)
> GET /drip?duration=2&numbytes=29&code=200&delay=2 HTTP/2
> Host: httpbin.org
> user-agent: curl/7.77.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< date: Sat, 22 Jan 2022 19:51:59 GMT
< content-type: application/octet-stream
< content-length: 29
< server: gunicorn/19.9.0
< access-control-allow-origin: *
< access-control-allow-credentials: true
< 
* Connection #0 to host httpbin.org left intact
*****************************%   

[abhinavsingh]

Answer I posted here actually uses TLS interception. We also have integration tests under TLS interception mode. Ref #1039 (comment)