Fails with Let's Encrypt ECDSA certificate chain

[dhalbert]

https://api.intra.io/api/v1/healthz, which has a Let's Encrypt cert chain, causes failure. It is Cloudflare based.

The cert chain as reported by Firefox is:
*.intra.io server cert <- Let's Encrypt E1 <- ISRG Root X2 (using elliptic keys)

This contrasts with the working Let's Encrypt chain:
End-entity certificate <- Let's Encrypt R3 intermediate cert <- ISRG Root X1 (special cross-signed version) <- DST Root CA X3
(see adafruit/certificates#1 for background)

The URL above does work on native Espressif wifi, like ESP32-S2.

Turning on debug tracing shows what might be a crash:

Fetching and parsing json from https://api.intra.io/api/v1/healthz
*** Get host by name
*** Get socket
Allocated socket #0
*** Socket connect mode 2
*** Open socket to api.intra.io 443 2
*** Closing socket #0
Traceback (most recent call last):
  File "adafruit_requests.py", line 515, in _get_socket
  File "adafruit_requests.py", line 748, in connect
  File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 77, in connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 819, in socket_connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 715, in socket_open
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 341, in _send_command_get_response
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 297, in _wait_response_cmd
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 206, in _wait_for_ready
TimeoutError: ESP32 not responding

The cert chain that doesn't work uses elliptic keys, but those encryption methods seem to be turned on in the sdkconfig here.

Thanks to @crhuber for reporting this. See #55 (comment) and following.

@tyeth you mentioned two URL's you are using, one with each chain. If you have time, could you verify that one works and one fails, in line with the above?

[dhalbert]

Fails with #57 (labeled as 1.7.7 tentatively), and also 1.7.4 and 1.7.5 NINA-FW builds, so it's not a new cert problem.

[crhuber]

@dhalbert the only thing i can suspect about this is that it is some CloudFlare related thing like the size of the headers, http protocol config

[dhalbert]

I found out Let's Encrypt has set up test websites that use both of its chains. See https://letsencrypt.org/certificates/, and look under "Root Certificates" for "We've set up websites..." to find them.

https://valid-isrgrootx1.letsencrypt.org/
chain: server <- Let's Encrypt R3 intermediate <- ISRG Root X1 (and to CA Root CA X3 if needed)
uses RSA keys

https://valid-isrgrootx2.letsencrypt.org/
chain: server <- Let's Encrypt E1 intermediate <- ISRG Root X2
uses ECDSA keys

The first server works fine, as expected. The second fails in the same way that api.intra.io fails. These servers don't seem to use CloudFlare -- they are straightforwardly configured.

So it appears the problem has to do with this particular cert chain, maybe because it is ECDSA.

[dhalbert]

I tried the test servers using an Arduino Nano 33 IOT, with its https://github.com/arduino/nina-fw firmware updated to 1.5.0, the latest version. As with our test, https://valid-isrgrootx1.letsencrypt.org/ worked, but https://valid-isrgrootx2.letsencrypt.org/ did not. I was using the WiFiSSLClient example program. When trying the valid-isrgrootx2.letsencrypt.org server, the example program simply printed:

Attempting to connect to SSID: PBE2
Connected to WiFi
SSID: [redacted]
IP Address: [redacted]
signal strength (RSSI):-46 dBm

Starting connection to server...

disconnecting from server.

By comparison, with valid-isrgrootx1.letsencrypt.org, the HTML of the home page was printed out.

So upgrading Adafruit's NINA-FW to Arduino's version is not going to fix this problem. EDIT: it does work, with proper roots.pem. See below

tagging @ladyada for interest

[tyeth]

@dhalbert I did plan to have a quick look, but got distracted while looking at the linked issues (esp-idf or esp-arduino one) and the minimum test case they asked for... I found both the sites worked with just the esp mbedtls CMN(common) certs included using the x509 example with latest master of esp-idf.
I haven't tried out our cert bundle specifically in esp-idf rather than CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_CMN=y.
As you suggested about the cyphers, I noticed when looking at my E1 version (.cloud is E1 - 4deep, .technology is R3 - 3deep), that the weaker clients preferred CHACHA which I'd happen to notice while Scott was doing a deepdive and turning off some cyphers to save space. I was going to enable these and retry CPY.
https://www.ssllabs.com/ssltest/analyze.html?d=good%2denough.cloud&s=2606%3a4700%3a3032%3a0%3a0%3a0%3a6815%3a1c3e&latest
[Wait for 100% then goto bottom where it shows simulated clients and the cyphers they'd end up with]

CONFIG_MBEDTLS_POLY1305_C=y
CONFIG_MBEDTLS_CHACHA20_C=y
CONFIG_MBEDTLS_CHACHAPOLY_C=y

Sorry for the edits and brain-half-dump, needed to rid the thoughts before bed.

[dhalbert]

@tyeth Since https://valid-isrgrootx2.letsencrypt.org/ works with regular ESP32 and ESP32-xx, but does not work with either adafruit or arduino NINA-FW, we might look to see if there are some differences in the settings between the sdkconfig here and the one in CircuitPython. But the one here does turn on ECSDA in various forms, and I don't see a suspicious =n.

[crhuber]

@dhalbert any updates on this one?

[dhalbert]

@crhuber After further testing, I found that arduino NINA-FW does work, when the proper root certificates are included. We think we will rebase our fork of NINA-FW against the latest arduino NINA-FW and add back the features we added. But this work has not been started yet.

We forked a long time ago and there have been many changes to arduino NINA-FW since then. Our code changes are relatively minor.

[terop]

@dhalbert any update on this? I have the same problem as in issue #66.
It looks like the upstream sync effort in PR #62 isn't going anywhere, not at least very quickly.

[dhalbert]

I have asked about #62 but haven't gotten a reply yet.