Sync with upstream

[hathach]

sync with upstream, try to keep as close as upstream as possible while reserved as much modification as possible (comparing to upstream arduino/nina-fw@master...adafruit:nina-fw:master). File changes may look a lot (376) but most of them are new file addtion. Also I don't know why PR commit start with initial one (weird).

• CommandHandler.cpp our setClientCert() is conflicted with upstream's setEnt(). Both is located at command = 0x40 --> use our command, skip upstream.
• WiFISSLClient.cpp where most our changes is located, minor changes to debug message, move thing around a bit to as close as upstream. Add note for heavy heap usage ssl function, since we have issued with lacking heap here (solved by reduce max sockets)
• sdkconfig reduce max socket so that ssl can work (otherwise we got malloc failed) and also change setting to match upstream
  • CONFIG_LWIP_MAX_SOCKETS=8 (previously 10), required for ssl socket to work !!!
  • CONFIG_ESP32_DEFAULT_CPU_FREQ_MHZ=160 (previously 80)
  • CONFIG_TCPIP_TASK_STACK_SIZE=2560 (previously 2048)
  • CONFIG_BTDM_CONTROLLER_BLE_MAX_CONN=CONFIG_BTDM_CONTROLLER_BLE_MAX_CONN_EFF=7 (previously 3)
• Lots of small changes, but we are very close to upstream for. Hopefully that make sync up in the future easier

I only take a few examples from WiFiNINA: WiFiSSLClient, WiFiWebClient. @danh @brentru @ladyada please help to review and test this out. If you have any sophisticated that test most of nina function, please let me know (I am not too familliar with nina). Once merged, we can start another PR to upgrade IDF from v3 to v5

PS: just tested with valid-isrgrootx2.letsencrypt.org (previously fixed by #57) and got -0x2700 (MBEDTLS_ERR_X509_CERT_VERIFY_FAILED). Will try to fix and more revise later.

PS2: ci is updated to also build with debug message ON, uploaded as NINA_W102-*_debug.bin in debug matrix for testing convenience.

[ladyada]

o_O wow OK! nice work
cc @brentru @dhalbert

[dhalbert]

@hathach I didn't look at this carefully enough. Are you waiting for us to test this? You mentioned it's not working with that Let's Encrypt site. It's using the certs submodule now, right?

[hathach]

@hathach I didn't look at this carefully enough. Are you waiting for us to test this? You mentioned it's not working with that Let's Encrypt site. It's using the certs submodule now, right?

@dhalbert yeah, I was waiting for your feedback on wip port before making more changes to address the encrypted site issue. Then got switched to other works, and kind of forgot about this issue. I will try to put it back for more tests.

[justmobilize]

@dhalbert so I just tried this on a Matrix Portal M4 and can't get and SSL connect to work (both release and debug):

New

Adafruit CircuitPython 9.0.4 on 2024-04-16; Adafruit Matrix Portal M4 with samd51j19
>>>
paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== import board
=== import busio
=== import digitalio
=== import os
=== import adafruit_connection_manager
=== from adafruit_esp32spi.adafruit_esp32spi import ESP_SPIcontrol
===
=== esp32_cs = digitalio.DigitalInOut(board.ESP_CS)
=== esp32_ready = digitalio.DigitalInOut(board.ESP_BUSY)
=== esp32_reset = digitalio.DigitalInOut(board.ESP_RESET)
=== spi_bus = busio.SPI(board.SCK, MOSI=board.MOSI, MISO=board.MISO)
=== radio = ESP_SPIcontrol(spi_bus, esp32_cs, esp32_ready, esp32_reset)
=== print(f"firmware: {radio.firmware_version}")
===
=== radio.connect_AP(os.getenv("CIRCUITPY_WIFI_SSID"), os.getenv("CIRCUITPY_WIFI_PASSWORD"))
===
=== pool = adafruit_connection_manager.get_radio_socketpool(radio)
=== ssl_context = adafruit_connection_manager.get_radio_ssl_context(radio)
=== connection_manager = adafruit_connection_manager.get_connection_manager(pool)
===
=== print("Try http://wifitest.adafruit.com")
=== socket = connection_manager.get_socket("wifitest.adafruit.com", 80, "http:", is_ssl=False, ssl_context=ssl_context, timeout=60)
=== connection_manager.close_socket(socket)
===
=== print("Try https://io.adafruit.com")
=== socket = connection_manager.get_socket("io.adafruit.com", 443, "https:", is_ssl=True, ssl_context=ssl_context, timeout=60)
=== connection_manager.close_socket(socket)
===
firmware: bytearray(b'1.8.0\x00')
3
Try http://wifitest.adafruit.com
Try https://io.adafruit.com
Traceback (most recent call last):
  File "adafruit_connection_manager.py", line 271, in get_socket
  File "adafruit_connection_manager.py", line 63, in connect
  File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 77, in connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 819, in socket_connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 715, in socket_open
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 341, in _send_command_get_response
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 297, in _wait_response_cmd
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 206, in _wait_for_ready
TimeoutError: ESP32 not responding

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 26, in <module>
  File "adafruit_connection_manager.py", line 282, in get_socket
RuntimeError: Error connecting socket: ESP32 not responding
>>>

Current

Adafruit CircuitPython 9.0.4 on 2024-04-16; Adafruit Matrix Portal M4 with samd51j19
>>>
paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== import board
=== import busio
=== import digitalio
=== import os
=== import adafruit_connection_manager
=== from adafruit_esp32spi.adafruit_esp32spi import ESP_SPIcontrol
===
=== esp32_cs = digitalio.DigitalInOut(board.ESP_CS)
=== esp32_ready = digitalio.DigitalInOut(board.ESP_BUSY)
=== esp32_reset = digitalio.DigitalInOut(board.ESP_RESET)
=== spi_bus = busio.SPI(board.SCK, MOSI=board.MOSI, MISO=board.MISO)
=== radio = ESP_SPIcontrol(spi_bus, esp32_cs, esp32_ready, esp32_reset)
=== print(f"firmware: {radio.firmware_version}")
===
=== radio.connect_AP(os.getenv("CIRCUITPY_WIFI_SSID"), os.getenv("CIRCUITPY_WIFI_PASSWORD"))
===
=== pool = adafruit_connection_manager.get_radio_socketpool(radio)
=== ssl_context = adafruit_connection_manager.get_radio_ssl_context(radio)
=== connection_manager = adafruit_connection_manager.get_connection_manager(pool)
===
=== print("Try http://wifitest.adafruit.com")
=== socket = connection_manager.get_socket("wifitest.adafruit.com", 80, "http:", is_ssl=False, ssl_context=ssl_context, timeout=60)
=== connection_manager.close_socket(socket)
===
=== print("Try https://io.adafruit.com")
=== socket = connection_manager.get_socket("io.adafruit.com", 443, "https:", is_ssl=True, ssl_context=ssl_context, timeout=60)
=== connection_manager.close_socket(socket)
===
firmware: bytearray(b'1.7.7\x00')
3
Try http://wifitest.adafruit.com
Try https://io.adafruit.com
>>>