Deploying with remote having an expired certificate fails

[vvuk]

Hrm, I'm a bit stuck. I let my cert on my TrueNAS box expire. Now when I try to deploy an updated one, I get a SSL verification error from the API call, whether I try to do the update from the web front end or the script. This isn't a problem with this script as such, but it would be nice if it at least gave a hint on how to resolve this scenario. The only thing that I came up with (that worked) was to ssh into the server, run date to force the date to be somewhere within the cert valid period, quickly set the cert, and then run ntpdate pool.ntp.org to reset the date.

For reference, the actual error is below. Perhaps if the script sees "certificate has expired" in the output it could suggest the above steps?

Certificate import successful
Certificate list successful
Error setting active certificate!
{
 "message": "Cannot connect to host 127.0.0.1:6443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123)')]",
 [manually better formatted traceback]

"Traceback (most recent call last):
  File \"/usr/lib/python3/dist-packages/aiohttp/connector.py\", line 969, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File \"/usr/lib/python3.9/asyncio/base_events.py\", line 1081, in create_connection
    transport, protocol = await self._create_connection_transport(
  File \"/usr/lib/python3.9/asyncio/base_events.py\", line 1111, in _create_connection_transport
    await waiter
  File \"/usr/lib/python3.9/asyncio/sslproto.py\", line 528, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File \"/usr/lib/python3.9/asyncio/sslproto.py\", line 188, in feed_ssldata
    self._sslobj.do_handshake()
  File \"/usr/lib/python3.9/ssl.py\", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File \"/usr/lib/python3/dist-packages/middlewared/restful.py\", line 788, in do
    result = await self.middleware.call(methodname, *method_args, **method_kwargs)
  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1368, in call
    return await self._call(
  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1317, in _call
    return await methodobj(*prepared_call.args)
  File \"/usr/lib/python3/dist-packages/middlewared/service.py\", line 576, in update
    rv = await self.middleware._call(
  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1317, in _call
    return await methodobj(*prepared_call.args)
  File \"/usr/lib/python3/dist-packages/middlewared/schema.py\", line 1247, in nf
    res = await f(*args, **kwargs)
  File \"/usr/lib/python3/dist-packages/middlewared/schema.py\", line 1379, in nf
    return await func(*args, **kwargs)
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/system_general/update.py\", line 255, in do_update
    verrors = await self.validate_general_settings(new_config, 'general_settings_update')
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/system_general/update.py\", line 115, in validate_general_settings
    verrors.extend(await validate_port(
  File \"/usr/lib/python3/dist-packages/middlewared/async_validators.py\", line 111, in validate_port
    return await middleware.call('port.validate_port', schema, port, bind_ip, whitelist_namespace)
  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1368, in call
    return await self._call(
  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1317, in _call
    return await methodobj(*prepared_call.args)
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/ports/ports.py\", line 59, in validate_port
    port_mapping = await self.ports_mapping(whitelist_namespace)
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/ports/ports.py\", line 102, in ports_mapping
    for attachment in filter(lambda entry: entry['namespace'] != whitelist_namespace, await self.get_in_use()):
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/ports/ports.py\", line 41, in get_in_use
    used_ports = await delegate.get_ports()
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/attachments.py\", line 12, in get_ports
    for chart_release in filter(lambda c: c['used_ports'], await self.middleware.call('chart.release.query')):
  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1368, in call
    return await self._call(
  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1317, in _call
    return await methodobj(*prepared_call.args)
  File \"/usr/lib/python3/dist-packages/middlewared/schema.py\", line 1247, in nf
    res = await f(*args, **kwargs)
  File \"/usr/lib/python3/dist-packages/middlewared/schema.py\", line 1379, in nf
    return await func(*args, **kwargs)
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py\", line 106, in query
    if not await self.middleware.call('kubernetes.validate_k8s_setup', False):
  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1368, in call
    return await self._call(
  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1317, in _call
    return await methodobj(*prepared_call.args)
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py\", line 508, in validate_k8s_setup
    await Node.get_instance(request_kwargs={'timeout': 2})
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/core_api.py\", line 27, in get_instance
    return await super().get_instance(NODE_NAME, **kwargs)
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py\", line 88, in get_instance
    instance = await cls.query(
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py\", line 99, in query
    return await cls.call(
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py\", line 84, in call
    return await cls.api_call(uri, mode, body, headers, **kwargs)
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py\", line 45, in api_call
    async with cls.request(endpoint, mode, body, headers, timeout) as resp:
  File \"/usr/lib/python3.9/contextlib.py\", line 175, in __aenter__
    return await self.gen.__anext__()
  File \"/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py\", line 29, in request
    async with await getattr(session, mode)(
  File \"/usr/lib/python3/dist-packages/aiohttp/client.py\", line 520, in _request
    conn = await self._connector.connect(
  File \"/usr/lib/python3/dist-packages/aiohttp/connector.py\", line 535, in connect
    proto = await self._create_connection(req, traces, timeout)
  File \"/usr/lib/python3/dist-packages/aiohttp/connector.py\", line 892, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File \"/usr/lib/python3/dist-packages/aiohttp/connector.py\", line 1051, in _create_direct_connection
    raise last_exc
  File \"/usr/lib/python3/dist-packages/aiohttp/connector.py\", line 1020, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File \"/usr/lib/python3/dist-packages/aiohttp/connector.py\", line 971, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host 127.0.0.1:6443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123)')]

[danb35]

That's (part of) the reason for the verify setting in deploy_config. If you set that to false, you should be able to connect without further issue.

[vvuk]

It's not -- that only controls the ssl verification for this script's own HTTPS connection. What's happening is the TrueNAS backend API call (i.e. the thing that happens in response to the /api/v2.0/system/general call to set the cert) is making another call back to 127.0.0.1, and that verify is failing. I'm actually running the cert update script on another host entirely.

[vvuk]

(Note above "Certificate import successful"/list/etc -- the script itself can make API calls just fine)