Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSL: CERTIFICATE_VERIFY_FAILED - Downloading MY_ATP.pdb #22

Open
sxm525 opened this issue May 22, 2024 · 14 comments
Open

SSL: CERTIFICATE_VERIFY_FAILED - Downloading MY_ATP.pdb #22

sxm525 opened this issue May 22, 2024 · 14 comments
Assignees

Comments

@sxm525
Copy link

sxm525 commented May 22, 2024

I am getting "SSL: CERTIFICATE_VERIFY_FAILED" during db startup, could you please help.

Find below my full log,

[user@BRL72J3 ~]$ podman logs -t f2f6d724008ad02027d72c8c393a8221cafe9b5c3a1bbd08e0d6880f5a3f70fd
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable PODMAN_IGNORE_CGROUPSV1_WARNING to hide this warning.
2024-05-22T15:31:39.345767000-04:00 Archive: /u01/POD1.zip
2024-05-22T15:31:39.347480000-04:00 creating: /u01/app/oracle/oradata/
2024-05-22T15:31:39.348648000-04:00 creating: /u01/app/oracle/oradata/POD1/
2024-05-22T15:31:39.349110000-04:00 creating: /u01/app/oracle/oradata/POD1/17204CBA6183008CE063C4D75E6492EC/
2024-05-22T15:31:39.349152000-04:00 creating: /u01/app/oracle/oradata/POD1/17204CBA6183008CE063C4D75E6492EC/datafile/
2024-05-22T15:31:39.349178000-04:00 creating: /u01/app/oracle/oradata/POD1/175E8D561B6D4CDEE0636402000A1C40/
2024-05-22T15:31:39.349644000-04:00 creating: /u01/app/oracle/oradata/POD1/175E8D561B6D4CDEE0636402000A1C40/datafile/
2024-05-22T15:31:39.442155000-04:00 inflating: /u01/app/oracle/oradata/POD1/redo01.log
2024-05-22T15:31:39.532134000-04:00 inflating: /u01/app/oracle/oradata/POD1/redo02.log
2024-05-22T15:31:39.532407000-04:00 creating: /u01/app/oracle/oradata/POD1/datafile/
2024-05-22T15:31:39.585161000-04:00 inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_temp_m33n3tv9_.tmp
2024-05-22T15:32:06.735885000-04:00 inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_system_m33lvr7m_.dbf
2024-05-22T15:32:06.814266000-04:00 inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_undotbs1_m33n3t3g_.dbf
2024-05-22T15:32:15.172292000-04:00 inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_sysaux_m33lz7xh_.dbf
2024-05-22T15:32:15.601036000-04:00 inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_data_m33m2s61_.dbf
2024-05-22T15:32:15.618548000-04:00 creating: /u01/app/oracle/oradata/POD1/175E8F7D3A3D4D65E0636402000AA841/
2024-05-22T15:32:15.618700000-04:00 creating: /u01/app/oracle/oradata/POD1/175E8F7D3A3D4D65E0636402000AA841/datafile/
2024-05-22T15:32:15.660253000-04:00 creating: /u01/app/oracle/oradata/POD1/onlinelog/
2024-05-22T15:32:15.673523000-04:00 creating: /u01/app/oracle/oradata/POD1/171F1841E82CF4B3E063C4D75E643770/
2024-05-22T15:32:15.673625000-04:00 creating: /u01/app/oracle/oradata/POD1/171F1841E82CF4B3E063C4D75E643770/datafile/
2024-05-22T15:32:15.696223000-04:00 creating: /u01/app/oracle/oradata/POD1/175E81D3E11F4989E0636402000AD075/
2024-05-22T15:32:15.696298000-04:00 creating: /u01/app/oracle/oradata/POD1/175E81D3E11F4989E0636402000AD075/datafile/
2024-05-22T15:32:15.696324000-04:00 creating: /u01/app/oracle/oradata/POD1/controlfile/
2024-05-22T15:32:15.696351000-04:00 inflating: /u01/app/oracle/oradata/POD1/controlfile/o1_mf_m33m65kk_.ctl
2024-05-22T15:32:16.320877000-04:00 TIME ELAPSED Unzipping /u01/POD1.zip: 0 minutes and 37 seconds elapsed
2024-05-22T15:32:18.065494000-04:00 User input JSON not found
2024-05-22T15:32:18.128315000-04:00 MY ADB WORKLOAD_TYPE is ATP
2024-05-22T15:32:18.128971000-04:00 MY ADB CUSTOM NAME is MYATP
2024-05-22T15:32:18.151187000-04:00 BUILDER: Configuring TCPS
2024-05-22T15:32:18.152329000-04:00 BUILDER: Cleanup /u01/app/oracle/wallets/tls_wallet
2024-05-22T15:32:18.164291000-04:00 BUILDER: Creating auto login wallet for server
2024-05-22T15:32:18.456193000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:18.456334000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:18.456874000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:18.456945000-04:00
2024-05-22T15:32:18.489135000-04:00 Enter password:
2024-05-22T15:32:18.489506000-04:00 Enter password again:
2024-05-22T15:32:18.733760000-04:00 Operation is successfully completed.
2024-05-22T15:32:18.749360000-04:00 BUILDER: Creating a self-signed certificate using orapki utility; VALIDITY: 10 years
2024-05-22T15:32:18.909708000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:18.911516000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:18.911602000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:18.911624000-04:00
2024-05-22T15:32:19.240145000-04:00 Cannot modify auto-login (sso) wallet
2024-05-22T15:32:19.241408000-04:00 Enter wallet password:
2024-05-22T15:32:21.144281000-04:00 Operation is successfully completed.
2024-05-22T15:32:21.151481000-04:00 BUILDER: exporting server's cert
2024-05-22T15:32:21.347089000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:21.347278000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:21.347709000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:21.347776000-04:00
2024-05-22T15:32:22.043356000-04:00 Operation is successfully completed.
2024-05-22T15:32:22.059921000-04:00 BUILDER: exporting server's cert
2024-05-22T15:32:22.318691000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:22.319136000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:22.319746000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:22.319829000-04:00
2024-05-22T15:32:23.128622000-04:00 Operation is successfully completed.
2024-05-22T15:32:23.145920000-04:00 BUILDER: exporting encrypted private key
2024-05-22T15:32:23.437932000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:23.438636000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:23.438723000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:23.438760000-04:00
2024-05-22T15:32:23.453315000-04:00 Private key password:
2024-05-22T15:32:23.454948000-04:00 Enter password:
2024-05-22T15:32:23.469735000-04:00 Enter password again:
2024-05-22T15:32:41.651260000-04:00 Enter wallet password:
2024-05-22T15:32:41.979993000-04:00 Operation is successfully completed.
2024-05-22T15:32:41.994723000-04:00 BUILDER: exporting private and certificates together in PEM
2024-05-22T15:32:41.999597000-04:00 BUILDER: generating keystore.jks and truststore.jks
2024-05-22T15:32:42.152267000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:42.152415000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:42.152441000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:42.152480000-04:00
2024-05-22T15:33:00.351039000-04:00 Enter wallet password:
2024-05-22T15:33:00.687456000-04:00 Enter Key store password:
2024-05-22T15:33:00.688343000-04:00 Enter Trust store password:
2024-05-22T15:33:00.785159000-04:00 Operation is successfully completed.
2024-05-22T15:33:00.800159000-04:00 BUILDER: generating sqlnet.ora for client
2024-05-22T15:33:00.800328000-04:00 BUILDER: Generating tnsnames.ora based on the new CN
2024-05-22T15:33:00.803804000-04:00 BUILDER: Overriding service names based on user input
2024-05-22T15:33:00.805435000-04:00 BUILDER: generating ojdbc.properties
2024-05-22T15:33:00.805852000-04:00 BUILDER: zipping wallet for ORDS
2024-05-22T15:33:00.814854000-04:00 updating: README (stored 0%)
2024-05-22T15:33:00.814996000-04:00 updating: adb_container.cert (deflated 24%)
2024-05-22T15:33:00.815033000-04:00 updating: cwallet.sso (stored 0%)
2024-05-22T15:33:00.815062000-04:00 updating: cwallet.sso.lck (stored 0%)
2024-05-22T15:33:00.815108000-04:00 updating: ewallet.p12 (stored 0%)
2024-05-22T15:33:00.815139000-04:00 updating: ewallet.p12.lck (stored 0%)
2024-05-22T15:33:00.815194000-04:00 updating: ewallet.pem (deflated 27%)
2024-05-22T15:33:00.815272000-04:00 updating: keystore.jks (stored 0%)
2024-05-22T15:33:00.815342000-04:00 updating: ojdbc.properties (deflated 49%)
2024-05-22T15:33:00.815451000-04:00 updating: sqlnet.ora (deflated 16%)
2024-05-22T15:33:00.815494000-04:00 updating: tnsnames.ora (deflated 87%)
2024-05-22T15:33:00.815539000-04:00 updating: truststore.jks (deflated 5%)
2024-05-22T15:33:00.817348000-04:00 TIME ELAPSED Wallet Generation: 0 minutes and 42 seconds elapsed
2024-05-22T15:33:01.258150000-04:00 User has requested to download '.pdb' archive file from Object Storage bucket
2024-05-22T15:33:01.258531000-04:00 Downloading MY_ATP.pdb..
2024-05-22T15:33:01.267569000-04:00 Traceback (most recent call last):
2024-05-22T15:33:01.267744000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
2024-05-22T15:33:01.267786000-04:00 chunked=chunked)
2024-05-22T15:33:01.267823000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 344, in _make_request
2024-05-22T15:33:01.267860000-04:00 self._validate_conn(conn)
2024-05-22T15:33:01.267895000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in validate_conn
2024-05-22T15:33:01.267931000-04:00 conn.connect()
2024-05-22T15:33:01.267972000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 358, in connect
2024-05-22T15:33:01.268007000-04:00 ssl_context=context)
2024-05-22T15:33:01.268043000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/util/ssl
.py", line 354, in ssl_wrap_socket
2024-05-22T15:33:01.268078000-04:00 return context.wrap_socket(sock, server_hostname=server_hostname)
2024-05-22T15:33:01.268153000-04:00 File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
2024-05-22T15:33:01.268195000-04:00 _context=self, _session=session)
2024-05-22T15:33:01.268226000-04:00 File "/usr/lib64/python3.6/ssl.py", line 810, in init
2024-05-22T15:33:01.268259000-04:00 self.do_handshake()
2024-05-22T15:33:01.268360000-04:00 File "/usr/lib64/python3.6/ssl.py", line 1070, in do_handshake
2024-05-22T15:33:01.268424000-04:00 self._sslobj.do_handshake()
2024-05-22T15:33:01.268462000-04:00 File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
2024-05-22T15:33:01.268491000-04:00 self._sslobj.do_handshake()
2024-05-22T15:33:01.268526000-04:00 ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
2024-05-22T15:33:01.268564000-04:00
2024-05-22T15:33:01.268594000-04:00 During handling of the above exception, another exception occurred:
2024-05-22T15:33:01.268623000-04:00
2024-05-22T15:33:01.268650000-04:00 Traceback (most recent call last):
2024-05-22T15:33:01.268677000-04:00 File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2024-05-22T15:33:01.268709000-04:00 timeout=timeout
2024-05-22T15:33:01.268737000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
2024-05-22T15:33:01.268762000-04:00 _stacktrace=sys.exc_info()[2])
2024-05-22T15:33:01.268798000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
2024-05-22T15:33:01.268826000-04:00 raise MaxRetryError(_pool, url, error or ResponseError(cause))
2024-05-22T15:33:01.268855000-04:00 urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='objectstorage.us-phoenix-1.oraclecloud.com', port=443): Max retries exceeded with url: /n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
2024-05-22T15:33:01.269425000-04:00
2024-05-22T15:33:01.269500000-04:00 During handling of the above exception, another exception occurred:
2024-05-22T15:33:01.269540000-04:00
2024-05-22T15:33:01.269567000-04:00 Traceback (most recent call last):
2024-05-22T15:33:01.269588000-04:00 File "/u01/scripts/download_my_container_pdb.py", line 102, in
2024-05-22T15:33:01.269607000-04:00 downloader.download()
2024-05-22T15:33:01.269634000-04:00 File "/u01/scripts/download_my_container_pdb.py", line 61, in download
2024-05-22T15:33:01.269655000-04:00 r = requests.get(download_url, stream=True)
2024-05-22T15:33:01.269676000-04:00 File "/usr/lib/python3.6/site-packages/requests/api.py", line 75, in get
2024-05-22T15:33:01.269696000-04:00 return request('get', url, params=params, **kwargs)
2024-05-22T15:33:01.269718000-04:00 File "/usr/lib/python3.6/site-packages/requests/api.py", line 60, in request
2024-05-22T15:33:01.269738000-04:00 return session.request(method=method, url=url, **kwargs)
2024-05-22T15:33:01.269760000-04:00 File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 535, in request
2024-05-22T15:33:01.269781000-04:00 resp = self.send(prep, **send_kwargs)
2024-05-22T15:33:01.269799000-04:00 File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 648, in send
2024-05-22T15:33:01.269820000-04:00 r = adapter.send(request, **kwargs)
2024-05-22T15:33:01.269840000-04:00 File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
2024-05-22T15:33:01.269861000-04:00 raise SSLError(e, request=request)
2024-05-22T15:33:01.269884000-04:00 requests.exceptions.SSLError: HTTPSConnectionPool(host='objectstorage.us-phoenix-1.oraclecloud.com', port=443): Max retries exceeded with url: /n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

@aosingh aosingh self-assigned this May 23, 2024
@aosingh
Copy link
Member

aosingh commented May 23, 2024

@sxm525

Does this always happen during container start ? I don't expect to see this failure.

@sxm525
Copy link
Author

sxm525 commented May 23, 2024

Yes, I tried multiple times and it failed with the same error.

@aosingh
Copy link
Member

aosingh commented May 23, 2024

  • Which Operating system ?

  • What is the podman version ?

podman version
  • Are you running podman in root or rootless mode ?

  • What is the output of following curl command on the host machine ?

curl https://objectstorage.us-phoenix-1.oraclecloud.com:443

@sxm525
Copy link
Author

sxm525 commented May 23, 2024

OS - Windows 10
podman version - 5.0.3
Podman running in rootless mode.

Find below the curl output,
curl https://objectstorage.us-phoenix-1.oraclecloud.com:443
{"code":"NotFound","message":"Not Found"}

@aosingh
Copy link
Member

aosingh commented May 23, 2024

Could you SSH to the podman virtual machine and then try the same curl request ?

podman machine ssh
curl https://objectstorage.us-phoenix-1.oraclecloud.com:443

What is the container start command used ?

@sxm525
Copy link
Author

sxm525 commented May 23, 2024

Find below my curl output from podman machine,

[user@BR~]$ curl https://objectstorage.us-phoenix-1.oraclecloud.com:443
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I used below syntax,
podman run -d
-p 1521:1522
-p 1522:1522
-p 8443:8443
-p 27017:27017
-e WORKLOAD_TYPE=ATP
-e WALLET_PASSWORD=***
-e ADMIN_PASSWORD=***
--cap-add SYS_ADMIN
--device /dev/fuse
--name adb-free
ghcr.io/oracle/adb-free:latest-23ai

@aosingh
Copy link
Member

aosingh commented May 24, 2024

The issues seems to be that on the podman virtual machine which is running on the Windows host, there is no TLS CA bundle to verify the certificate's validity. Could you connect to any host using HTTPS on the podman VM ?

I don't have a Windows machine myself but have been trying to find someone to reproduce this. This does not happen on Linux or MacOS.

Another option could be download the root CA for objectstorage.us-phoenix-1.oraclecloud.com and add it to the podman VM's truststore

sudo cp <oci-root-ca.pem> /etc/pki/ca-trust/source/anchors
sudo update-ca-trust

@sxm525
Copy link
Author

sxm525 commented May 24, 2024

I copied the PEM file and tried but same issue. Build failed during download "MY_ATP.pdb" file.
But I am able to download this file from windows machine & podman machine without any issue. Find below my output.

This is failing only build runtime,

ERROR,
#############################################
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='objectstorage.us-phoenix-1.oraclecloud.com', port=443): Max retries exceeded with url: /n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
######################################

Find below my curl out for file MY_ATP.pdb

Windows machine,
##############
C:\Users>curl --output MY_ATP.pdb https://objectstorage.us-phoenix-1.oraclecloud.com/n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 363M 100 363M 0 0 3405k 0 0:01:49 0:01:49 --:--:-- 2838k

Podman machine,
############
[user@BRL72J3 ~]$ curl --output MY_ATP.pdb https://objectstorage.us-phoenix-1.oraclecloud.com/n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
24 363M 24 87.6M 0 0 2674k 0 0:02:19 0:00:33 0:01:46 2449k

@aosingh
Copy link
Member

aosingh commented May 24, 2024

Could we try updating the VM OS packages ?

podman machine ssh 'sudo rpm-ostree upgrade --check'
podman machine stop && podman machine start

and then retry starting the container ?

@henrikedsparr
Copy link

Is there a fix for this. I'm getting the same results even though i added all certificates to the VM.
I can fetch the image from the VM but not the container during startup.
I'm running podman 5.0.3 and RPM 4.19.1.1
I'm not able to update the OS packages since rpm-ostree is not present, just regular rpm.

@aosingh
Copy link
Member

aosingh commented Jun 27, 2024

@henrikedsparr @sxm525

After some reading and consulting with users using Windows, It is recommended to use Window Subsystem for Linux (WSL2)

Please refer the link : https://medium.com/@sociable_flamingo_goose_694/setup-wsl-for-local-docker-development-on-windows-f0767e0a72d4 to setup WSL2 and install docker engine.

This will start an Linux VM on your Windows host using WSL in which the adb-container can run using docker.

@henrikedsparr
Copy link

@aosingh Thanks for the information. Yesterday i managed to solve it by building my own container image and adding the certificates to /etc/pki/ca-trust/source/anchors and running update-ca-trust.
So if you are using Zscaler you need to add certificates to the container this way for it to work.

@alexesca
Copy link

can this be closed?

@sxm525
Copy link
Author

sxm525 commented Sep 20, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants