No trusted root CA found for FTS Daemon Found

[vbalbarin]

It appears as if the pod daemons-renew-fts-proxy-on-helm-install-* created during the installation of the daemons Helm chart

helm install daemons rucio/rucio-daemons -f daemons.yaml

does not have access to a root CA.

If we pull the logs for daemons-renew-fts-proxy-on-helm-install-* when it is reporting Error, we see the following:

kubectl logs -f daemons-renew-fts-proxy-on-helm-install-sjjqn
=================== /renew_fts_proxy.sh ========================
#! /bin/bash

# Delegate the proxy to the requested servers



fts-rest-delegate -v -f -H 9999 --key=/opt/rucio/keys/userkey.pem --cert=/opt/rucio/certs/usercert.pem -s https://fts:8446



Loaded CN=Rucio User
User certificate: /opt/rucio/certs/usercert.pem
User private key: /opt/rucio/keys/userkey.pem
Bad endpoint: https://fts:8446 (HTTPSConnectionPool(host='fts', port=8446): Max retries exceeded with url: // (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)))

As a sanity check, the FTS service within the Minikube cluster environment was validated from the client pod

# kubectl exec -it client -- /bin/bash

# # Inside client pod

# curl --cacert /etc/grid-security/certificates/5fca1cb1.0 https://fts:8446
{"delegation": {"major": 1, "minor": 0, "patch": 0}, "api": {"major": 3, "minor": 11, "patch": 0}, "core": null, "schema": {"patch": 1, "minor": 0, "major": 8}, "_links": {"curies": [{"name": "fts", "href": "https://gitlab.cern.ch/fts/fts3"}], "fts:whoami": {"href": "/whoami", "title": "Check user certificate"}, "fts:joblist": {"href": "/jobs{?vo_name,user_dn,dlg_id,state_in}", "title": "List of active jobs", "templated": true}, "fts:job": {"href": "/jobs/{id}", "title": "Job information", "templated": true, "hints": {"allow": ["GET", "DELETE"]}}, "fts:configaudit": {"href": "/config/audit", "title": "Configuration"}, "fts:submitschema": {"href": "/api-docs/schema/submit", "title": "JSON schema of messages"}, "fts:apidocs": {"href": "/api-docs/", "title": "API Documentation"}, "fts:jobsubmit": {"href": "/jobs", "hints": {"allow": ["POST"], "representations": ["fts:submitschema"]}}, "fts:optimizer": {"href": "/optimizer/", "title": "Optimizer"}, "fts:archive": {"href": "/archive/", "title": "Archive"}}}

Note, the root CA /etc/grid-security/certificates/5fca1cb1.0 is a secret deployed by the Kubernetes deployment file client.yaml.

This same Base64 encoded value is referenced by the fts.yaml deployment file.

However, it is not included as a secret accessible by the pod during the installation of the rucio/rucio-daemons Helm chart.

If we look at the source code for values.yaml file in the Helm chart, there doesn't appear to be a way to pass a secrets value and pod mount point for the daemons-renew-fts-proxy-on-helm-install-* to use.

[chicodelarosa]

@vbalbarin I have been researching this issue for a while now.

From Kubernetes' documentation secrets can be made available to other pods as a file.

I will go ahead and give it a try.

[vbalbarin]

Yes, if you do `kubectl get secrets` you should be able to access all of the secrets within the default namespace. What I haven't been able to figure out is how you can reference the secret using the `daemons.yaml` file which supplies the overrides to the `values.yaml` file that the published helm chart uses. That `values.yaml` doesn't seem to have a spot to reference secrets. /Vincent

…

On Thu, Sep 22, 2022 at 6:28 PM Dan Alberto Rosa De Jesús < ***@***.***> wrote: @vbalbarin <https://github.com/vbalbarin> I have been researching this issue for a while now. From Kubernetes' documentation <https://kubernetes.io/docs/concepts/configuration/secret/> secrets can be made available to other pods as a file. I will go ahead and give it a try. — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXLKNWRUCTWJHNCGBTU3FLV7TMPTANCNFSM6AAAAAAQOJ76V4> . You are receiving this because you were mentioned.Message ID: ***@***.***>