authz-keycloak and mTLS does not work

[michalspondr]

I want to have mTLS between APISIX 3.9.1 and Keycloak 25.0.4. Keycloak mTLS is configured correctly, communication works between curl and Keycloak. I assume certificates are also created correctly, because mTLS works between different client and Keycloak using same certificates. Both APISIX and Keycloak run as Docker containers. /etc/hosts contains adp and keycloak domains set to 127.0.0.1, also all possible way of alternative domain names are added to certificate as extension. APISIX is run in a standalone mode, thus admin API is not used, I use fixed configuration:

deployment:
  role: data_plane
  role_data_plane:
    config_provider: yaml

apisix:
  enable_admin: false
  config_center: yaml
  declarative: true
  ssl:
    enable: true
    listen_port: 9443
    ssl_trusted_certificate: /usr/local/apisix/conf/cert/rootCA.pem
    cert: /usr/local/apisix/conf/cert/apisix.crt
    key: /usr/local/apisix/conf/cert/apisix.key

In route definitions I use authz-keycloak plugin (note https://keycloak is used):

plugins:
  - name: authz-keycloak

ssls:
  - id: apisix
    cert: |
      -----BEGIN CERTIFICATE-----
...
      -----END CERTIFICATE-----
    key: |
      -----BEGIN PRIVATE KEY-----
...
      -----END PRIVATE KEY-----
    snis:
      - localhost
      - apisix
      - adp

routes:
  -
    uri: /api/status
    plugins:
      authz-keycloak:
        token_endpoint: https://keycloak:8443/realms/myrealm/protocol/openid-connect/token
        client_secret: ckzLqTB3OkqUiKHBnrxgWi6WcURkb1U7
        permissions:
          - adpresource#read
        client_id: adp
        ssl_verify: true
    upstream:
      type: roundrobin
      nodes:
        "internalsystem:8094": 1

#END

cert and key have same content as /usr/local/apisix/conf/cert/apisix.crt and /usr/local/apisix/conf/cert/apisix.key (I was not able to use the files directly, that's why I hardcoded the certificates).

Now I am able to get access token from Keycloak:
curl --cert apisix.crt --key apisix.key --cacert rootCA.pem -d "client_id=myclient" -d "client_secret=$SECRET" -d "grant_type=client_credentials" https://keycloak:8443/realms/myrealm/protocol/openid-connect/token

But I use this access token like this:
curl --cert client.crt --key client.key --cacert rootCA.pem https://adp:9443/api/status -H "Authorization: Bearer $ACCESS_TOKEN"
I get this error in APISIS log:

apisix    | 2024/08/28 10:45:27 [error] 113#113: *213573 SSL_read() failed (SSL: error:0A000412:SSL routines::ssl/tls alert bad certificate:SSL alert number 42), client: 192.168.10.1, server: _, request: "GET /api/status HTTP/2.0", host: "adp:9443"
apisix    | 2024/08/28 10:45:27 [error] 113#113: *213573 [lua] authz-keycloak.lua:643: evaluate_permissions(): Error while sending authz request to https://keycloak:8443/realms/myrealm/protocol/openid-connect/token: error, client: 192.168.10.1, server: _, request: "GET /api/status HTTP/2.0", host: "adp:9443"
apisix    | 2024/08/28 10:45:27 [warn] 113#113: *213573 [lua] plugin.lua:1160: run_plugin(): authz-keycloak exits with http status code 503, client: 192.168.10.1, server: _, request: "GET /api/status HTTP/2.0", host: "adp:9443"

I am able to call Keycloak API using mTLS from APISIX Docker container itself, but the APISIX application itself can not do it.

Any idea where is my configuration wrong? If Keycloak KC_HTTPS_CLIENT_AUTH parameter is set to request instead of required, then TLS is used correctly. Also without the authz-keycloak it works obviously, but then there is no OAuth involved.

Thank you for any hints.

[dosubot[bot]]

@michalspondr Zdravím! Jsem tady, abych ti pomohl s tvým problémem. Jsem bot, který ti může pomoci s opravou chyb, odpovědět na otázky a stát se přispěvatelem.

I found a similar open issue that might be relevant to your problem: authz-keycloak plugin keeps answering invalid_grant/Invalid bearer token. The suggested solution is to double-check if your request's grant type is valid ^[1].

Additionally, there is a closed issue that might provide some insights: bug: apisix failed to verify the validity of the server certificate. One of the solutions mentioned is to enable the proxy_ssl_verify directive by modifying the configuration file ^[2].

Given your setup, here are a few things you might want to check:

1. SSL Configuration: Ensure that the ssl_trusted_certificate is correctly set and that the certificate chain is complete.
2. Proxy SSL Verification: You might need to enable proxy_ssl_verify in your configuration to ensure that APISIX verifies the server certificate of Keycloak.
3. Certificate Paths: Double-check the paths and contents of your certificates and keys to ensure they are correctly referenced and valid.

You can try adding the proxy_ssl_verify directive in your configuration file to see if it resolves the issue. Here is an example of how you might modify your configuration:

apisix:
  ssl:
    enable: true
    listen_port: 9443
    ssl_trusted_certificate: /usr/local/apisix/conf/cert/rootCA.pem
    cert: /usr/local/apisix/conf/cert/apisix.crt
    key: /usr/local/apisix/conf/cert/apisix.key
    proxy_ssl_verify: true

If the issue persists, you might want to review the Keycloak and APISIX logs for more detailed error messages that could provide further clues.

To continue talking to Dosu, mention @dosu.

[michalspondr]

proxy_ssl_verify: true did not resolve the problem. Still the same error message.

Certificates seems to be valid:

openssl verify -CAfile /usr/local/apisix/conf/cert/rootCA.pem /usr/local/apisix/conf/cert/apisix.crt
/usr/local/apisix/conf/cert/apisix.crt: OK

Validating keycloak.crt using same rootCA.pem was also OK.

[michalspondr]

Response from APISIX Slack:
"Whether mTLS is supported between APISIX and plugin-integrated services will be determined by the plugin. Unlike upstream mTLS, which is already supported, most plugins do not support it. Unfortunately authz-keycloak is one of them."