Merge pull request #524 from topipoh/keycloak-24-saml

Add post about Keycloak 24

_config.yml

@@ -417,4 +417,8 @@ authors:
   mikaelstr:
     name: Mikael Stridfeldt
     email: b641627adb7c352bbb7e4ec63cecd67b
-    url: https://se.linkedin.com/in/mikael-stridfeldt-640b9b220/
+    url: https://se.linkedin.com/in/mikael-stridfeldt-640b9b220/
+  topipoh:
+    name: Topi Pohjosaho
+    email: 1657def2bf4cf5804dcd2c3110efeb8c
+    url: https://www.linkedin.com/in/topi-p-398b91103/

_posts/2024-06-06-keycloak-24-saml.md

@@ -0,0 +1,153 @@
+---
+layout: post
+title: Debugging Keycloak and suomi.fi
+author: topipoh
+excerpt: Since version 24, Keycloak refuses to decrypt SAML assertions using a signing certificate. This is how we worked around it.
+tags:
+  - Keycloak
+  - SAML
+  - suomi.fi
+  - AWS
+
+---
+
+**TL;DR** Since version 24, Keycloak refuses to decrypt SAML assertions using a signing certificate. 
+We worked around this by adding the same certificate to Keycloak as an encryption certificate. 
+This can be done using the Web UI or `kcadm.sh` with a JSON payload.
+
+## Background
+Like many others, our AWS based system uses [Keycloak](https://www.keycloak.org/) to implement [suomi.fi](https://www.suomi.fi/) 
+Single Sign-On. Read more about this setup from [our cloud blog](https://cloud.solita.fi/en/integrating-aws-cognito-with-suomi-fi-and-others-eidas-services-via-saml-interface/).
+
+## Problem
+It was supposed to be a routine upgrade from Keycloak 23 to 24. I had not worked with Keycloak before, but we have good 
+documentation for the process.
+
+However, after updating our test Keycloak to 24.0.4, suomi.fi login stopped working.
+
+![Keycloak internal error](/img/keycloak-24-saml/keycloak-internal-error.png)
+
+In the Keycloak log we could see the error `PL00092: Null Value:Key for EncryptedData not found`.
+
+```
+2024-05-16 10:23:54,030 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-1) Uncaught server error: org.keycloak.broker.provider.IdentityBrokerException: Could not process response from SAML identity provider.
+  at org.keycloak.broker.saml.SAMLEndpoint$Binding.handleLoginResponse(SAMLEndpoint.java:598)
+  at org.keycloak.broker.saml.SAMLEndpoint$Binding.handleSamlResponse(SAMLEndpoint.java:681)
+  at org.keycloak.broker.saml.SAMLEndpoint$Binding.execute(SAMLEndpoint.java:287)
+  at org.keycloak.broker.saml.SAMLEndpoint.postBinding(SAMLEndpoint.java:192)
+  at org.keycloak.broker.saml.SAMLEndpoint$quarkusrestinvoker$postBinding_e2ae3e4e98121b36952f2279cd4bb60100612099.invoke(Unknown Source)
+  at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
+  at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
+  at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
+  at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
+  at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
+  at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
+  at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
+  at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
+  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
+  at java.base/java.lang.Thread.run(Thread.java:840)
+Caused by: java.lang.RuntimeException: PL00092: Null Value:Key for EncryptedData not found.
+  at org.keycloak.saml.common.DefaultPicketLinkLogger.nullValueError(DefaultPicketLinkLogger.java:195)
+  at org.keycloak.saml.processing.core.util.XMLEncryptionUtil.decryptElementInDocument(XMLEncryptionUtil.java:287)
+  at org.keycloak.saml.processing.core.saml.v2.util.AssertionUtil.decryptAssertion(AssertionUtil.java:612)
+  at org.keycloak.broker.saml.SAMLEndpoint$Binding.handleLoginResponse(SAMLEndpoint.java:458)
+  ... 14 more
+```
+
+A lot of digging ensued. Eventually, my colleague found this in [version 21 release notes](https://www.keycloak.org/docs/21.1.2/upgrading/index.html#saml-sp-metadata-changes):
+
+> #### SAML SP metadata changes
+> 
+> In this version, Keycloak will refuse to decrypt assertions encrypted using a realm key generated for signing purpose.
+> This change means all encrypted communication from IDP to SP (where Keycloak acts as the SP) will stop working.
+> 
+> There are two ways to make this work:
+> - either update the IDP configuration with the metadata generated by a newer version of Keycloak,
+> - or run Keycloak in backward compatibility mode that will make Keycloak work with the metadata generated by older 
+>   Keycloak versions.
+>   This mode can be enabled using `-Dkeycloak.saml.deprecated.encryption=true` flag.
+>   **Note this backward compatibility mode is planned to be removed in Keycloak 24.**
+
+The backwards compatibility flag `-Dkeycloak.saml.deprecated.encryption=true` has been removed in version 24, but it was
+not mentioned in version 24 release notes.
+
+This is the removed code in 
+[SAMLEndpoint.java](https://github.com/keycloak/keycloak/blob/23.0.3/services/src/main/java/org/keycloak/broker/saml/SAMLEndpoint.java#L453-L463.). 
+
+Keycloak now refuses to use a key with use **SIG** (signature) for decryption in 
+[SAMLDecryptionKeysLocator.java](https://github.com/keycloak/keycloak/blob/24.0.4/services/src/main/java/org/keycloak/protocol/saml/SAMLDecryptionKeysLocator.java#L121):
+```
+Stream<KeyWrapper> keysStream = session.keys().getKeysStream(realm)
+        .filter(key -> key.getStatus().isEnabled() && KeyUse.ENC.equals(key.getUse()));
+```
+
+## Solution
+
+We solved this by simply adding the same certificate with use **ENC** (encryption).
+This can be done using the admin console, or programmatically.
+The result is that we have the same certificate twice under *Realm settings -> Keys*, both with use SIG 
+and ENC. This way we can upgrade Keycloak without changing suomi.fi configuration.
+
+Note that Keycloak key use (SIG/ENC) has nothing to do with 
+[x.509 key usage extension](https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1.3) - Keycloak does not seem to 
+care about that.
+
+
+### Adding the encryption certificate programmatically
+
+We have a bash script that is run after Keycloak starts. The script adds our signing certificate from a Java keystore, 
+if not already added.
+
+We added this to the script:
+
+```
+# Add the same certificate for encryption use
+# In version 24, Keycloak will refuse to decrypt assertions encrypted using a realm key 
+# generated for signing purpose.
+ENCRYPTION_ALGORITHM="RSA-OAEP"
+provider_id=$(kcadm.sh get keys -r suomifi | \
+  jq -r ".keys[] select(.algorithm==\"${ENCRYPTION_ALGORITHM}\") | .providerId")
+if [ -z "$provider_id" ]
+then
+  echo "PROVISION: Add encryption certificate"
+
+  # Escape newlines
+  PRIVATE_KEY=$(awk '{printf "%s\\n", $0}' $PRIVATE_KEY_FILE)
+  PUBLIC_KEY=$(awk '{printf "%s\\n", $0}' $PUBLIC_KEY_FILE)
+
+  JSON_STRING="{
+    \"name\" : \"rsa-enc\",
+    \"providerId\" : \"rsa-enc\",
+    \"providerType\" : \"org.keycloak.keys.KeyProvider\",
+    \"parentId\" : \"suomifi\",
+    \"config\" : {
+      \"privateKey\": [\"${PRIVATE_KEY}\"],
+      \"certificate\" : [\"${PUBLIC_KEY}\"],
+      \"active\" : [ \"true\" ],
+      \"priority\" : [ \"103\" ],
+      \"enabled\" : [ \"true\" ],
+      \"algorithm\" : [ \"${ENCRYPTION_ALGORITHM}\" ]
+    }
+  }"
+
+  # Add certificate from JSON payload (RSA-OAEP not supported by JavaKeystoreKeyProvider)
+  echo "$JSON_STRING" | kcadm.sh create components \
+    --file - \
+    --target-realm suomifi \
+    --no-config \
+    --server http://localhost:8080 \
+    --realm master \
+    --user "$KEYCLOAK_ADMIN" \
+    --password "$KEYCLOAK_ADMIN_PASSWORD"
+else
+  echo "PROVISION: Found certificate with algorithm ${ENCRYPTION_ALGORITHM}, no need to add."
+fi
+```
+
+The official documentation does not mention using JSON payloads, but this is the only way we got this to work. 
+Keycloak is not able to add a key with the algorithm RSA-OAEP from a Java keystore.
+
+We found the JSON payload solution from this blog post: 
+[Managing custom realm keys in Keycloak programmatically](https://www.puppeteers.net/blog/managing-custom-realm-keys-in-keycloak-programmatically/).
+
+Hopefully this post helps someone else.