chore: document OpenSSL-FIPS restriction on RSA key size

[jouho]

Resolved issues:

Resolves #4651

Description of changes:

• Document the reason for error described in openssl 1.0.2 fips: unexpected failure with 512 bit RSA #4651.
• Throw a more meaningful error when the error happens

Description of the error:

Error message:

FAILED test 76
!(((s2n_test_cert_permutation_load_server_chain(&cert, "rsae", "pkcs", "512", "sha1"))) == (-1)) is not true  (s2n_security_policy_cert_preferences_test.c:317)
Error Message: 'size mismatch'
 Debug String: 'Error encountered in s2n_rsa.c:123'
 System Error: Success (0)

RSA_public_encrypt() returns -1, which causes "size mismatch" error:

s2n-tls/crypto/s2n_rsa.c

Lines 121 to 123 in b20075b

	int r = RSA_public_encrypt(in->size, (unsigned char *) in->data, (unsigned char *) out->data,
	s2n_unsafe_rsa_get_non_const(pub_key), RSA_PKCS1_PADDING);
	POSIX_ENSURE((int64_t) r == (int64_t) out->size, S2N_ERR_SIZE_MISMATCH);

This is because in openssl fips module, a function checks to see if RSA key size is bigger than or equal to 1024, and returns -1 when the check fails. openssl-fips-2.0.13 is the fips module we use for openssl-1.0.2-fips. This fips module can be downloaded from here. (Not sure if there is publicly accessible page to view the code on browser)

# RSA_eay_public_encrypt() inside openssl-fips-2.0.13/crypto/rsa/rsa_eay.c
if (FIPS_module_mode() && !(rsa->flags & RSA_FLAG_NON_FIPS_ALLOW)
    && (BN_num_bits(rsa->n) < OPENSSL_RSA_FIPS_MIN_MODULUS_BITS))
    {
    RSAerr(RSA_F_RSA_EAY_PUBLIC_ENCRYPT, RSA_R_KEY_SIZE_TOO_SMALL);
    return -1;
    }

Therefore using 512 bits cert causes the error above. aws_lc_fips doesn't seem to have this minimum requirement for RSA key size.

Testing:

No functional changes

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[lrstewart]

Nice job figuring this out!

[jmayclin]

Is it just 1.0.2 FIPs that has this restriction, or does 3.x also have this restriction?

[jouho]

Not too sure if 3.x has the same restriction, but I wasn't able to find the same function that checks for rsa key size in 3.x.

This if (s2n_is_in_fips_mode() && !s2n_libcrypto_is_awslc()) check wouldn't include 3.x though, because the way we are detecting FIPS is not supported in 3.0 and higher. We call FIPS_mode() to determine if libcrypto is in fips mode. This function is not present in 3.x

[jouho]

In 3.X, it seems the minimum key size for RSA is 2048:
https://github.com/openssl/openssl/blob/4f5febe2c684a803553171940634c1b6f4b7ba40/include/openssl/rsa.h#L48