OpenSSL extension fails to build on TruffleRuby 24.2.0-dev

[nirvdrum]

I'm unable to build TruffleRuby from source on both macOS and Linux due to compilation issue with the OpenSSL extension using OpenSSL 3. The extension will compile with OpenSSL 1.1, but that's difficult to set up on Linux. The truffleruby-dev builds from ruby-build are also impacted, with the extension failing to compile.

This appears to have been broken when the CRuby 3.3 sources were imported.

> jt build
$ unset JAVA_HOME; /home/nirvdrum/dev/workspaces/truffleruby-ws/mx/mx --env jvm-ce scheckimports --ignore-uncommitted --warn-only
$ /home/nirvdrum/dev/workspaces/truffleruby-ws/mx/mx --java-home labsjdk-ce-latest-24+23-jvmci-b01 --env jvm-ce build
                                             WARNING: JVM standalone 'RUBY_JAVA_STANDALONE_JAVA24' misses LibGraal. Make sure that it is part of the main GraalVM distribution
Building org.truffleruby.cext with GNU Make... [rebuild needed by GNU Make]
make[1]: Entering directory '/home/nirvdrum/dev/workspaces/truffleruby-ws/truffleruby/src/main/c/openssl'
compiling ossl.c
compiling ossl_asn1.c
compiling ossl_bio.c
compiling ossl_bn.c
compiling ossl_cipher.c
compiling ossl_config.c
compiling ossl_digest.c
compiling ossl_engine.c
In file included from ossl.h:175,
                 from ossl.c:10:
openssl_missing.h:195: error: "TS_VERIFY_CTS_set_certs" redefined [-Werror]
  195 | #  define TS_VERIFY_CTS_set_certs(ctx, crts) ((ctx)->certs=(crts))
      |
In file included from ossl.h:175,
                 from ossl_bn.c:11:
openssl_missing.h:195: error: "TS_VERIFY_CTS_set_certs" redefined [-Werror]
  195 | #  define TS_VERIFY_CTS_set_certs(ctx, crts) ((ctx)->certs=(crts))
      |
In file included from ossl.h:175,
                 from ossl_digest.c:10:
openssl_missing.h:195: error: "TS_VERIFY_CTS_set_certs" redefined [-Werror]
  195 | #  define TS_VERIFY_CTS_set_certs(ctx, crts) ((ctx)->certs=(crts))
      |
In file included from ossl.h:175,
                 from ossl_cipher.c:10:
openssl_missing.h:195: error: "TS_VERIFY_CTS_set_certs" redefined [-Werror]
  195 | #  define TS_VERIFY_CTS_set_certs(ctx, crts) ((ctx)->certs=(crts))
      |
In file included from ossl.h:175,
                 from ossl_asn1.c:10:
openssl_missing.h:195: error: "TS_VERIFY_CTS_set_certs" redefined [-Werror]
  195 | #  define TS_VERIFY_CTS_set_certs(ctx, crts) ((ctx)->certs=(crts))
      |
In file included from ossl.h:31:
/home/linuxbrew/.linuxbrew/Cellar/openssl@3/3.4.0/include/openssl/ts.h:439: note: this is the location of the previous definition
  439 | #  define TS_VERIFY_CTS_set_certs(ctx, cert) TS_VERIFY_CTX_set_certs(ctx,cert)
      |
In file included from ossl.h:31:
/home/linuxbrew/.linuxbrew/Cellar/openssl@3/3.4.0/include/openssl/ts.h:439: note: this is the location of the previous definition
  439 | #  define TS_VERIFY_CTS_set_certs(ctx, cert) TS_VERIFY_CTX_set_certs(ctx,cert)
      |
In file included from ossl.h:31:
/home/linuxbrew/.linuxbrew/Cellar/openssl@3/3.4.0/include/openssl/ts.h:439: note: this is the location of the previous definition
  439 | #  define TS_VERIFY_CTS_set_certs(ctx, cert) TS_VERIFY_CTX_set_certs(ctx,cert)
      |
In file included from ossl.h:31:
/home/linuxbrew/.linuxbrew/Cellar/openssl@3/3.4.0/include/openssl/ts.h:439: note: this is the location of the previous definition
  439 | #  define TS_VERIFY_CTS_set_certs(ctx, cert) TS_VERIFY_CTX_set_certs(ctx,cert)
      |
In file included from ossl.h:31:
/home/linuxbrew/.linuxbrew/Cellar/openssl@3/3.4.0/include/openssl/ts.h:439: note: this is the location of the previous definition
  439 | #  define TS_VERIFY_CTS_set_certs(ctx, cert) TS_VERIFY_CTX_set_certs(ctx,cert)
      |
In file included from ossl.h:175,
                 from ossl_bio.c:10:
openssl_missing.h:195: error: "TS_VERIFY_CTS_set_certs" redefined [-Werror]
  195 | #  define TS_VERIFY_CTS_set_certs(ctx, crts) ((ctx)->certs=(crts))
      |
In file included from ossl.h:31:
/home/linuxbrew/.linuxbrew/Cellar/openssl@3/3.4.0/include/openssl/ts.h:439: note: this is the location of the previous definition
  439 | #  define TS_VERIFY_CTS_set_certs(ctx, cert) TS_VERIFY_CTX_set_certs(ctx,cert)
      |
In file included from ossl.h:175,
                 from ossl_engine.c:10:
openssl_missing.h:195: error: "TS_VERIFY_CTS_set_certs" redefined [-Werror]
  195 | #  define TS_VERIFY_CTS_set_certs(ctx, crts) ((ctx)->certs=(crts))
      |
In file included from ossl.h:175,
                 from ossl_config.c:10:
openssl_missing.h:195: error: "TS_VERIFY_CTS_set_certs" redefined [-Werror]
  195 | #  define TS_VERIFY_CTS_set_certs(ctx, crts) ((ctx)->certs=(crts))
      |
In file included from ossl.h:31:
/home/linuxbrew/.linuxbrew/Cellar/openssl@3/3.4.0/include/openssl/ts.h:439: note: this is the location of the previous definition
  439 | #  define TS_VERIFY_CTS_set_certs(ctx, cert) TS_VERIFY_CTX_set_certs(ctx,cert)
      |
In file included from ossl.h:31:
/home/linuxbrew/.linuxbrew/Cellar/openssl@3/3.4.0/include/openssl/ts.h:439: note: this is the location of the previous definition
  439 | #  define TS_VERIFY_CTS_set_certs(ctx, cert) TS_VERIFY_CTX_set_certs(ctx,cert)
      |
cc1: all warnings being treated as errors
make[1]: *** [Makefile:489: ossl_engine.o] Error 1
make[1]: *** Waiting for unfinished jobs....
cc1: all warnings being treated as errors
make[1]: *** [Makefile:489: ossl_bio.o] Error 1
cc1: all warnings being treated as errors
make[1]: *** [Makefile:489: ossl_config.o] Error 1
cc1: all warnings being treated as errors
make[1]: *** [Makefile:489: ossl_digest.o] Error 1
cc1: all warnings being treated as errors
make[1]: *** [Makefile:489: ossl.o] Error 1
cc1: all warnings being treated as errors
make[1]: *** [Makefile:489: ossl_cipher.o] Error 1
cc1: all warnings being treated as errors
make[1]: *** [Makefile:489: ossl_asn1.o] Error 1
cc1: all warnings being treated as errors
make[1]: *** [Makefile:489: ossl_bn.o] Error 1
make[1]: Leaving directory '/home/nirvdrum/dev/workspaces/truffleruby-ws/truffleruby/src/main/c/openssl'
make: *** [Makefile:96: openssl/openssl.so] Error 2

Building org.truffleruby.cext with GNU Make: Failed due to error: 2
FAILED (pid 1686802 exit 1): /home/nirvdrum/dev/workspaces/truffleruby-ws/mx/mx --java-home labsjdk-ce-latest-24+23-jvmci-b01 --env jvm-ce build

[eregon]

This looks a bit like a wrongly cached src/main/c/openssl/extconf.h due to the redefined errors & openssl_missing.h.
@nirvdrum Could you try git clean -Xdf src/main/c/openssl and build and report if that solves it?

i.e. the check at

truffleruby/src/main/c/openssl/extconf.rb

Lines 132 to 138 in 49a4475

	# TruffleRuby: do not perform all checks again if extconf.h already exists
	extconf_h = "#{__dir__}/extconf.h"
	in_development = ENV.key?('MX_HOME')
	if in_development && File.exist?(extconf_h) && File.mtime(extconf_h) >= File.mtime(__FILE__)
	$extconf_h = extconf_h
	else
	### START of checks

doesn't catch changing to a different libssl, which the default changed since 9ddc5e8

[eregon]

If that's not it, I would guess somehow the extconf.rb checks and the compilations end up using different openssl or a mix of openssl versions (e.g. one version for headers, one version for libssl).

[nirvdrum]

I performed a git clean -dxf on the entire source tree and still have the build issue on both macOS 15.1.1 and Ubuntu 24.04. I was, however, wrong about the truffleruby-dev build. That is working on macOS and Linux.

With that said, I found a fix on Linux. I use Linuxbrew and an update pulled in OpenSSL 3.4.0. Either the extension doesn't yet support it or my Linuxbrew installation is broken. Pulling it off the PATH so I'm using the system-provided libraries fixed the post installation script and jt build.

I still haven't got a solution on macOS, other than sticking with OpenSSL 1.1.1. It might be that OpenSSL 3.4.0 doesn't work with the extension. But, then I don't know why truffleruby-dev can build a functional extension. I'll have to spend some more time trying to debug this.

[eregon]

Same issue in https://github.com/ruby/truffleruby-dev-builder/actions/runs/11985521615/job/33417531063

 compiling ossl.c
In file included from ossl.c:10:
In file included from ./ossl.h:175:
./openssl_missing.h:195:11: error: 'TS_VERIFY_CTS_set_certs' macro redefined [-Werror,-Wmacro-redefined]
#  define TS_VERIFY_CTS_set_certs(ctx, crts) ((ctx)->certs=(crts))
          ^
/opt/homebrew/opt/openssl@3/include/openssl/ts.h:439:11: note: previous definition is here
#  define TS_VERIFY_CTS_set_certs(ctx, cert) TS_VERIFY_CTX_set_certs(ctx,cert)
          ^
1 error generated.
make[1]: *** [ossl.o] Error 1

[eregon]

@andrykonchin Could you investigate this soon? It prevents new dev builds to succeed on macos-14.
Probably reverting 9ddc5e8 would help, that's fine for now as GitHub runners added openssl 1.1 back. But obviously only a temporary solution.
We should find out what's the problem and fix it.
It's possible it's an incompatibility between the openssl gem at the version in 3.3.5 and libssl 3.x.

It also sounds similar to ruby/openssl#650

[andrykonchin]

The hotfix is merged in 47f7d0e

[eregon]

Quick observation, this happens on openssl@3 specifically 3.4.0 (on both Linux & macOS) and that is known to have some incompatibilities: https://github.com/openssl/openssl/blob/openssl-3.4.0/NEWS.md#openssl-34
I wonder if OpenSSL 3.4.0 even works for CRuby, will check.
EDIT: it worked fine on CRuby, due to no -Werror.

[eregon]

I found the issue, the problem was we compile core C extensions with -Werror, deprecations are warned by default, and that meant have_func("TS_VERIFY_CTS_set_certs(NULL, NULL)", ts_h) in openssl extconf.rb resulted to false, even it still exists in OpenSSL 3.4.
Fix in #3759