OpenSSL: improve MinGW support on Windows #6079

Doekin · 2024-12-27T16:59:52Z

OpenSSL targeting MinGW can be built on Linux or MSYS2, but not directly on Windows. This PR introduces a workaround to enable building on Windows using Git Bash.

waruqi · 2024-12-28T17:17:30Z

packages/o/openssl/xmake.lua

-                if jom then
-                    package:add("deps", "jom", {private = true})
+            elseif is_subhost("windows") then
+                package:add("deps", "strawberry-perl", { private = true })


why remove {system = false}, it will break some things. it will use incorrect perl in some systems and ci envs.

Could you please share more details about the scenarios where the system Perl might fail? There might be a workaround we can implement to address this issue. Using the system Perl helps save disk space.

For your reference, I tested with Perl installed via scoop, and it seemed to work as expected.

I can't reproduce it because I don't remember what happened before. But since there is a clear comment to fix this problem by passing system = false, we shouldn't just remove it.

The comment indicated that Perl bundled with GitForWindows fails to build OpenSSL. I encountered this issue too at the beginning. The error message is as follows:

Can't locate Pod/Usage.pm in @INC (you may need to install the Pod::Usage module) (@INC entries checked: . /usr/lib/perl5/site_perl /usr/share/perl5/site_perl /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl) at configdata.pm line 11398. BEGIN failed--compilation aborted at configdata.pm line 11398. Compilation failed in require. BEGIN failed--compilation aborted.

I looked into the config script and found that Pod::Usage is only used to generate help messages, which are shown with the ./Configure help command. I guess that the help message is not needed when installing the package. So, I removed the use of the Pod::Usage module, and it builds successfully using Perl from GitForWindows.

waruqi · 2024-12-28T17:19:01Z

packages/o/openssl/xmake.lua

@@ -32,15 +32,18 @@ package("openssl")
            if package:is_plat("android") and is_subhost("windows") and os.arch() == "x64" then
                -- when building for android on windows, use msys2 perl instead of strawberry-perl to avoid configure issue
                package:add("deps", "msys2", {configs = {msystem = "MINGW64", base_devel = true}, private = true})
-            elseif is_subhost("windows") and not package:is_precompiled() then


and why not remove not package:is_precompiled()?
If you are using the build-artifacts binary, you do not need to compile openssl and install its build dependencies.

OK, there is no is_precompiled now

OK, there is no is_precompiled now

but we need is_precompiled.

Sorry, I misunderstood.

If you are downloading a precompiled binary, then all build dependencies do not need to be installed.

https://github.com/xmake-mirror/build-artifacts/releases?q=openssl&expanded=true

I guess the outer if not package:is_precompiled() then might work .

xmake-repo/packages/o/openssl/xmake.lua

Lines 30 to 31 in a7144ca

on_load(function (package)

if not package:is_precompiled() then

Doekin · 2024-12-29T04:42:05Z

Supported Combinations of Targets and Windows Environments

Build for Windows in PowerShell with Strawberry Perl
Build for MinGW in Git Bash with bundled Perl

Unsupported Combinations of Targets and Environments

Build for MinGW in PowerShell
- This is expected due to the reliance on many Unix-like shell commands in the makefile.

waruqi · 2024-12-29T06:21:12Z

packages/o/openssl/xmake.lua

+            package:add("deps", "msys2", {configs = {msystem = "MINGW64", base_devel = true}, private = true})
+        elseif is_subhost("windows") then
+            import("lib.detect.find_tool")
+            local perl = find_tool("perl", {paths={"$(env PERL)"}})


we should not call find_tool in on_load

waruqi · 2024-12-29T06:22:01Z

packages/o/openssl/xmake.lua

+    if on_check then
+        on_check(function (package)
+            import("lib.detect.find_tool")
+            local perl = assert(find_tool("perl", {paths={"$(env PERL)"}}), "package(openssl): perl not found!")


use package:find_tool and add spaces =

waruqi · 2024-12-29T06:23:17Z

packages/o/openssl/xmake.lua

@@ -83,7 +102,13 @@ package("openssl")
            table.insert(configs, "no-makedepend")
            table.insert(configs, "/FS")
        end
-        os.vrunv("perl", configs)
+        import("lib.detect.find_tool")
+        local perl = assert(find_tool("perl", {paths={"$(env PERL)"}}), "package(openssl): perl not found!")


we should not call find_tool, we should use add_deps to bind perl path end and run perl

SirLynix · 2024-12-29T13:50:31Z

I'm sorry but I don't understand what benefit this PR does, it seems complicated to just use the system perl instead of installing strawberry perl or am I missing something?

Doekin · 2024-12-29T14:22:34Z

Thank you for your feedback. Let me clarify the reasoning behind this PR:

When building OpenSSL for Windows, we can make use of a previously installed Strawberry Perl, as OpenSSL‘s configure script won't find the Perl bundled with Git Bash if we are in PowerShell or Command Prompt.
For building OpenSSL for MinGW, Strawberry Perl cannot generate the Unix-style paths that OpenSSL requires, which leads to build failures. I added prompts to guide users to build in the appropriate environment. Suitable environments include Git Bash and MSYS2 shell.

Additionally, when building OpenSSL in Git Bash, we may encounter errors due to the missing POD::Usage Perl module, which is not included in the Perl bundled with Git Bash. However, this module is not essential for configuring OpenSSL's build; it is used to generate help information in the configure script. We can remove the use of POD::Usage to continue building.

I hope this clarifies the purpose of the changes.

Doekin · 2024-12-31T01:50:52Z

Changes up to now

Allow Strawberry-Perl not installed by XMake to be used when targeting Windows.
Fix failure when running the configure script in Git Bash.

Example:

xrepo install -vD -p mingw openssl

Avoid the creation of NUL file on Windows.

Example:

Fix The command line is too long error.

Example:

xmake install -vD -p mingw --toolchain=zig openssl

Patch for using llvm-rc as the Windows resource compiler when targeting MinGW.

waruqi · 2024-12-31T04:34:21Z

packages/o/openssl/makefile/patch.lua

+    if package:is_plat("mingw") and package:has_tool("mrc", "llvm_rc", "llvm-rc", "rc") then
+        local cc = package:build_getenv("cc")
+        local cflags = opt.buildenvs.CFLAGS
+        local tmpfile = path.unix(os.tmpfile() .. ".c")


we need to remove tmpfile after running this script

waruqi · 2024-12-31T04:35:54Z

packages/o/openssl/makefile/patch.lua

+        local cflags = opt.buildenvs.CFLAGS
+        local tmpfile = path.unix(os.tmpfile() .. ".c")
+        io.writefile(tmpfile, "int main(void) { return 0; }\n")
+        local compile_out, compile_err = try {function() return os.iorun(format("%s -v %s %s", cc, cflags, tmpfile)) end}


use lua table as argv and call os.iorunv(cc, argv)

I encountered an issue where the CFLAGS might be set to -m64 --target=x86_64-w64-windows-gnu, resulting in a concatenated command like clang "-v" "-m64 --target=x86_64-w64-windows-gnu" "a.c". This causes the following error:

clang-19: error: unknown argument: '-m64 --target=x86_64-w64-windows-gnu' clang version 19.1.6 (https://github.com/llvm/llvm-project.git e21dc4bd5474d04b8e62d7331362edcc5648d7e5) Target: x86_64-w64-windows-gnu Thread model: posix InstalledDir: D:/Applications/Scoop/apps/mingw-mstorsjo-llvm-ucrt/19.1.6-20241217/bin

because buildenvs.CFLAGS has been merged to string. not lua table.

waruqi · 2024-12-31T04:38:04Z

packages/o/openssl/xmake.lua

@@ -83,6 +100,7 @@ package("openssl")
            table.insert(configs, "no-makedepend")
            table.insert(configs, "/FS")
        end
+        import("configure.patch")(package)


Did you test all openssl versions? does this patch work for all versions?

I tested the latest version (1.1.1-w) with the following combinations of targets and environments:

Platform: Windows, Host: Windows

Platform: MinGW, Host: MSYS2 (Git Bash)

Platform: MinGW, Host: Linux

Platform: Linux, Host: Linux

The patch does not work for versions 1.0.0 and 1.0.2-u, resulting in the error:

error: @programdir\core\sandbox\modules\io.lua:188: cannot open file: Configurations/10-main.conf, Unknown Error (3)

Version 1.1.0-l failed with the error:

Unsupported options: no-tests

The patch works for versions 1.1.1-h and 1.1.1-p.

you can use package:version():ge("1.1.1") to limit patch

OpenSSL versions prior to 1.1.1 do not support the no-tests build flag, and there is a conflict between /FS and VC-WIN64A. After removing these flags, I still could not build OpenSSL 1.1.0-l for MinGW. While 1.1.0-l builds for Windows, 1.0.2-u does not.

you can improve on_check to detect these cases (use version + is_plat).

I discovered that OpenSSL 1.0 requires a different installation method. Detailed explanations can be found here: How to Build OpenSSL 1.0 on Windows.

waruqi · 2024-12-31T05:04:36Z

packages/o/openssl/makefile/patch.lua

+end
+
+function main(package, opt)
+    -- _patch_for_llvm_rc(package, opt)


It was an unintentional change.

star-hengxing · 2024-12-31T13:05:48Z

This package does't have git urls, so you don't need to use package:gitref()

openssl: enhance test

4a58502

Doekin marked this pull request as draft December 27, 2024 23:50

This comment was marked as outdated.

Sign in to view

Doekin force-pushed the openssl branch from 6c983cf to 13c765a Compare December 28, 2024 03:14

openssl: improve cross-build support on Windows

4556226

Doekin force-pushed the openssl branch from 13c765a to 4556226 Compare December 28, 2024 03:15

Doekin marked this pull request as ready for review December 28, 2024 03:17

star-hengxing requested a review from SirLynix December 28, 2024 05:49

waruqi reviewed Dec 28, 2024

View reviewed changes

Doekin added 4 commits December 29, 2024 08:27

openssl: Remove always-true if condition

17ea9ac

openssl: handle mismatched Perl

abe12e9

openssl: fix Lua script

79e98d4

openssl: fix removing DLL import libraries unexpectedly

79c6f04

Doekin added 2 commits December 29, 2024 13:05

openssl: fix BSD platform

0e4066b

openssl: fix wrong config

d5e12bc

Doekin force-pushed the openssl branch from d186b20 to d5e12bc Compare December 29, 2024 05:14

openssl: skip building tests

661f083

waruqi reviewed Dec 29, 2024

View reviewed changes

openssl: improve Perl dependency handling for Windows

2121619

Doekin marked this pull request as draft December 29, 2024 07:11

openssl: fix overlong make recipe

2cf58d8

Doekin marked this pull request as ready for review December 29, 2024 09:21

Doekin added 2 commits December 29, 2024 17:43

openssl: trim trailing whitespace

43c7fb9

openssl: improve Perl platform compatibility warning message

0de6b83

Doekin added 3 commits December 30, 2024 08:45

openssl: ensure ARFLAGS is not overridden with an empty string

aeecc86

openssl: restore the check for precompiled packages

a7144ca

openssl: add LLVM resource compiler support for MinGW

409ef37

waruqi reviewed Dec 31, 2024

View reviewed changes

openssl: remove temporary file after use

6d5fbd8

Doekin force-pushed the openssl branch from 8486776 to 6d5fbd8 Compare December 31, 2024 05:08

Doekin added 3 commits December 31, 2024 13:19

openssl: add try-catch to configuration and Makefile patch functions

d6a3f76

openssl: improve compatibility with older versions

c86fa6e

openssl: add warning for building versions earlier than 1.1.1

dbe0b01

openssl: improve version checks

977b69c

waruqi merged commit c0a3807 into xmake-io:dev Dec 31, 2024
67 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenSSL: improve MinGW support on Windows #6079

OpenSSL: improve MinGW support on Windows #6079

Doekin commented Dec 27, 2024 •

edited

Loading

This comment was marked as outdated.

waruqi Dec 28, 2024

Doekin Dec 29, 2024

waruqi Dec 29, 2024

Doekin Dec 29, 2024

waruqi Dec 28, 2024

waruqi Dec 29, 2024

Doekin Dec 29, 2024 •

edited

Loading

waruqi Dec 30, 2024

Doekin Dec 30, 2024

waruqi Dec 30, 2024

Doekin Dec 30, 2024 •

edited

Loading

Doekin commented Dec 29, 2024 •

edited

Loading

waruqi Dec 29, 2024

waruqi Dec 29, 2024

waruqi Dec 29, 2024

SirLynix commented Dec 29, 2024

Doekin commented Dec 29, 2024 •

edited

Loading

Doekin commented Dec 31, 2024 •

edited

Loading

waruqi Dec 31, 2024

Doekin Dec 31, 2024

waruqi Dec 31, 2024

Doekin Dec 31, 2024 •

edited

Loading

waruqi Dec 31, 2024

waruqi Dec 31, 2024

Doekin Dec 31, 2024

waruqi Dec 31, 2024

Doekin Dec 31, 2024

waruqi Dec 31, 2024 •

edited

Loading

Doekin Dec 31, 2024

waruqi Dec 31, 2024

Doekin Dec 31, 2024

star-hengxing commented Dec 31, 2024

	on_load(function (package)
	if not package:is_precompiled() then

OpenSSL: improve MinGW support on Windows #6079

OpenSSL: improve MinGW support on Windows #6079

Conversation

Doekin commented Dec 27, 2024 • edited Loading

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Doekin Dec 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Doekin Dec 30, 2024 • edited Loading

Choose a reason for hiding this comment

Doekin commented Dec 29, 2024 • edited Loading

Supported Combinations of Targets and Windows Environments

Unsupported Combinations of Targets and Environments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SirLynix commented Dec 29, 2024

Doekin commented Dec 29, 2024 • edited Loading

Doekin commented Dec 31, 2024 • edited Loading

Changes up to now

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Doekin Dec 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

waruqi Dec 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

star-hengxing commented Dec 31, 2024

Doekin commented Dec 27, 2024 •

edited

Loading

Doekin Dec 29, 2024 •

edited

Loading

Doekin Dec 30, 2024 •

edited

Loading

Doekin commented Dec 29, 2024 •

edited

Loading

Doekin commented Dec 29, 2024 •

edited

Loading

Doekin commented Dec 31, 2024 •

edited

Loading

Doekin Dec 31, 2024 •

edited

Loading

waruqi Dec 31, 2024 •

edited

Loading