Ergonomics of Cross Compiling

[kbknapp]

The survey mentions the ergonomics of cross compiling quite frequently. While not directly tied to writing CLI applications, it's somewhat implied if you're releasing for multiple platforms.

I've opened this tracking issue to discuss the current best practices/guides, and if there are specific ways in which we can improve the experience.

[steveklabnik]

Including lld with rustc is gonna be huge; not needing a cross linker is a big deal. The forge has instructions for cross compiling to Windows, and they are..... intense.

…

On Mar 24, 2018, at 11:10 PM, Kevin K. ***@***.***> wrote: The survey mentions the ergonomics of cross compiling quite frequently. While not directly tied to writing CLI applications, it's somewhat implied if you're releasing for multiple platforms. I've opened this tracking issue to discuss the current best practices/guides, and if there are specific ways in which we can improve the experience. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[epage]

Not having this problem, I feel like I don't understand it enough to know the requirements or priority.

What situations does a CLI need cross-compilation?

Assumptions

• CLIs have limited platform support. CLIs generally don't run on wasm, android, etc.
• A developer is using Linux/Mac CI service (Travis) and Windows CI service (appveyor).

So questioning the assumptions:

• Are there platforms we are targeting that these CI services are insufficient?
• Are there important scenarios where someone is restricted to a single platform for running their CI?
• Or reasons to target someone cross-compiling without any CI?

[artem-zinnatullin]

@epage

Are there important scenarios where someone is restricted to a single platform for running their CI?

This part in particular is problematic for cross-compilation if you run your build in containerized environment like Docker to get reproducible build environment.

Inside the container it's always Linux and targeting macOS/Windows is really problematic from Linux (or any other combination of actual vs target OS) :(

[softprops]

I would say there's definitely more ceremony to this than I'd like. I've used trust for a few projects. It works but it's a lot of boilerplate to get right.

If there were I bar I'd like to reach, it would look something like the level of effort required to build cross platform golang binary

$ GOOS=linux GOARCH=arm go build ...

Yes, you can manage tool chains with rustup, but it's admittedly an awkward workflow to jump back and forth between rustup and cargo.

[epage]

@artem-zinnatullin

This part in particular is problematic for cross-compilation if you run your build in containerized environment like Docker to get reproducible build environment.

So sounds like the issue is when a user is rolling their own CI rather than being able to leverage one of the community CIs?

Inside the container it's always Linux and targeting macOS/Windows is really problematic from Linux (or any other combination of actual vs target OS) :(

Also, for cross-compiling to windows, how much of a problem will -sys crates be? I think my CLI depends on at least 3 -sys crates; there may be more. I imagine cross-compiling those would be a lot harder because the -sys crate author needs to do whatever is needed to ensure the C library can be cross-compiled on Linux for windows.

[epage]

@softprops

I would say there's definitely more ceremony to this than I'd like. I've used trust for a few projects. It works but it's a lot of boilerplate to get right.

How much of that ceremony is for cross-compiling vs just the CI process and building binaries? I'm investigating reducing the barrier for CIs creating pre-built binaries for #8.

[artem-zinnatullin]

@epage

So sounds like the issue is when a user is rolling their own CI rather than being able to leverage one of the community CIs?

We Dockerize build to be OS and CI agnostic, but run the build on Travis for instance.

[jhwgh1968]

As someone who is working on converting a UNIX C library to Rust in part to add Windows support, I felt the need to chime in when I read this thread. Apologies if this is too little too late.

Full disclosure: not only do I have a strong C background, I seem to have more expertise (and tolerance!) than even many C programmers with getting open source compilers to do unusual cross compiles.

---

Also, for cross-compiling to windows, how much of a problem will -sys crates be?

My rule of thumb is: when cross-compiling, the only thing your sys crates should depend on is C language functions. If they need anything else -- files, sockets, special memory management, etc. -- then deeper examination is required. If all it takes is changing some configuration flags for the library to support Windows, then it's easy. But with most UNIX-based libraries, it's a lot messier than that.

(There are some APIs Windows implements in common with UNIX, but not only are these few and far between, they often have subtle differences that require #[cfg(os)] blocks anyway.)

Aside from patching the library itself to support Windows, the best way I have found (which is still pretty bad) is to use a tool like Corrode to convert it to unsafe Rust, and then replace the OS-specific APIs with calls into the portable parts of Rust standard library.

But that cannot be done mindlessly. There are a lot of assumptions (e.g. the size of int in C) that are baked into the platform the compiler ran on, which may differ on the target. Their side effects can range from sub-optimal performance to unique UB, since the Rust generated by Corrode does exactly what the C code did, which may be quite dangerous.

This means the programmer will still have to read the generated Rust code in order to identify and debug such things, which limits the size of the libraries (and minimum skill level of the programmer) this approach is feasible for.

---

Yes, you can manage tool chains with rustup, but it's admittedly an awkward workflow to jump back and forth between rustup and cargo.

I use Linux, and rustup to update my native toolchain, which is the only one. It supports the GNU Windows target via "rustup target add", and uses a cross-targeted GNU binutils to link that my distro offers as a binary package. I just add an explicit "--target" parameter to the cargo build command to build for Windows, and build for native most of the time.

Perhaps you are implying a more complex workflow, but I don't find the process awkward at all. In fact, compared to many C compilers, it is refreshingly straightforward.

---

Inside the container it's always Linux and targeting macOS/Windows is really problematic from Linux

When it comes to testing in CI, I was actually about to express optimism about testing CLI binaries as defined by this WG. Because I routinely test Windows CLI binaries on Linux.

After cross-compiling them, I manually launch them from Wine's clean-room rewrite of cmd.exe (DOS emulation on Windows NT and later; it runs in a UNIX tty), and everything behaves like real Windows. This includes some annoying stdio quirks and one UB crash which were mild pain points, but they were pain points that were reproduced accurately. 😏

Once I open up my library to the world, that is basically how I plan to do my CI, and that's when I'll find out how well this workflow can fit into a docker image. In the mean time, I will defer to the expertise of others.

---

One last comment: I think LLD will be a big help for Windows hosts cross-compiling to Linux targets, but that is not the whole picture. As my stumbling around in response to a forum question about that shows, I am not quite sure how people on Windows are supposed to get equivalent libc shims as I have for Linux.

If I were put in front of a Windows computer and told to figure it out, I probably could. But I would not expect most users of Rust to able to, especially those without my C background. There seem to be many things blocking that, though (GPL licensing among them), so I'm not sure what this WG can do to help.

[Screwtapello]

A little while ago I wrote a blog post about my experiences cross-compiling from Debian to Windows, and I concur with @jhwgh1968: as long as you're willing to use the GNU ABI, your Linux distro packages the mingw-w64 cross-compiler (or you're willing to build it yourself), and your Rust project doesn't have any weird C-library dependencies, cross-compiling is delightfully easy, even without LLD.

On the other hand, if you want to use the MSVC ABI, getting it set up on Linux is a delicate manual operation, which possibly (I haven't checked) breaks the terms of the MSVC EULA, even with LLD.

After cross-compiling them, I manually launch them from Wine's clean-room rewrite of cmd.exe.

You should look into the Linux kernel's binfmt support. Being able to run cargo test --target x86_64-pc-linux-gnu from your ordinary Unix terminal and have it Do The Right Thing is pretty sweet.

My experiences testing with Wine hasn't been so happy, though. I've been trying to use the reqwests crate, and although it works perfectly on Real Windows, something about the crypto subsystem breaks Wine. Also, Wine's "create a symlink" API is just a stub; it returns success but doesn't actually do anything. Still, there's a lot of Window programs that do work near-flawlessly under Wine, so I think it's still a good first step, and a lot quicker and easier than booting up a VM.

[jhwgh1968]

You should look into the Linux kernel's binfmt support. Being able to run cargo test --target x86_64-pc-linux-gnu from your ordinary Unix terminal and have it Do The Right Thing is pretty sweet.

I do remember reading about that quite a while ago when I first started looking into Wine, but never bothered to read the details since it didn't do much for me. But with cargo's workflow, you've made it sound quite enticing. 😉

My experiences testing with Wine hasn't been so happy, though. I've been trying to use the reqwests crate, and although it works perfectly on Real Windows, something about the crypto subsystem breaks Wine.

I am not surprised. A number of widely-used crypto libraries for Rust seem to seek high performance by using specialized C functions, optimized assembly, or other unsafe shenanigans that are exactly the sort of thing that can cause the UB I mentioned.

Just now I did a cross compile of a medium-small Rocket application, and everything seemed to work -- but I didn't try to make any TLS connections.

[yoshuawuyts]

I've started work on crossgen, a cross-compilation tool based on cross / trust. The idea is to have a single command to setup the CI environment, and generate the encryption keys for deploys.

There's a few issues we could use help on:

• windows support
• token generation
• extract name from cargo.toml

I'm currently working on adding automatic changelogs to it too so it helps document what changed between versions. Hope this comes in useful for folks! ✨