Linker warnings when building Rust examples

[skade]

I have the following build error for the Rust examples:

vagrant@packer-vmware-iso:~/rumprun-packages/rust/examples/hello$ make
rustc --target=x86_64-rumprun-netbsd hello.rs
error: linking with `x86_64-rumprun-netbsd-gcc` failed: exit code: 1
note: "x86_64-rumprun-netbsd-gcc" "-Wl,--as-needed" "-m64" "-L" "/home/vagrant/rumprun-packages/rust/build/destdir/lib/rustlib/x86_64-rumprun-netbsd/lib" "hello.0.o" "-o" "hello" "-Wl,--gc-sections" "/home/vagrant/rumprun-packages/rust/build/destdir/lib/rustlib/x86_64-rumprun-netbsd/lib/libstd-bb943c5a.rlib" "/home/vagrant/rumprun-packages/rust/build/destdir/lib/rustlib/x86_64-rumprun-netbsd/lib/libcollections-bb943c5a.rlib" "/home/vagrant/rumprun-packages/rust/build/destdir/lib/rustlib/x86_64-rumprun-netbsd/lib/librustc_unicode-bb943c5a.rlib" "/home/vagrant/rumprun-packages/rust/build/destdir/lib/rustlib/x86_64-rumprun-netbsd/lib/librand-bb943c5a.rlib" "/home/vagrant/rumprun-packages/rust/build/destdir/lib/rustlib/x86_64-rumprun-netbsd/lib/liballoc-bb943c5a.rlib" "/home/vagrant/rumprun-packages/rust/build/destdir/lib/rustlib/x86_64-rumprun-netbsd/lib/liballoc_system-bb943c5a.rlib" "/home/vagrant/rumprun-packages/rust/build/destdir/lib/rustlib/x86_64-rumprun-netbsd/lib/liblibc-bb943c5a.rlib" "/home/vagrant/rumprun-packages/rust/build/destdir/lib/rustlib/x86_64-rumprun-netbsd/lib/libcore-bb943c5a.rlib" "-L" "/home/vagrant/rumprun-packages/rust/build/destdir/lib/rustlib/x86_64-rumprun-netbsd/lib" "-L" "/home/vagrant/rumprun-packages/rust/examples/hello/.rust/lib/x86_64-rumprun-netbsd" "-L" "/home/vagrant/rumprun-packages/rust/examples/hello/lib/x86_64-rumprun-netbsd" "-Wl,-Bstatic" "-Wl,-Bdynamic" "-l" "pthread" "-l" "unwind" "-l" "compiler-rt"
note: /usr/bin/ld: error in hello(.eh_frame); no .eh_frame_hdr table will be created.
/usr/bin/ld: hello(.eh_frame+0x4425c): reloc against `.text': error 2
collect2: error: ld returned 1 exit status

error: aborting due to previous error
make: *** [hello] Error 101

(Full output here: https://gist.github.com/skade/6e2c7138725ffe20538c)

The system is a stock Ubuntu 14.04 LTS. I have built rumprun with CC=gcc-4.8 and CC=gcc-4.9, but with no success.

I can provide a VMWare VM (Vagrant) on request or try to rebuild the whole stack on Virtualbox if necessary.

[anttikantee]

Can you dump the contents of .eh_frame from hello.0.o? (run rustc with -C save-temps, then run both readelf -w hello.0.o and readelf -x .eh_frame hello.0.o)

Not sure what I'm looking for, but seems like a good place to start ...

[anttikantee]

Oh, and the most obvious thing is that if you can install a "normal" rustc which produces host binaries and check if that works. And if it works, can you compile the same version as the package uses and test that too? If that works too, then we just need to narrow down the differences.

[skade]

Compiling with the rustc built by rumpkernel-packages without --target works. Do I need to dig further?

I added the output of readelf -x and readelf -w here: https://gist.github.com/skade/6e2c7138725ffe20538c

[anttikantee]

Ah, I didn't realize you could run the existing one without --target. Looks like it works. @gandro, can you confirm that the rumprun-packages is expected to do something sensible when run like that?

Hmm. Then I guess we should know which stage the error is coming from. Can you edit a set -x into x86_64-rumprun-netbsd-gcc (it's a shell script) and re-run rustc with --target?

[gandro]

@gandro, can you confirm that the rumprun-packages is expected to do something sensible when run like that?

Yes. The built Rust compiler generates binaries for the host by default, i.e. if you don't specify --target. (To run the compiler you need the std crate, which is why it is compiled for both, the host and rumprun)

[skade]

set -x output added to the gist.

[anttikantee]

Ah, ok, so linking the relocatable succeeds. Next up:

export RUMPRUN_STUBLINK=succeed

Then run the readelf bits for hello. Also, just for kicks, see what happens if you try to rumpbake hello. I'm assuming you get the same error, but let's see.

[anttikantee]

and by rumpbake I mean rumpbake hw_virtio hello{,.bin}

[anttikantee]

sigh. rumpbake hw_virtio hello.bin hello

[skade]

See the output here: https://gist.github.com/skade/6e2c7138725ffe20538c#file-readelf-with-succeed

rumpbake gives:

vagrant@packer-vmware-iso:~/rumprun-packages/rust/examples/hello$ rumpbake hw_virtio hello.bin hello

!!!
!!! NOTE: rumpbake is experimental. syntax may change in the future
!!!

/usr/bin/ld: error in /tmp/rumpbake.xpwBnE/tmp1.obj(.eh_frame); no .eh_frame_hdr table will be created.
/usr/bin/ld: /tmp/rumpbake.xpwBnE/tmp1.obj(.eh_frame+0x4425c): reloc against `.text': error 2
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status

[anttikantee]

No, set RUMPRUN_STUBLINK to succeed (not RUMPRUN) and run readelf against hello, not hello.0.o.

[skade]

Heh, it actually ran under RUMPRUN_STUBLINK=succeed I tried to add that one for visual effect.

readelf -w hello gives me a lot of warnings of this kind:

vagrant@packer-vmware-iso:~/rumprun-packages/rust/examples/hello$ readelf -w hello > output
readelf: Warning: skipping invalid relocation offset 0x4425c in section .eh_frame
readelf: Warning: skipping invalid relocation offset 0x44270 in section .eh_frame
readelf: Warning: skipping invalid relocation offset 0x44284 in section .eh_frame
[snip]
readelf: Warning: skipping invalid relocation offset 0x5b038 in section .eh_frame
readelf: Warning: skipping invalid relocation offset 0x5b04c in section .eh_frame
readelf: Warning: skipping invalid relocation offset 0x5b068 in section .eh_frame
readelf: Warning: skipping invalid relocation offset 0x5b084 in section .eh_frame
readelf: Warning: skipping invalid relocation offset 0x5b098 in section .eh_frame
readelf: Warning: Invalid length 0xffffff88 in FDE at 0x000078
readelf: Warning: Unsupported or unknown Dwarf Call Frame Instruction number: 0x1b

The output file is rather large, is it still necessary after this?

[anttikantee]

Visual effect: best when portraying the truth ;-)

and the hexdump (-x) for .eh_frame?

[anttikantee]

Also, can you set -v and -Wl,--verbose in the x86_64-rumprun-netbsd-gcc ferment stage. I also redirected the output to /tmp/debug (since I don't know how to get rustc to not eat it). So, for me, everything put together looks like this:

        ${CC} ${CFLAGS} -no-integrated-cpp -v \
            -specs=/home/pooka/rumpkernel/rumprun/app-tools/specs-compile_or_ferment \
            -Wl,-r -Wl,-u,main -Wl,--verbose \
            "$@" /home/pooka/rumpkernel/rumprun/app-tools/x86_64-rumprun-netbsd-recipe.s ${EXTRALIBS} > /tmp/debug 2>&1 || die

Then send the output of /tmp/debug

[anttikantee]

The "debug" item in the gist is still missing -v from the compiler invocation.

Also, it looks like the linker script is different from what I have with a newer Ubuntu or even from what I could find from the GNU binutils repository for 2.24. Is it possible for you to try with binutils 2.25? I wouldn't be too surprised if there was something subtly broken in 2.24 than only our setup manages to tickle. (which is not to say I'm 100% sure it's not our bug, just that it's a reasonable thing to test)

[anttikantee]

I wouldn't be too surprised if this diff (which is in binutils 2.25) fixes it:
https://sourceware.org/ml/binutils/2014-10/msg00081.html

The gist is still missing readelf -x .eh_frame hello (for the RUMPRUN_STUBLINK=succeed case), which would help make sure. But I think it's easier for you to try with new binutils than me to start manually decoding a hex dump.

Now, if I'm allowed to get ahead of myself (knock knock), if the linker is the problem, I don't think there's much we'll do except say "use a newer linker for rust and its exceptions". Using a relocatable object (i.e. one produced by ld -r) as our intermediate linking format has always made me slightly nervous, but I can't really think of a better way either. Might be worthwhile to contact the binutils guys for comments, though, maybe they have better ideas. attn @mato

Well, yea, the above paragraph of course depends on 2.25 actually fixing the linking, so I dearly hope I didn't jinx it ;-)

[skade]

With ld 2.25:

=== calling "hello.img" main() ===

Hello, rumprun!

=== main() of "hello.img" returned 0 ===

It still gives a warning on bake:

vagrant@packer-vmware-iso:~/rumprun-packages/rust/examples/hello$ rumpbake hw_virtio hello-tcp.img hello

/usr/local/bin/ld: error in /tmp/rumpbake.9IGaLV/tmp1.obj(.eh_frame); no .eh_frame_hdr table will be created.

[anttikantee]

Do you mean hello gives a warning, or only hello-tcp? Also, the message you pasted says "error". Does hello-tcp actually work?

What's your g++ version?

[skade]

Both work. The error is only seen when using rumpbake directly.

g++ is:

gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)

[anttikantee]

What do you mean "directly on the binary"? You have to bake before you can run and therefore observe a working result, so I'm quite confused now.

[skade]

Sorry, wrote too much, corrected. The second example uses cargo rumpbake, which probably eats the error.

[anttikantee]

So let's summarize:

• both hello and hello-tcp work (as in they run and produce the expected result)
• using "cargo rumpbake" or rumpbake "directly" makes no functional difference (both still run fine)
• using "rumpbake" directly produces a message from the linker saying "error", but the result still runs fine

Did I get that right?

[skade]

Yes.

[mato]

On Friday, 09.10.2015 at 05:04, Antti Kantee wrote:

So let's summarize:

• both hello and hello-tcp work (as in they run and produce the expected result)
• using "cargo rumpbake" or rumpbake "directly" makes no functional difference (both still run fine)
• using "rumpbake" directly produces a message from the linker saying "error", but the result still runs fine

Did I get that right?

I've just built the rust examples on my Xen box running Debian stable, GCC
4.9.2 and binutils 2.25 and can confirm the above points.

@anttikantee: Regarding ld -r as an intermediate format, incremental
linking (which is what ld -r does) is a fairly standard thing to do. While
our use of it may be considered "creative", I don't think it's
fundamentally wrong. If we find linker bugs let's report them upstream.

[anttikantee]

Nuts.

Anyway, just as a data point, I don't locally get the c'est-ne-pas-error you're seeing.

[anttikantee]

On 09/10/15 12:26, Martin Lucina wrote:

@anttikantee: Regarding ld -r as an intermediate format, incremental
linking (which is what ld -r does) is a fairly standard thing to do. While
our use of it may be considered "creative", I don't think it's
fundamentally wrong. If we find linker bugs let's report them upstream.

From the binutils thread linked above:

=== snip ===
I'm very tempted to say that people
who use ld -r to package object files deserve to suffer for their
misguided actions..
=== snip ===

If the binutils attitude is that, I wouldn't be too surprised if we'd
have to do a lot of reporting as we delve deeper into the more creative
uses of C++ et al.

[mato]

Another data point: If I perform a full manual link of hello with no -r involved anywhere in the rust link process then I don't get the error about .eh_frame. So it is the ld -r that's triggering it.
By "full manual link" I mean:

• Add logging to x86_64-rumprun-netbsd-gcc to extract final link command rustc is using
• rustc --target=x86_64-rumprun-netbsd --emit=obj hello.rs
• x86_64-rumprun-netbsd-objcopy --redefine-sym main=rumpbake_main1 hello.o hello.1.o
• Hack together specs_compile_or_ferment and specs_bake
• Do full link in a single step (actual link command elided for brevity)

[mato]

It'd be useful to have some way of verifying the exception handling in Rust actually works. Not knowing anything about rust, could someone knowledgeable write a simple test program that actuall uses exceptions?

[skade]

Rust has no exceptions. It only has stack unwinding on panic. Panics cannot be recovered.

struct HasDrop;

impl Drop for HasDrop {
    fn drop(&mut self) {
        println!("Dropping!");
    }
}

fn main() {
    let x = HasDrop;
    panic!("PAAANIC!");

    // do stuff

} // x goes out of scope here

/cc @steveklabnik might know someone with deeper knowledge or have it.

[mato]

The above program produces:

=== calling "./drop.bin" main() ===

thread '<main>' panicked at 'PAAANIC!', drop.rs:11
Page fault at linear address 0x8, rip 0x1a6995, regs 0xcc03d8, sp 0xcc0480, our_sp 0xcc03c0, code 0
Thread: lwp
RIP: e030:[<00000000001a6995>] 

The faulting address is rb_tree_find_node + 21 in section .text. So something is hosed, somewhere. Am going to try with a C++ test program to see if our EH/unwind setup works at all.

[steveklabnik]

My understanding is that panics end up with basically the same implementation as exceptions have, we just don't have the catching mechanism.

This is slightly above my pay grade, but a page fault to me would indicate that you're not generating landing pads/guard pages? Maybe? I need more coffee.

[mato]

It looks like our EH is seriously broken (not surprising, since I don't think anyone's actually tested it). I've opened an issue (rumpkernel/rumprun#55) with a simple C++ test program. The ld .eh_frame error is orthogonal to this, but until we have EH working we have no way to tell if it's harmless or not.

[anttikantee]

If rust uses exceptions only for panics, we can say that we fully support bug-free rust programs ;-)

[anttikantee]

Can the synopsis be changed to something less dramatic, e.g. "error" -> "warning"?

[Geal]

Hi all,

just a data point here, for OSX, I followed the tutorial to build the ELF cross compiler, and the ld version it generates is 2.23.2.
From what I understand (I'm not too familiar with NetBSD), I need to download a recent version, put it in external/gpl3/binutils and rebuild the toolchain, right?

[skade]

@Gael if you could confirm that that works, you'd save my weekend :D

[Geal]

Bad news, it does not work. Compilation fails for bfd, bfd.info and other stuff. There may be additional steps like patching, but I don't know how it works on this platform.

[Geal]

Apparently, the NetBSD-current sources only have ld 2.23.

[anttikantee]

I'd download binutils (and gcc) and build them manually, IIRC it's pretty much just configure+make. The advantage of the build.sh method was really only that you need so few commands, but if you need to start manually downloading and patching things too, that advantage is already lost.

[skade]

polyfractal in IRC mentioned that you may have pointers where to fix this in rumprun itself, sadly, I lost it :(.

[Geal]

I built binutils manually with x86_64-elf as target, and replaced x86_64--netbsd-ld and x86_64--netbsd-ld.bfd, but the issue is still present. Do I need to rebuild the entire toolchain?
Maybe I'm trying too hard here, but it would be great to have a working toolchain on OSX with straightforward build instruction for that binutils issue.

[anttikantee]

It's mentioned already above: getting rid of a relocatable object as the intermediate format between cc and rumprun-bake. As long as the folks writing the linker think using a relocatable object is misguided (see quote above), don't think we'll have an easy life down that path.

The obvious first step for doing it correct would be to ask the binutils folks what they'd suggest instead.

The quickest way to hack it, as far as I can imagine, is to record the params to "cc -o" in the link phase, place them into the "object", and just replay the entire command line along with the libs specified by rumprun-bake. If done that way, we lose the ability to copy the "object" around, but if that's a price we have to pay for things working, then so be it (at least for now). Bundling things up into a custom, copyable format (e.g. tarball) is probably going to be difficult and require full understanding of ld parameters -- for example, you can read an object, but can't necessarily write to that path, so you need to mangle all the parameters.

[Geal]

Alright, if I build binutils 2.25 and GCC 5.1 manually on OSX, then rumprun, I get a working toolchain (no need to rebuild rustc and cargo). I updated the wiki page with the instructions.

[anttikantee]

On 26/11/15 13:43, Geoffroy Couprie wrote:

Alright, if I build binutils 2.25 and GCC 5.1 manually on OSX, then rumprun, I get a working toolchain (no need to rebuild rustc and cargo). I updated the wiki page with the instructions.

Thanks for figuring out how to do that, and even more so for documenting it.

Can you coordinate with @mattgray to re-polish the page? I think we
should provide only one set of instructions. Maybe those instructions
could just turn into a helper script in the rumprun repo? (though, as a
!OSXuser I don't know what the implications of the "brew" step are)

[skade]

(though, as a !OSXuser I don't know what the implications of the "brew" step are)

The implications are: imagine an emacs vs. vim debate, but about package managers.