Link using the linker directly

116209c
Opened by Brian Anderson at 2020-05-10 00:37:22

Right now we link through a C compiler, but Rust should not depend on having a C compiler available nor expose details of cc linking semantics.

I know there have been other issues on this but I can't find them.

  1. I really like this idea, this has bit us in the past by unknowningly bringing in dependencies we weren't expecting.

    However, I don't think that this will be an easy thing to do. I found out that compilers add a lot of command line parameters to the system linker. This will be a difficult thing to do on all systems, but it may be doable. I'm pasting below a survey of what I've got a VM for. I got all of these via cc -v foo.c and seeing how it invoked the linker. (I didn't wrap things in code blocks so sane word wrap is here)

    OSX

    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld -demangle -dynamic -arch x86_64 -macosx_version_min 10.9.0 -o a.out /var/folders/9s/r4snvt950_d9_rcl6w50tvw80000gn/T/a-QOBPrL.o -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/5.0/lib/darwin/libclang_rt.osx.a

    Looks like everything here is guessable except for libclang_rt.osx.a, and that should probably get replaced by our own version of libcompiler-rt anyway.

    Arch linux

    "/usr/bin/ld.gold" --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o a.out /usr/bin/../lib64/gcc/x86_64-unknown-linux-gnu/4.8.2/../../../../lib64/crt1.o /usr/bin/../lib64/gcc/x86_64-unknown-linux-gnu/4.8.2/../../../../lib64/crti.o /usr/bin/../lib64/gcc/x86_64-unknown-linux-gnu/4.8.2/crtbegin.o -L/usr/bin/../lib64/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/usr/bin/../lib64/gcc/x86_64-unknown-linux-gnu/4.8.2/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/bin/../lib64/gcc/x86_64-unknown-linux-gnu/4.8.2/../../.. -L/lib -L/usr/lib /tmp/foo-93f224.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/bin/../lib64/gcc/x86_64-unknown-linux-gnu/4.8.2/crtend.o /usr/bin/../lib64/gcc/x86_64-unknown-linux-gnu/4.8.2/../../../../lib64/crtn.o

    There's a lot of things going on here. We would need to find libgcc, libgcc_s, crtn.o, and maybe crtend.o? The libraries seem standard, but the crt files seem sketchy.

    Ubuntu

    /usr/bin/ld -z relro --hash-style=gnu --as-needed --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o a.out /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.6/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../.. -L/lib/x86_64-linux-gnu -L/lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib /tmp/foo-fw3tGx.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/4.6/crtend.o /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/crtn.o

    This looks fairly similar to arch. There's some weird flags I haven't seen much before, but they're in theory fairly guessable.

    FreeBSD

    /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o a.out /usr/lib/crt1.o /usr/lib/crti.o /usr/lib/crtbegin.o -L/usr/lib /tmp/foo-zCRcGs.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/crtend.o /usr/lib/crtn.o

    Nice an simple! Like ubuntu/arch, we'd have to find things like ld-elf, crtX.o, and the gcc libs

    Windows (MinGW)

    I'm not sure how to copy/paste out of a windows terminal, but the command line used collect2.exe instead of ld.exe (not sure if that even exists), and it had a whole bunch of libraries I've never even heard of: moldname, mingwew, msvcrt, advapi32, shell32, user32, kernel32, mingw32, gcc, gcc_eh.

    Windows also had the usual crtN.o files to link in.

    Alex Crichton at 2014-01-30 22:57:11

  2. @alexcrichton: Arch Linux also passes -O1 --sort-common --as-needed -z relro via LDFLAGS when building packages, but doesn't patch the toolchain to force it like Ubuntu.

    (I think --hash-style=gnu became the default)

    Daniel Micay at 2014-01-30 23:04:10

  3. I hope if we can figure out how to invoke the linker correctly enough to get through the bots, that will be a great start, and patches for fixing other systems would arrive quickly. We can also leave in a fallback mode for awhile that just uses C for linking.

    Brian Anderson at 2014-01-31 04:31:59

  4. We can also do it incrementally, using C for platforms that haven't been converted.

    Brian Anderson at 2014-01-31 04:33:10

  5. One thing I recently realized, we can make some temporary progress by passing -nodefaultlibs to gcc. I think that this will still pass through lots of flags to the linker, but we shouldn't be picking up libs like libgcc by default.

    Alex Crichton at 2014-02-04 16:29:26

  6. An excellent example of the trickery that we'd need to perform is what clang does apparently: https://github.com/llvm-mirror/clang/blob/master/lib/Driver/Tools.cpp

    It looks like there's a whole bunch of logic going on in there to find all these extra files. Including libclang would help, but it would indeed be a pretty big dependency to say that rustc is itself built on top of clang.

    Alex Crichton at 2014-06-01 07:51:48

  7. @brson, @alexcrichton: do you think it is OK to target a specific linker (ld probably), or do we want linker driver to be completely scriptable, similar to gcc tool specs? Though I am afraid that over time, the latter will tend to develop into a Turing-complete language. :frowning:

    vadimcn at 2014-08-26 04:13:35

  8. I would expect to target the system ld directly. The main impetus for this is to shed dependencies and to get away from a C compiler dependency, it's unclear to me how realistic of a goal it is though (it sure would be nice!)

    Alex Crichton at 2014-08-26 14:12:04

  9. Triage: we allow you to choose a linker, but still have a gcc dependency. Except with MSVC, which uses link.exe.

    Steve Klabnik at 2015-09-03 15:02:13

  10. Triage: no change. However, see also: https://github.com/rust-lang/rust/pull/36120

    Steve Klabnik at 2016-10-23 22:24:49

  11. The clang code for doing this has moved slightly: https://github.com/llvm-mirror/clang/blob/release_50/lib/Driver/ToolChains/Darwin.cpp?utf8=%E2%9C%93#L1910

    Tamir Duberstein at 2017-11-25 00:37:58

  12. If someone still wants to pursue this (in which I'm not actually sure), then it can be done step-by-step.

    The next logical step is move from -nodefaultlibs (already used, makes gcc omit libraries like -lc or -lgcc) to -nostdlib (makes gcc omit CRT objects like crt1.o as well).

    The CRT objects used by different toolchains are documented in https://github.com/rust-lang/rust/pull/71769. The object files can be shipped with rustc (like it's currently done with musl), or found on the system (that may be harder, but should cause less issues like mingw example shows).

    Vadim Petrochenkov at 2020-05-09 15:58:37

  13. Some negative experience with linking CRT objects manually instead of relying on gcc on musl targets: https://github.com/rust-lang/rust/pull/40113#issuecomment-311049045 https://github.com/rust-lang/rust/pull/40113#issuecomment-311049710

    So, musl experience repeats the mingw one - using system CRT objects and libraries is strongly preferable to using those shipped with rustc.

    Vadim Petrochenkov at 2020-05-09 22:46:47