Bootstrap fails due to system LLVM libffi link error

04a220b
Opened by Taylor Trump at 2024-10-13 23:30:09

Trying to bootstrap rust on aarch64 Linux using --llvm-root fails:

error: linking with `cc` failed: exit code: 1
note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc.0.o" "-o" "aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc" "-Wl,--gc-sections" "-pie" "-Wl,-O1" "-nodefaultlibs" "-L" "/usr/lib/llvm-3.8/lib" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_driver-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_passes-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_borrowck-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_lint-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_resolve-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_typeck-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_plugin-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_privacy-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_trans-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_platform_intrinsics-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_incremental-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "syntax_ext-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_mir-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_const_eval-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_metadata-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_save_analysis-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_const_math-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "getopts-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "flate-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_back-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rbml-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_data_structures-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "syntax-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "serialize-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "log-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "term-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "fmt_macros-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "arena-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_llvm-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "graphviz-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "std-fe3cdf61" "-l" "rt" "-l" "dl" "-l" "tinfo" "-l" "pthread" "-l" "z" "-l" "m" "-l" "stdc++" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "util" "-Wl,-rpath,$ORIGIN/../lib" "-Wl,-rpath,/usr/local/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,--enable-new-dtags" "-s" "-l" "compiler-rt"

note: ../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_float'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint64'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_pointer'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint32'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_void'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint8'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_call'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint16'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_double'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_prep_cif'
collect2: error: ld returned 1 exit status

error: aborting due to previous error
../mk/target.mk:220: recipe for target 'aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc' failed
make: *** [aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc] Error 101

For some reason the correct llvm configuration is not being detected and -lffi has to be passed manually for the bootstrap to succeed.

  1. In-tree, we configure LLVM without -lffi so maybe there's a discrepancy when we encounter an LLVM with it compiled in.

    Brian Anderson at 2016-06-27 20:38:07

  2. Yup, the LLVM I was bootsrapping against was clearly built with LLVM_ENABLE_FFI.

    Taylor Trump at 2016-06-27 21:11:59

  3. Running into the same isse. Shouldn't llvm-config output this requirement?

    Btw for anyone reading this: a quick fix is to add:

    #[link(name = "ffi")] extern {}

    to src/librustc_llvm/lib.rs.

    jethrogb at 2016-07-10 23:34:08

  4. I'm also experiencing this on x86_64 Ubuntu 16.04 using the llvm-dev package and --llvm-root.

    Jorge Aparicio at 2016-08-17 15:24:57

  5. The title should be edited then, and btw, the quickest non-permanent solution looks like this: RUSTFLAGS='-C link-args=-lffi'

    Taylor Trump at 2016-08-19 19:10:50

  6. It seems this is a duplicate of #29694

    Jorge Aparicio at 2016-08-27 23:32:24

  7. Recently, I have been annoyed by this a lot. I've been boostraping rustc against system llvm a lot in different Docker containers and I have to remember to apply @jethrogb patch/fix to every rust branch/worktree I work on and then I have to be careful to not commit this fix to the branch I'm working on. :disappointed:

    Could we simply commit @jethrogb's fix into master? I tested bootstrapping rust against our llvm fork with that fix merged into master (5a5736db916ac30ca67945bbf0aee41ced1fcf05) and at least, on Linux, it bootstraped without problems.

    P.S. @petevine's RUSTFLAGS solution didn't work for me.

    Jorge Aparicio at 2016-09-15 21:03:18

  8. @japaric That's all there's to it, and it definitely works for the Makefiles bootstrap.

    Taylor Trump at 2016-09-15 21:21:27

  9. This is of course not intentional, and fixes are always welcome! The trick here is detecting whether we're building against an LLVM that needs libffi. For example the in-tree LLVM doesn't and we shouldn't link to it as a result.

    Alex Crichton at 2016-09-16 08:09:49

  10. FWIW, Fedora's LLVM is built with FFI, but this is usually fine because libLLVM.so links to libffi.so itself. However, with llvm-static.rpm installed, llvm-config starts giving static link options (but not -lffi), and ldd -r librustc_llvm-*.so indeed shows a few undefined symbol: ffi_....

    Josh Stone at 2016-09-16 16:56:28

  11. IOW I blame llvm-config, not that blame solves anything. (I also don't like that it doesn't let me choose shared linkage even when static libraries are present...)

    Josh Stone at 2016-09-16 17:00:38

  12. @cuviper Regarding 'choose shared over static llvm': If I understand correctly, since 3.9 llvm-config will prefer the combo-dynlib by default over any static libs present. And even if you don't have the combo, but say shared and static component libs installed, there are now options to specifically choose --link-static or --link-shared.

    Relevant llvm commit explains it: https://github.com/llvm-mirror/llvm/commit/93083d498658f4c6da98e8950bd9f58c76080299

    Doesn't fix this bug here, external llvm + ffi enabled + intentionally linking llvm static libs still triggers it, just thought it would be nice to know.

    (BTW, the ffi-issue seems to be sort of a known shortcoming of llvm-config --system-libs. It only statically lists the LLVMSupport sysdeps, but ffi is the (currently only?) sysdep from another component - LLVMExecutionEngine)

    userwithuid at 2016-10-30 08:51:39

  13. @userwithuid Thanks, I wasn't aware of that! I'll try that out, and maybe we should expose those explicit link options somehow too.

    Josh Stone at 2016-10-30 15:30:00

  14. Please tell me the current situation; rustbuild is enabled by default, the old workarounds may not work.

    Tatsuyuki Ishi at 2017-01-29 11:44:11

  15. Yeah, last time I tried it didn't even build due to some completely unfamiliar link errors. (edit: I meant exactly this)

    If it's still a problem in the new build system, please open a more specific issue.

    Taylor Trump at 2017-01-30 11:52:37

  16. I'm quite sure that this is still a problem; please don't blindly close it.

    Tatsuyuki Ishi at 2017-01-30 11:55:30

  17. Yes this should remain open. @MagaTailor I don't understand what you mean, this issue is quite specific.

    jethrogb at 2017-01-30 20:00:54

  18. I went ahead and filed a new bug 39880.

    Hussam Al-Tayeb at 2017-02-16 10:53:20

  19. @csmoe Is this an issue again in post 1.31.1 releases? I have version 1.31.1 installed and did not see this issue when compiling rust against system llvm 7.0.1.

    Hussam Al-Tayeb at 2019-01-09 17:09:45

  20. @hussamT IIRC it was only an issue with linking external LLVM statically, and even then only if that build is configured with -DLLVM_ENABLE_FFI:BOOL=ON. At least on Fedora rawhide, I still get:

    $ rpm -q llvm
llvm-7.0.1-1.fc30.x86_64
$ llvm-config --link-static --system-libs
-lz -lrt -ldl -ltinfo -lpthread -lm

    ... missing -lffi, even though some of the static libs do use it. But this is really an LLVM problem, as noted in https://github.com/rust-lang/rust/issues/34486#issuecomment-257139448, so I'm not sure we need this Rust issue open except as a "known issue, see elsewhere" tracker.

    Our official Fedora rust builds are linked dynamically (--enable-llvm-link-shared), and this is fine.

    Josh Stone at 2019-01-09 17:40:47

  21. Triage: this seems to still be an issue today: https://github.com/llvm-mirror/llvm/blob/2c4ca6832fa6b306ee6a7010bfb80a3f2596f824/tools/llvm-config/llvm-config.cpp#L705

    jyn at 2022-06-27 04:55:13

  22. @jyn514 FYI that linked comment is not the problem. The problem is LLVM doesn't add libffi to LLVM_SYSTEM_LIBS. Some time ago I have fixed libz3 which had the same issue: https://reviews.llvm.org/D85195 but libffi is not that easy to fix. Somebody could use it as an guidance on fixing libffi though.

    Mateusz MikuĊ‚a at 2022-06-27 17:44:04