Program cross compiled for mips-unknown-linux-musl crashes with SIGILL

Opened by Jorge Aparicio at 2019-03-26 12:48:22

cc @Bigomby @alexcrichton

Originally reported as japaric/rust-cross#27

Original report below:

Hi, I'm trying to cross compile a Rust app for LEDE (fork of OpenWRT) but I'm stuck.

The application is just an hello_world example:

fn main() {
    println!("Hello, world");
}

These are the steps I performed:

In first place I downloaded the LEDE SDK and compiled successfully the image and the toolchain for my device (mips-openwrt-linux-musl-gcc).
Install the Rust std using rustup add mips-unknown-linux-musl.

Create a .cargo/config file in my project with the following content:

[target.mips-unknown-linux-musl]
linker = "mips-openwrt-linux-musl-gcc"

Build the app using cargo build --release --target=mips-unknown-linux-musl.
Deploy the executable to the device using scp.
Got Illegal instruction

On target

$ ldd hello_world
/lib/ld-musl-mips-sf.so.1 (0x558c2000)                                                 
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x77cc4000)                                       
libc.so => /lib/ld-musl-mips-sf.so.1 (0x558c2000)  

$ ldd --version
musl libc (mips-sf)
Version 1.1.15
Dynamic Program Loader
Usage: ldd [options] [--] pathname

On host

$ mips-openwrt-linux-musl-gcc -v
Reading specs from /opt/lede/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/bin/../lib/gcc/mips-openwrt-linux-musl/5.4.0/specs
COLLECT_GCC=mips-openwrt-linux-musl-gcc
COLLECT_LTO_WRAPPER=/opt/lede/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/bin/../libexec/gcc/mips-openwrt-linux-musl/5.4.0/lto-wrapper
Target: mips-openwrt-linux-musl
Configured with: /home/diego/source/build_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/gcc-5.4.0/configure --with-bugurl=http://www.lede-project.org/bugs/ --with-pkgversion='LEDE GCC 5.4.0 r2032' --prefix=/home/diego/source/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15 --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=mips-openwrt-linux-musl --with-gnu-ld --enable-target-optspace --disable-libgomp --disable-libmudflap --disable-multilib --disable-libmpx --disable-nls --without-isl --without-cloog --with-host-libstdcxx=-lstdc++ --with-float=soft --with-gmp=/home/diego/source/staging_dir/host --with-mpfr=/home/diego/source/staging_dir/host --with-mpc=/home/diego/source/staging_dir/host --disable-decimal-float --with-mips-plt --with-diagnostics-color=auto-if-env --disable-libssp --enable-__cxa_atexit --with-headers=/home/diego/source/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/include --disable-libsanitizer --enable-languages=c,c++ --enable-shared --enable-threads --with-slibdir=/home/diego/source/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/lib --enable-lto --with-libelf=/home/diego/source/staging_dir/host
Thread model: posix
gcc version 5.4.0 (LEDE GCC 5.4.0 r2032)

Disabling jemalloc

If I use rust nightly and disable jemalloc as following:

#![feature(alloc_system)]
extern crate alloc_system;

fn main() {
    println!("Hello, world");
}

Then it works! But as I keep adding code the application eventually crashes with another Illegal instruction. For example:

Works

TcpStream::connect("130.206.193.115:80").unwrap();

Does not work

TcpStream::connect("google.com:80").unwrap();'

I don't know what I'm doing wrong. Any help will be appreciated.

@Bigomby

I don't know what I'm doing wrong.

The only thing that I think you could be doing wrong is compiling the binary for MIPS (big endian) but running it on a MIPSel (little endian) device but then not even the "Hello, world!" program that uses system_alloc would work so I'm going to discard that possibility.

I'd like to check a few things:
- What device are you running the binaries on? I'd like to know known the exact architecture and CPU model but if you can at least tell me the brand and model of the router/board/device you are using, that would be a great start.
- Could you point me to the LEDE SDK, its URL, that you are using?
- Do you have access to GDB on the target device? Any chance you could get a backtrace/disassemble of a program at the moment that it crashes?
My guess, since C code is involved in all the crashes, is that we are building our binary releases of std with C compiler flags that generate instructions that are not available on your device. We'll hopefully only need to tweak those and then a new nightly release of std should work for you.

If it's not the C side, then we may need to tweak our LLVM codegen options.

Or it could be a bad libc definition.
Jorge Aparicio at 2016-11-01 03:55:09

The device is a Huawei EchoLife HG556a.
I got the SDK from this repo https://github.com/lede-project/source (commit 76847c7)

I got this when running with gdb

#![feature(alloc_system)]
extern crate alloc_system;

use std::net::TcpStream;

fn main() {
  TcpStream::connect("google.com:80").unwrap();
}

(gdb) r
Starting program: /root/kinton-yeelight 
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Program received signal SIGILL, Illegal instruction.
0x5556269c in ?? ()
(gdb) bt
#0  0x5556269c in ?? ()
#1  0x55565fdc in _$LT$str$u20$as$u20$std..net..addr..ToSocketAddrs$GT$::to_socket_addrs::h7d1fe6bb64ce9c00 ()
#2  0x55560cec in _ftext ()

Diego Fernández at 2016-11-01 15:39:49

This is what happends when using jemalloc

fn main() {
    println!("Hello world");
}

(gdb) r
Starting program: /root/hello_world 
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Program received signal SIGILL, Illegal instruction.
je_arena_mapbits_binind_get () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:801
801     /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h: No such file or directory.
(gdb) bt
#0  je_arena_mapbits_binind_get () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:801
#1  je_arena_salloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:1317
#2  je_isalloc () at include/jemalloc/internal/jemalloc_internal.h:924
#3  je_iallocztm () at include/jemalloc/internal/jemalloc_internal.h:937
#4  a0ialloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:321
#5  je_a0malloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:336
#6  je_arena_tdata_get_hard () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:519
#7  0x5558f7dc in je_arena_tdata_get () at include/jemalloc/internal/jemalloc_internal.h:818
#8  je_decay_ticker_get () at include/jemalloc/internal/jemalloc_internal.h:842
#9  je_arena_decay_ticks () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:1253
#10 je_arena_decay_tick () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:1264
#11 je_arena_malloc_large () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:2476
#12 0x55590ad4 in je_arena_malloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:1287
#13 je_arena_palloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:2601
#14 0x555b2420 in je_ipallocztm () at include/jemalloc/internal/jemalloc_internal.h:982
#15 je_tcache_create () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/tcache.c:328
#16 0x555849b0 in je_tcache_get () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/tcache.h:236
#17 je_imalloc () at include/jemalloc/internal/jemalloc_internal.h:954
#18 imallocx_no_prof () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:2135
#19 mallocx () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:2164
#20 0x5557bb30 in alloc_jemalloc::imp::__rust_allocate () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/liballoc_jemalloc/lib.rs:106
#21 0x5557aaa4 in alloc::heap::allocate () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/liballoc/heap.rs:59
#22 alloc::raw_vec::{{impl}}::with_capacity<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/liballoc/raw_vec.rs:96
#23 collections::vec::{{impl}}::with_capacity<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/vec.rs:356
#24 collections::slice::hack::to_vec<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/slice.rs:160
#25 collections::slice::{{impl}}::to_vec<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/slice.rs:1161
#26 collections::slice::{{impl}}::to_owned<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/slice.rs:1281
#27 collections::str::{{impl}}::to_owned () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/str.rs:167
#28 0x5556c934 in std::rt::lang_start () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libstd/rt.rs:50
#29 0x55561cd8 in main ()
warning: GDB can't find the start of the function at 0x77f78acf.

    GDB is unable to find the start of the function at 0x77f78acf
and thus can't determine the size of that function's stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
    This problem is most likely caused by an invalid program counter or
stack pointer.
    However, if you think GDB should simply search farther back
from 0x77f78acf for code which looks like the beginning of a
function, you can increase the range of the search using the `set
heuristic-fence-post' command.

Diego Fernández at 2016-11-01 15:58:53

A SIGILL probably means that we're miscompiling jemalloc somehow (maybe some weird codegen option) but if it still faults with the system allocator then it's likely not exclusively a jemalloc problem.
Alex Crichton at 2016-11-01 16:03:02

I tried to run the program on an OM2P (MIPS device) and it's working fine, even with jemalloc. OM2P uses an Atheros AR7240 so the issue seems to be something more specific to Broadcom 6358.

UPDATE: It crashes also when using glibc instead musl on the Huawei EchoLife HG556a.

mips-openwrt-linux-gnu-gcc -v</summary>

Reading specs from /opt/l/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/bin/../lib/gcc/mips-openwrt-linux-gnu/6.2.0/specs
COLLECT_GCC=/opt/l/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/bin/mips-openwrt-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/opt/l/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/bin/../libexec/gcc/mips-openwrt-linux-gnu/6.2.0/lto-wrapper
Target: mips-openwrt-linux-gnu
Configured with: /home/diego/lede/build_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/gcc-6.2.0/configure --with-bugurl=http://www.lede-project.org/bugs/ --with-pkgversion='LEDE GCC 6.2.0 r2065' --prefix=/home/diego/lede/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24 --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=mips-openwrt-linux-gnu --with-gnu-ld --enable-target-optspace --disable-libgomp --disable-libmudflap --disable-multilib --disable-libmpx --disable-nls --without-isl --without-cloog --with-host-libstdcxx=-lstdc++ --with-float=soft --with-gmp=/home/diego/lede/staging_dir/host --with-mpfr=/home/diego/lede/staging_dir/host --with-mpc=/home/diego/lede/staging_dir/host --disable-decimal-float --with-mips-plt --with-diagnostics-color=auto-if-env --enable-libssp --enable-__cxa_atexit --with-headers=/home/diego/lede/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/include --disable-libsanitizer --enable-languages=c,c++ --enable-shared --enable-threads --with-slibdir=/home/diego/lede/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/lib --enable-lto --with-libelf=/home/diego/lede/staging_dir/host
Thread model: posix
gcc version 6.2.0 (LEDE GCC 6.2.0 r2065)

</summary>

Diego Fernández at 2016-11-04 13:29:28

Interesting. Then this could be due to codegen options related to "architecture level" (-march) or CPU optimizations (-mcpu) in the sense that the current rustc options generate instructions that are not available on some of these devices. Unfortunately, I don't know the MIPS ISA well enough to confirm this.
Jorge Aparicio at 2016-11-07 14:47:29
This is probably due to the ISA level. The HG556a seems to use MIPS32r1 and the OM2P uses MIPS32r2. I think LLVM defaults to MIPS32r2 unless told otherwise. It would be interesting to get a disassembly at the SIGILL, then I could tell you for certain that this was the problem.
James Cowgill at 2017-02-18 01:58:06
I'm experiencing this issue too (on a Broadcom chip, though). The chip is BCM63168. I have a working OpenWRT toolchain for it. Using mips-unknown-linux-musl as a target with rustc\Cargo works for a simple hello world application, but for anything more complicated I get an illegal instruction.
Matan Rosenberg at 2017-09-17 13:24:49
I have suffered similar losses in the battle for Illegal Instructions on mips. My research so far and i hope this helps:

Minimal build for VU+ DUO2 mipsel-unknown-linux-gnu with Rust on Windows

To save you some hours i describe the process how i build a hello world without ending up with Illegal instruction errors on the BCM7425B2 STB platform.

Target Specs
```
root@vuduo2:~# cat /proc/cpuinfo
system type             : BCM7425B2 STB platform
machine                 : Unknown
processor               : 0
cpu model               : Broadcom BMIPS5000 V1.1  FPU V0.1
BogoMIPS                : 864.25
cpu MHz                 : 1305.097
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 64
extra interrupt vector  : yes
hardware watchpoint     : no
isa                     : mips1 mips2 mips32r1
ASEs implemented        :
shadow register sets    : 1
kscratch registers      : 0
core                    : 0
VCED exceptions         : not available
VCEI exceptions         : not available

processor               : 1
cpu model               : Broadcom BMIPS5000 V1.1  FPU V0.1
BogoMIPS                : 655.36
cpu MHz                 : 1305.097
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 64
extra interrupt vector  : yes
hardware watchpoint     : no
isa                     : mips1 mips2 mips32r1
ASEs implemented        :
shadow register sets    : 1
kscratch registers      : 0
core                    : 0
VCED exceptions         : not available
VCEI exceptions         : not available
```
What you need

To compile for the target i used Codescape GNU Tools 2018.09-03 Binaries https://codescape.mips.com/components/toolchain/2018.09-03/downloads.html installed in C:\Toolchain\MIPS\mips-2016.05.

Rust components used:
```
E:\Projects\Rust\vuduo2\hello-vuduo>rustup show
Default host: x86_64-pc-windows-gnu

installed targets for active tool chain
--------------------------------------

mipsel-unknown-linux-gnu
x86_64-pc-windows-gnu

active tool chain
----------------

nightly-x86_64-pc-windows-gnu (default)
rustc 1.35.0-nightly (719b0d984 2019-03-13)
```
Install "The sysroot manager that lets you build and customize std" xargo.
```
cargo install xargo
```
Setup

Environment

Paths are specific to my setup and locations. This can vary in your case and you have to adjust them.

Add your cross-compiler to the search path
```
set PATH=C:\Toolchain\MIPS\mips-2016.05\bin;%PATH%
```
Specify the compiler setup to use:
```
set CC=mips-linux-gnu-gcc
set CFLAGS_mipsel-unknown-linux-gnu=-EL
set AR_mipsel-unknown-linux-gnu=mips-linux-gnu-ar
```
- where CC specifies the compiler you use to build the target and std crate binaries
- CFLAGS_mipsel-unknown-linux-gnu the specific compiler switches to use. In this case -EL tells the compiler to build code for little endian processors (the el in mipsel). Other compilers may come for a specific architecture and are named mipsel-linux-gnu-gcc, but in my case the compiler is mips-linux-gnu-gcc and can output in both endianess. You have to consider this when using a different tool chain. To learn more about the capabilities of the compiler use mips-linux-gnu-gcc --target-help.
- The AR_mipsel-unknown-linux-gnu binary util is used to create, modify, and extract from archives (libraries).
Build files

Any minimal Cargo.toml. Nothing special:
```
[package]
name = "hello-vuduo"
version = "0.1.0"
authors = ["Jedzia <jed69@gmx.de>"]
edition = "2018"
```
Build file for Xargo Xargo.toml:
```
[dependencies]
std = {features = ["panic_unwind"]}

[build]
target = "mipsel-linux-gnu-gcc"
```
- std: The std dependency is what you want to build on your own. This is the cause of Illegal instruction errors on mips32r1 targets as the upstream target from the Rust repository is built for mips32r2 with fpxx and nooddspreg enabled. This change was done on Mar 15, 2018: bump mipsel isa leval and enable fpxx #48874
- The [build] target specifies the default architecture to build. Can also be specified at the command-line with
```
  cargo build --target=mipsel-unknown-linux-gnu
  xargo build --target=my-unknown-system-dos
```
Cargo configuration

Local cargo configuration .cargo/config:
```
[build]
target = "mipsel-unknown-linux-gnu"

[target.mipsel-unknown-linux-gnu]
rustflags = [
  "-C", "link-arg=-EL",
  "-C", "target-cpu=mips32",
  "-C", "target-feature=+mips32,-mips32r2,-fpxx,-nooddspreg",
]

ar = "mips-linux-gnu-ar"
linker = "mips-linux-gnu-gcc"
runner = "run.bat"  
```
- The [build] target specifies the default architecture to build (see above).
- Under [target.mipsel-unknown-linux-gnu] we specify the flags used for creating the binary and linking it. -C is forwarded to the rustc-compiler and indicates a code-generation option. rustc -C help will list all options available. The default target specs for the current rustc compiler can be listed with: rustc -Z unstable-options --print target-spec-json --target mipsel-unknown-linux-gnu. The used options in detail:
  - link-arg=-EL is daisy-chained to the linker(mips-linux-gnu-gcc): little endian output is generated.
  - target-cpu=mips32 for use with our good old mips32r1. A list can be retrieved with rustc --print target-cpus. We are building for mips so with our target: rustc --target mipsel-unknown-linux-gnu --print target-cpus.
  - target-feature: For a list of available options use rustc --print target-features or in our case: rustc --target mipsel-unknown-linux-gnu -C target-cpu=mips32 -C target-feature=help.
    - +mips32: Enable Mips32 ISA Support.
    - -mips32r2: Disable Mips32r2 ISA Support ... illegal instruction.
    - -fpxx: Disable support for FPXX. The FPXX extension mandates that all code must execute correctly when run using 32-bit or 64-bit registers. The code can be interlinked with either FP32 or FP64, but not both.
    - -nooddspreg: Disable odd numbered single-precision registers.
- ar: binary util used to create, modify, and extract from archives (libraries). Use our target specific one.
- linker: mips-linux-gnu-gcc is used as linker in conjunction with the specified link-arg=-EL flag to produce little endian binaries.
- runner: What (x/c)argo uses to execute the run command. The first command-line parameter is the relative path of your binary. I use a combination of scp to the target-box or copy to a network share and ssh to run it remotely. Something like: run.bat: cp %1 \VUDUO2/Harddisk/jedzia/rust ssh jedzia@vuduo2 /media/hdd/jedzia/rust/hello-vuduo
I was able to successfully compile and run with the fpxx and nooddspreg options enabled ("-C", "target-feature=+mips32,-mips32r2,+fpxx,+nooddspreg"). Maybe they are not necessary for the VU+ Duo2. But i can't prove this as i only compile a little hello world with floating point arithmetic in it. When it fails you have to tweak it for your target. I have disabled these options as they are part of the #48874 change set (see above).

Compiling

run
```
xargo build
```
or if you have set up a runner in .cargo/config for remote execution
```
xargo run 
```
to see it work.

and
```
cargo build
```
to see it fail again with illegal instruction as it is compiled against the Rust mipsel-unknown-linux-gnu target and not our custom one built by xargo.
Jedzia at 2019-03-25 01:44:27
I cant withstand not to add an example: https://github.com/Jedzia/RokerDepp
Jedzia at 2019-03-26 12:48:22

Disabling jemalloc

Minimal build for VU+ DUO2 mipsel-unknown-linux-gnu with Rust on Windows

What you need

Setup

Environment

Build files

Cargo configuration

Compiling