Cannot link to `--crate-type dylib`

Opened by Alex Crichton at 2020-09-07 20:17:35

$ cat foo.rs
pub fn foo() {}
$ cat bar.rs
extern crate foo;

fn main() {
    foo::foo();
}
$ rustc foo.rs --crate-type dylib
$ rustc bar.rs -L .
error: cannot satisfy dependencies so `std` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `core` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `collections` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `rustc_unicode` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `alloc` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `rand` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `libc` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `unwind` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `panic_unwind` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot link together two allocators: alloc_jemalloc and alloc_system
error: aborting due to 10 previous errors

I believe that there's no reason this executable couldn't be created, it should just use all dependencies through the dylib that was created in the previous step.

cd $(mktemp -d)
cat <<EOF > foo.rs
pub fn foo() {}
EOF
cat <<EOF > bar.rs
extern crate foo;

fn main() {
    foo::foo();
}
EOF
rustc foo.rs --crate-type dylib
rustc bar.rs -L .

error: cannot satisfy dependencies so `std` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `core` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `collections` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `rustc_unicode` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `alloc` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `rand` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `libc` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `unwind` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot satisfy dependencies so `panic_unwind` only shows up once
help: having upstream crates all available in one format will likely make this go away
error: cannot link together two allocators: alloc_jemalloc and alloc_system
error: aborting due to 10 previous errors

I believe that there's no reason this executable couldn't be created, it should just use all dependencies through the dylib that was created in the previous step.

Martin Nordholts at 2023-12-17 12:35:49

The workaround for us was to define a process where all git repos have zero tests, and all tests are moved into another git repo.

It's critical for us that we can build a Rust DSO. It would be nice if the test cases for a DSO could be included with the git repo that contains the DSO code.
MarkSwanson at 2016-07-28 20:06:05
@MarkSwanson note that the crate type "cdylib" is most appropriate for creating a Rust DSO. If you're trying to use a dylib linking to other Rust dylibs it'll be a nonstop world of pain right now unfortunately, but cdylib should be quite smooth.
Alex Crichton at 2016-07-28 22:09:31
I believe that there's no reason this executable couldn't be created, it should just use all dependencies through the dylib that was created in the previous step.

If this were a diamond dependency, where D depends on B and C which in turn both depend on A:
```
  A
 / \
B   C
 \ /
  D
```
would you also expect this to work?

Currently if you build B and C as dynamic, then they will each link their own static copy of A. When you try to link both into A, it will fail with error: cannot satisfy dependencies so 'A' only shows up once. In principle, if A is stateless then it doesn't matter which version gets used, or even a mix and match. But if A has state (eg lazy_static) then there's a semantic question of whether B and C should each get separate state or share the same state (and if so, how to make sure they do).

The simplest thing is if A is dynamically linked to both B and C, so a single instance is shared between B and C, and there's no need to disambiguate them.

(Edit: swapped A and D to match the answer below)
Jeremy Fitzhardinge at 2017-01-23 22:50:50
@jsgf it depends on how all the linkages work out, we've got a test for this in tree for the various versions though. The gist is:

| A | B | C | D | links? | |----|---|---|---|---| | rlib | rlib | rlib | rlib | ✓ | | dylib | rlib | rlib | rlib | ✓ | | rlib | dylib | rlib | rlib | ✓ | | dylib | dylib | rlib | rlib | ✓ | | rlib | rlib | dylib | rlib | ✓ | | dylib | rlib | dylib | rlib | ✓ | | rlib | dylib | dylib | rlib | | | dylib | dylib | dylib | rlib | ✓ | | rlib | rlib | rlib | dylib | ✓ | | dylib | rlib | rlib | dylib | ✓ | | rlib | dylib | rlib | dylib | ✓ | | dylib | dylib | rlib | dylib | ✓ | | rlib | rlib | dylib | dylib | ✓ | | dylib | rlib | dylib | dylib | ✓ | | rlib | dylib | dylib | dylib | | | dylib | dylib | dylib | dylib | ✓ |

The only failure mode is when B/C are both dylibs, statically linking A, so D can't be guaranteed one copy of A. In other cases A is only included once as it's statically linked into only one dylib.

The compiler isn't very smart about this today, though. It has a very simple heuristic where if it doesn't work on the first try it emits an error.
Alex Crichton at 2017-01-23 23:23:43
Compiling it like this does work:
```
> rustc foo.rs --crate-type dylib -C prefer-dynamic
> rustc bar.rs -L .
```
But then you have to dynamically link to libstd because it won't get statically linked into libfoo.
Zicklag at 2019-06-05 00:33:00
Is there any plan for the fix? I'm creating a large application and now the linkage took more than 2 minutes.

Should I make every module cdylib? That would change all the interface IIRC.
Clouds at 2020-09-07 20:17:35