Tracking issue for RFC 1861: Extern types

Opened by Aaron Turon at 2024-12-02 19:20:33

This is a tracking issue for RFC 1861 "Extern types".

Steps:

[x] Implement the RFC (#44295)
[ ] Adjust documentation (see instructions on forge)
[ ] Stabilization PR (see instructions on forge)

Unresolved questions:

Should we allow generic lifetime and type parameters on extern types? If so, how do they effect the type in terms of variance?
In std's source, it is mentioned that LLVM expects i8* for C's void*. We'd need to continue to hack this for the two c_voids in std and libc. But perhaps this should be done across-the-board for all extern types? Somebody should check what Clang does. Also see https://github.com/rust-lang/rust/issues/59095.

This is a tracking issue for RFC 1861 "Extern types".

Steps:
- [x] Implement the RFC (#44295)
- [ ] Adjust documentation (see instructions on forge)
- [ ] Stabilization PR (see instructions on forge)
Unresolved questions:
- [ ] Rust does not support types that don't have dynamically computed alignment -- we need the alignment to compute the field offset in structs. extern type violates this basic assumption, causing pain, suffering, and ICEs all over the compiler. What is the principled fix for this?
- [ ] Should we allow generic lifetime and type parameters on extern types? If so, how do they effect the type in terms of variance?
- [x] In std's source, it is mentioned that LLVM expects i8* for C's void*. We'd need to continue to hack this for the two c_voids in std and libc. But perhaps this should be done across-the-board for all extern types? Somebody should check what Clang does. Also see https://github.com/rust-lang/rust/issues/59095. RESOLVED because all pointer types are ptr now.
- [ ] How should this interact with unsized arguments? Currently it ICEs: https://github.com/rust-lang/rust/issues/115709
Ralf Jung at 2023-12-02 14:19:15
This is not explicitly mentioned in the RFC, but I'm assuming different instances of extern type are actually different types? Meaning this would be illegal:
```
extern {
    type A;
    type B;
}

fn convert_ref(r: &A) -> &B { r }
```
jethrogb at 2017-07-25 05:30:47
@jethrogb That's certainly the intention, yes.
Andrew Cann at 2017-07-25 05:32:11
Relatedly, is deciding whether we want to call it extern type or extern struct something that can still be done as part of the stabilization process, or is the extern type syntax effectively final as of having accepted the RFC?

EDIT: https://github.com/rust-lang/rfcs/pull/2071 is also relevant here w.r.t. the connotations of type "aliases". In stable Rust a type declaration is "effect-free" and just a transparent alias for some existing type. Both extern type and type Foo = impl Bar would change this by making it implicitly generate a new module-scoped existential type or type constructor (nominal type) for it to refer to.
Gábor Lehel at 2017-07-25 08:24:11
Can we get a bullet for the panic vs DynSized debate?
John Ericson at 2017-07-25 13:15:59
I've started working on this, and I have a working simple initial version (no generics, no DynSized).

I've however noticed a slight usability issue. In FFI code, it's frequent for raw pointers to be initialized to null using std::ptr::null/null_mut. However, the function only accepts sized type arguments, since it would not be able to pick a metadata for the fat pointer.

Despite being unsized, extern types are used through thin pointers, so it should be possible to use std::ptr::null.

It is still possible to cast an integer to an extern type pointer, but this is not as nice as just using the function designed for this. Also this can never be done in a generic context.
```
extern {
    type foo;
}
fn null_foo() -> *const foo {
    0usize as *const foo
}
```
Really we'd want is a new trait to distinguish types which use thin pointers. It would be implemented automatically for all sized types and extern types. Then the cast above would succeed whenever the type is bounded by this trait. Eg, the function std::ptr::null becomes :
```
fn null<T: ?Sized + Thin>() -> *const T {
    0usize as *const T
}
```
However there's a risk of more and more such traits creeping up, such as DynSized, making it confusing for users. There's also some overlap with the various custom RFCs proposals which allow arbitrary metadata. For instance, instead of Thin, Referent<Meta=()> could be used
Paul Liétar at 2017-08-10 21:29:11
I think we can add extern types now and live with str::ptr::null not supporting them for a while until we figure out what to do about Thin/DynSized/Referent<Meta=…> etc.
Simon Sapin at 2017-08-10 21:36:01
@SimonSapin yeah, it's definitely a minor concern for now.

I do think this problem of not having a trait bound to express "this type may be unsized be must have a thin pointer" might crop up in other places though.
Paul Liétar at 2017-08-10 21:38:53
Oh yeah, I agree we should solve that eventually too. I’m only saying we might not need to solve all of it before we ship any of it.
Simon Sapin at 2017-08-11 07:25:07
I've pushed an initial implementation in #44295
Paul Liétar at 2017-09-03 20:13:32
@plietar In #44295 you wrote

Auto traits are not implemented by default, since the contents of extern types is unknown. This means extern types are !Sync, !Send and !Freeze. This seems like the correct behaviour to me. Manual unsafe impl Sync for Foo is still possible.

While it is possible for Sync, Send, UnwindSafe and RefUnwindSafe, doing impl Freeze for Foo is not possible as it is a private trait in libcore. This means it is impossible to convince the compiler that an extern type is cell-free.

Should Freeze be made public (even if #[doc(hidden)])? cc @eddyb #41349.

Or is it possible to declare an extern type is safe-by-default, which opt-out instead of opt-in?
```
extern {
    #[unsafe_impl_all_auto_traits_by_default]
    type Foo;
}
impl !Send for Foo {}
```
kennytm at 2017-11-16 15:21:16
@kennytm What's the usecase? The semantics of extern type are more or less that of a hack being used before the RFC, which is struct Opaque(UnsafeCell<()>);, so the lack of Freeze fits. That prevents rustc from telling LLVM anything different from what C signatures in clang result in.
Eduard-Mihai Burtescu at 2017-11-16 17:18:25
@eddyb Use case: Trying to see if it's possible to make CStr a thin DST.

I don't see anything related to a cell in #44295? It is reported to LLVM as an i8 similar to str. And the places where librustc_trans involves the Freeze trait reads the real type, not the LLVM type, so LLVM treating all extern type as i8 should be irrelevant?
kennytm at 2017-11-16 17:55:57
@kennytm So with extern type CStr;, writes through &CStr would be legal, and you don't want that? The Freeze trait is private because it's used to detect UnsafeCell and not meant to be overriden.
Eduard-Mihai Burtescu at 2017-11-16 18:10:40
The original intent was to match the extern type CStr with the existing behavior of struct CStr([c_char]) which is Freeze. Eddyb and I discussed on IRC, which assures that (1) Freeze is mainly used to disable certain optimizations only and (2) as Freeze is a private trait, no one other than the compiler will rely on it. So the missing Freeze trait will be irrelevant for extern type CStr.
kennytm at 2017-11-16 18:28:04
Regarding the thin pointer issue, I imagine that const generics will eventually enable constant comparisons in where clauses - if size_of is made const also, that would let you write a bound of where size_of::<Ptr>() == size_of::<usize>() which IMO matches the intent pretty perfectly.
Parker Snell at 2017-11-18 00:23:16
This is the first I hear of allowing const expressions in where clauses. While it could be very useful, it seems far from given that this will be accepted into the language.
Simon Sapin at 2017-11-18 08:40:33
@SimonSapin Const expression in where clause will eventually be needed for const generics beyond RFC 2000 (spelled as with in rust-lang/rfcs#1932), but I do think this extension is out-of-scope for extern type or even custom DST in general.
kennytm at 2017-11-18 09:36:53
I didn't mean to assume that that will be supported, I meant that if it did become possible in a reasonable timeframe, which apparently is not likely, I think it'd be nice to have more regular syntax to express some ideas rather than more marker traits with special meaning given by the compiler. If that's not going to work, then great, it's one less thing to consider.
Parker Snell at 2017-11-18 15:08:29
One of my use cases for extern types is to represent opaque things from the macOS frameworks with them.

For that purpose, I actually want to be able to wrap opaque things in some generic types to encode certain invariants related to refcounting.

For example, I want a CFRef<T> type that derefs to CFShared<T> that itself derefs to T.
```
pub struct CFRef<T>(*const CFShared<T>);
pub struct CFShared<T>(T);
```
This is apparently not possible if T is an extern type.
```
pub extern type CFString;
fn do_stuff_with_shared_string(str: &CFShared<CFString>) { ... }
```
Would it be complicated to support such a thing?
Anthony Ramine at 2017-12-12 22:31:34
@nox struct CFShared<T: ?Sized>(T); ?

Note that this may not compile after we have implemented the DynSized trait since an extern type is not DynSized, and we can’t place a !DynSized field inside a struct (may need explicit #[repr(transparent)] to allow it.
kennytm at 2017-12-12 22:48:07
Can somebody provide an update on the status of this and maybe summarize the remaining open issues? I'd like to know if it is possible already to implement c_void in libc/core using extern type, for example.
gnzlbg at 2018-02-03 12:28:08
@gnzlbg as mentioned in the initial post in this issue:

In std's source, it is mentioned that LLVM expects i8* for C's void*. We'd need to continue to hack this for the two c_voids in std and libc. But perhaps this should be done across-the-board for all extern types? Somebody should check what Clang does.

There is no solution for this yet, I think.
jethrogb at 2018-03-18 16:05:58
@gnzlbg Whether we want DynSized needs to be resolved as well. The description of the initial implementation does a great job of laying out the footguns that exist without it. Thanks @plietar!
John Ericson at 2018-03-18 16:14:32

C Opaque struct:

typedef struct c_void c_void;

c_void* malloc(unsigned long size);

void call_malloc() {
	malloc(1);
}

clang version 5.0.1:

%struct.c_void = type opaque

define void @call_malloc() #0 {
  %1 = call %struct.c_void* @malloc(i64 1)
  ret void
}

declare %struct.c_void* @malloc(i64) #1

C void:

void* malloc(unsigned long size);

void call_malloc() {
	malloc(1);
}

clang version 5.0.1:

define void @call_malloc() #0 {
  %1 = call i8* @malloc(i64 1)
  ret void
}

declare i8* @malloc(i64) #1

Rust extern type:

#![feature(extern_types)]
#![crate_type="lib"]

extern "C" {
    type c_void;

    fn malloc(n: usize) -> *mut c_void;
}

#[no_mangle]
pub fn call_malloc() {
    unsafe { malloc(1); }
}

Rust nightly:

%"::c_void" = type {}

define void @call_malloc() unnamed_addr #0 !dbg !4 {
start:
  %0 = call %"::c_void"* @malloc(i64 1), !dbg !8
  br label %bb1, !dbg !8

bb1:                                              ; preds = %start
  ret void, !dbg !10
}

declare %"::c_void"* @malloc(i64) unnamed_addr #1

jethrogb at 2018-03-18 16:16:30

Great find! This %struct.c_void = type opaque sure looks worth imitating. If that's slower than i8* IMO that's a clang/LLVM bug to report.
John Ericson at 2018-03-18 16:23:54
Would this syntax be acceptable?
```
extern {
    #[repr(i8)]
    type c_void;
}
```
jethrogb at 2018-03-18 16:24:09
@jethrogb looks good to me, but I'd be tempted to keep it unstable as it only exists to hack around LLVM.
John Ericson at 2018-03-18 16:25:45
I'd be tempted to keep it unstable as it only exists to hack around LLVM.

That sounds good in principle, but I think there were plans to have the final public definition of c_void live in a crates.io crate.
jethrogb at 2018-03-18 16:27:21
@jethrogb hehe just quoted those plans. I do see the tension, bummer.
John Ericson at 2018-03-18 16:29:41
Great find! This %struct.c_void = type opaque sure looks worth imitating. If that's slower than i8* IMO that's a clang/LLVM bug to report.

It is definitely slower than i8*. For example, malloc declared as returning an opaque struct won't be recognized as malloc by most (all?) optimization passes. It's a known issue and according to some LLVM devs the right way to fix it is to get rid of pointee types altogether and have just one pointer type, but that doesn't seem to be happening any time soon so you'll have to emit i8*.
Catherine at 2018-03-18 16:33:10
@whitequark do you happen to have a link to the LLVM bug?
gnzlbg at 2018-03-19 08:14:18
@gnzlbg I'm not sure if there's one, that was from IRC discussions.
Catherine at 2018-03-19 08:49:01
I've tried to search for one without any luck so I've filled this one: https://bugs.llvm.org/show_bug.cgi?id=36795

What @whitequark pointed out looks correct, when using i8* LLVM can eliminate calls to malloc, while when using a type opaque c_void* it cannot.
gnzlbg at 2018-03-19 11:36:17
In the RFC:

As a DST, size_of and align_of do not work, but we must also be careful that size_of_val and align_of_val do not work either, as there is not necessarily a way at run-time to get the size of extern types either. For an initial implementation, those methods can just panic, but before this is stabilized there should be some trait bound or similar on them that prevents their use statically. The exact mechanism is more the domain of the custom DST RFC, RFC 1524, and so figuring that mechanism out will be delegated to it.

However RFC 1524 was closed. Its successor is probably https://github.com/rust-lang/rfcs/issues/2255, but that’s an issue rather than a PR for a new RFC.

Per https://github.com/rust-lang/rust/pull/46108#issuecomment-360903211 the lang team recently decided against having a DynSized trait. But that leaves an unresolved question in this open RFC.

In rustc 1.26.0-nightly (9c9424de5 2018-03-27), this compiles without warning and prints 0:
```
#![feature(extern_types)]

extern { type void; }

fn main() {
    let x: *const void = main as *const _;
    println!("{}", std::mem::size_of_val(unsafe { &*x }));
}
```
The libs team discussed defining a public void extern type in the standard library and changing the return type of memory allocation APIs to *mut void instead of *mut u8. However in that case we’d need to decide what to do about size_of_val + extern types before allocations APIs are stabilized. (Keeping void unstable wouldn’t help, if you can obtain a pointer to it you don’t need to name the type to call size_of_val.)

CC @rust-lang/lang
Simon Sapin at 2018-03-28 21:24:20
rust-lang/rfcs#1524 (Custom DST) is orthogonal to rust-lang/rfcs#2255 (Whether we want more ?Trait). The successor is https://internals.rust-lang.org/t/pre-erfc-lets-fix-dsts/6663.

In #46108 we decided against ?DynSized, but I think a DynSized without a ? (e.g. rust-lang/rfcs#2310 or https://github.com/rust-lang/rfcs/issues/2255#issuecomment-361040229) is still on the table.

BTW for consistency with common C extensions, if the size_of void cannot be undefined, it should be set to 1.
kennytm at 2018-03-28 21:39:40
Conclusions from the lang meeting at the all-hands:
- size_of_val should panic if called on an extern type
- We should have a best-effort lint to statically detect if you call size_of_val on an extern type, either directly or ideally also through a generic.
- None of this impacts the ability to do custom DSTs.
Does anyone have a specific good reason this shouldn't panic, and should instead abort?
Josh Triplett at 2018-03-30 12:24:53
ideally also through a generic.

Would this result in a monomorphization time error?
gnzlbg at 2018-03-30 12:29:34
@gnzlbg It sounded like folks were generally in favor of a monomorphization-time lint.
Taylor Cramer at 2018-03-30 12:30:33
Would appreciate if someone could also elaborate on the reasoning behind these conclusions. :)
Gábor Lehel at 2018-03-30 12:51:23
Rough summary: extern type is a special-purpose feature that exists for FFI, so adding a pile of trait machinery to statically detect and reject calls to size_of_val on one didn't seem worth it. We had a very strong consensus against returning a sentinel value, which left us with "either panic or abort". There was some discussion about whether we had any motivation to abort, but we couldn't think of any specific cases where panic would lead to breakage. Finally, we still do like the idea of statically detecting these issues, but we can do that with a lint for at least the most common cases.
Josh Triplett at 2018-03-30 13:05:53
@glaebhoerl Hey =) It's kind of hard to write up a detailed comment just now, but I want to say a few things. First off, like any weighty decision, I would describe this as a "preliminary conclusion", subject as always to revision if persuasive counterarguments arise. =)

As for reasoning, there are some minutes from discussion here but they're pretty brief. Here is my attempt at a summary of the key points as I remember them:
- The "desugaring" of T: ?Sized works is already fairly surprising to users as is.
  - Extending to a three-layer hierarchy makes it quite tongue twisting even for advanced users:
    - You say T to mean T: Sized
    - You say T: ?Sized to mean T: DynSized
    - You say T: ?DynSized to mean T
- We would like to extend to custom DSTs in the future; this is often cited as being connected to DynSized, but that doesn't seem entirely complete. We can still have a DynSized trait (or family of traits), but they don't have to be supplied by default:
  - If you write T, you get Sized so you're all set
  - If you write T: ?Sized, you get nothing, but have to add other bounds just like ordinary bounds
    - it does mean that size_of_val and align_of_val are always invokable for any type
      - but these are the most general case anyway (when you have the full value + its metadata); we're covering the hard case now.
- When we have custom DST, that implies that size_of_val and friends will run user code anyway. That code could panic.
  - Given that, we will have the possibility of size_of_val panicking.
- There was some mild concern that size_of_val might execute in unsafe code that is not panic safe, creating a footgun.
  - We could make it hard abort instead -- also if user code panics.
  - But we wanted more persuasive arguments, e.g. examples of code in the wild that would have a problem (brief inspection of code in the standard library didn't turn up such problems, but I didn't really look especially hard).
Niko Matsakis at 2018-03-30 13:14:52
Thanks!

(To be clear I'm skeptical about the value of ?DynSized as well, at least on its own, when its only utility would be to prevent misuse of size_of_val.)

I was mainly curious about the reasoning around the choice of "panic" versus "return 0". I don't think of 0 as being a sentinel value in this case, if "sentinel value" is understood as "something the caller has to check for specifically and handle specially". I agree that panicking is preferable to this.

I think of 0 as a "safest possible default value" -- that is, if someone asks for the size_of_value of an extern type, gets 0, and proceeds to read and write 0 bytes to and from memory, the effect will be that of a no-op, which is unlikely to actually cause any problems. The question is what (if any) scenarios are there where it would. (I might have asked this same question on the extern type RFC thread and someone might have even tried to answer it...)
Gábor Lehel at 2018-03-30 13:48:46

Note that making align_of_val panic also means that field access will potentially panic:

extern { type Opaque; }
struct TerribleOpaque {
    a: u8,
    b: Opaque,
}

let a: &TerribleOpaque = unsafe { ... };
let b: &Opaque = &a.b;   // <-- this line will panic.

struct GenericThingy<T: ?Sized> {
    c: u8,
    d: T,
}
let c: &GenericThingy<Opaque> = ...;
let d: &Opaque = &c.d;   // <-- this line will also panic.

kennytm at 2018-03-30 13:52:02

Returning a dummy value rather than asserting/panicking seems really unlike Rust, and not something we typically do. We don't do things like returning -1; we use Option, or we panic or assert.
Josh Triplett at 2018-03-30 14:00:23
@nikomatsakis could we keep this unstable until we have a custom DST experiment then? Or we could stabilize this with unstable DynSized which would just prevent generics over extern types in practice which is probably fine, while allowing it to be removed later based on DST experiment

but these are the most general case anyway (when you have the full value + its metadata); we're covering the hard case now.

You mean custom DSTs in practice would all implement DynSized?

Given that, we will have the possibility of size_of_val panicking.

Sure any code may panic, but using a panic to enforce a static invariant still leaves a bitter taste in month. If a library makes a "false instance" that is considered bad form. This only is different because of concerns about opt-in traits which may tip the scales in aggregate but doesn't address this problem.

I want a solution that doesn't feel born out of tragic trade-offs.
John Ericson at 2018-03-30 14:02:46
@Ericson2314 The expectation from the discussion was that a lint ought to be able to catch the vast majority of such cases.
Josh Triplett at 2018-03-30 14:14:36
I think of 0 as a "safest possible default value" -- that is, if someone asks for the size_of_value of an extern type, gets 0, and proceeds to read and write 0 bytes to and from memory, the effect will be that of a no-op, which is unlikely to actually cause any problems.

Let's say I'm trying to serialize a value (in a generic function, as it goes) somewhere and use size_of_val for that. Now, when I deserialize it, I have a problem.
Catherine at 2018-03-30 14:15:45
@joshtriplett I agree, and the point of my comment was to explain why I think this is unlike that.

@whitequark Thanks. To have a problem, you'd need the deserialization code to somehow derive a different value? How could/would that end up happening? (I guess if the deserialization happens in C? But then why is the Rust code using generics to hand-roll its own serialization instead of calling C?)

(I just want to be duly diligent and identify, as an existence proof (or 'smoking gun'), at least one plausible, concrete real-world scenario where this causes a major problem before we judge that it's 'obviously' a bad idea.)
Gábor Lehel at 2018-03-30 14:48:29

@glaebhoerl

fn serialize<T>(storage: &mut [u8], val: &T) {
  let size = mem::size_of_val(val);
  storage[..size].copy_from_slice(slice::from_raw(val as *const T as *const u8, size));
}

fn deserialize<T>(storage: &[u8], val: &mut T) {
  let size = mem::size_of_val(val);
  slice::from_raw_mut(val as *mut T as *mut u8, size).copy_from_slice(&storage[..size]);
}

extern {
  type Foo;
  fn alloc_foo() -> *mut Foo;
}

// somewhere:
let original_foo: &Foo = ...;
let new_foo: &mut Foo = unsafe { alloc_foo() as &mut Foo };
let buf: &mut [u8] = ...;
serialize(buf, original_foo);
deserialize(buf, new_foo);
// now new_foo contains uninitialized data.

Catherine at 2018-03-30 15:29:40

Ah I see. In that case the part which 'does know' the size is alloc_foo, which sounds realistic enough. I'm convinced, thanks again!
Gábor Lehel at 2018-03-30 16:05:02
Unrelatedly, I want to re-raise the question of whether we want to deprecate size_of_val and replace it with something which returns an Option. That would take some of the edge off of size_of_val panicking which nobody likes.
Gábor Lehel at 2018-03-30 16:06:32
On March 30, 2018 6:07:02 PM GMT+02:00, "Gábor Lehel" notifications@github.com wrote:

Unrelatedly, I want to re-raise the question of whether we want to deprecate size_of_val and replace it with something which returns an Option. That would take some of the edge off of size_of_val panicking which nobody likes.

That would make size_of_val significantly less useful, and in practice would lead to a lot of unwrapping (which just panics anyway).

It only fails when called on something it should never get called on, and we can detect that case with a lint.
Josh Triplett at 2018-03-30 16:25:12
extern type is a special-purpose feature that exists for FFI

could we keep this unstable until we have a custom DST experiment then

The libs team hopes to stabilize relatively soon (a subset of) allocator APIs after changing them to use *mut void to represent pointers to allocated memory, with void (name to be bikeshedded) an extern type. The type being !Sized is valuable to prevent the use of <*mut _>::offset without first casting to another pointer type, but the pointers must be thin.

So while FFI was a primary motivation it’s not the only case when extern types might show up, and it would be nice to be able to stabilize them without waiting for a full design for custom DSTs.
Simon Sapin at 2018-03-30 20:13:14
I think of 0 as a "safest possible default value" -- that is, if someone asks for the size_of_value of an extern type, gets 0, and proceeds to read and write 0 bytes to and from memory, the effect will be that of a no-op, which is unlikely to actually cause any problems.

If you are ever asking for the size of an extern type, something has gone awfully wrong somewhere. This size is by definition not knoweable. 0 is most certainly not a safe choice if e.g. the offset is used to get an address that definitely lives "after" the extern data in memory; the code would instead overwrite that data which is rather not a safe choice.

~~@whitequark how would reporting the (incorrect!) size 0 be helpful with deserialization? If you attempt to deserialize a type of which you do not know the size, and that deserialization somehow needs the size, then you are kind of in a hard place and something went wrong somewhere. "Just go on and pretend nothing happened" is not how Rust solves these kinds of problems.~~

@kennytm How does Rust even compute the layout of a struct like
```
struct TerribleOpaque {
    a: u8,
    b: Opaque,
}
```
given that rustc does not know the alignment of Opaque either? I expect such type definitions to be illegal. And vice versa, if rustc somehow does come up with a layout and a choice for the offset of b, then it can just use that value when doing &c.b at run-time. Field access will never panic; it compiles (AFAIK) to a constant-offset operation because the offset of the field is computed at compile-time, never at run-time.
Ralf Jung at 2018-03-30 22:43:34
@RalfJung

Field access will never panic; it compiles (AFAIK) to a constant-offset operation because the offset of the field is computed at compile-time, never at run-time.

No the offset &c.b can be computed at run-time when the field is a DST. Check this:
```
let y16: &GenericThingy<dyn Debug> = &GenericThingy { c: 10u8, d: 20u16 };
let y32: &GenericThingy<dyn Debug> = &GenericThingy { c: 30u8, d: 40u32 };
assert_eq!(
    (&y16.d as *const _ as *const u8 as usize) - (y16 as *const _ as *const u8 as usize),
    2 
);
assert_eq!(
    (&y32.d as *const _ as *const u8 as usize) - (y32 as *const _ as *const u8 as usize),
    4 
);
```
Although y16 and y32 have the same type, the offset of &self.d is different.

Currently, the offset of this DST field d: T of a DST struct GenericThingy<T> is computed by the compact-size-of the sized prefix, rounded up to the alignment of the DST field type T. Therefore, to compute the offset of the field, we must require the type T to have an alignment derivable from its metadata only. In the Custom DST proposal this means T: AlignFromMeta.

So yes TerribleOpaque is illegal. However,
1. The definition GenericThingy is clearly legal,
2. Since there is no DynSized or AlignFromMeta, there is nothing blocking us from instantiating GenericThingy<Opaque>
3. Unless you introduce post-monomorphization error, &c.d should have the same compile-time behavior whether it is GenericThingy<u8>, GenericThingy<[u8]> or GenericThingy<Opaque>.
This means either &c.d must panic at runtime, or choose a fallback alignment such as 1 or align_of::<usize>().
kennytm at 2018-03-30 23:21:10
@Ericson2314

could we keep this unstable until we have a custom DST experiment then?

I'm not inclined to wait. As @SimonSapin said, the FFI need is real now. Also, I'd like to drill into what specific choices around Custom DST are being forced here. It seems that what we are deciding is actually relatively narrow:

Will we try to narrow the range of types on which you can invoke size_of_val?

We are leaning towards "no" on that particular question, but that does not necessarily imply that all custom DST types must implement DynSized. We might, for example, say that size_of_val and friends use specialization to check for what sort of trait the reference implements (if any) and panic if there is no such trait implemented -- presuming that always applicable impls work out like I think they will, that would be eminently doable (and @aturon had an exciting idea for building on that work, too, that helps here).

Even if we did say that everything must implement DynSized, then it seems like we are distinguishing a class of types (including at least extern type) for which said implementations unconditionally panic. We are saying that it is not worth distinguishing that classic soundly in the trait system, but we could use lints to capture that class when generics are not involved. (And go further with monomorphization-time lints, if desired.)
Niko Matsakis at 2018-03-31 04:43:38
@kennytm

Note that making align_of_val panic also means that field access will potentially panic:

Yeah, good point! This would also be a consequence of custom DST. I think strengthens the case for "hard abort" and not panic -- it seems like predicting which field accesses could panic would be quite subtle, and a potential optimization hazard. (It may also argue for a monomorphization-time lint.)

If we did opt for "hard abort" instead of panic, I would say that the rule is:

Custom DST code is not permitted to panic (much like panicking across an FFI boundary). We will dynamically capture such panics and convert them into a hard abort.

Niko Matsakis at 2018-03-31 04:48:32
UPDATE: Ignore this, it doesn't work because of back-compat; you can do coercions in a generic context, obviously.

Hmm I wonder if we can modify the definition of CoerceUnsized to prevent "unsizing" a final field into an extern type altogether? That would, I believe, avoid the concern about field access (and moves more towards the specialized-based interpretation of size_of_val I proposed here). ~~

That is, we might have a trait DynSized, which is not implemented for extern type, and we say that you cannot use coerce unsized unless the target type implements it. But it is not required to invoke size_of_val.
Niko Matsakis at 2018-03-31 04:50:58
@glaebhoerl

Unrelatedly, I want to re-raise the question of whether we want to deprecate size_of_val and replace it with something which returns an Option. That would take some of the edge off of size_of_val panicking which nobody likes.

I see this as a separate question, but I am sympathetic to your desire. That said, I agree also with @joshtriplett that in many cases one will just unwrap the result -- at minimum, we ought to add some sort of function to readily test if a type has a defined size/alignment, so you can code defensively (even if we are not going to say you must).
Niko Matsakis at 2018-03-31 04:53:38
This means either &c.d must panic at runtime, or choose a fallback alignment such as 1 or align_of::<usize>().

Thanks, I clearly had not thought this through enough.

However, returning any arbitrary (and hence wrong!) fixed alignment seems catastrophic for the case you described -- it would let us compute the wrong address, as C code that knows the actual extern type could end up with a different layout than we do. So, doesn't your example show that we have to panic or abort?

I think strengthens the case for "hard abort" and not panic -- it seems like predicting which field accesses could panic would be quite subtle, and a potential optimization hazard. (It may also argue for a monomorphization-time lint.)

Agreed -- not just because of optimizations, but also because unsafe code has to be very aware where panics could be raised, for exception safety purposes.
Ralf Jung at 2018-03-31 06:25:14
Would these interact in any way with NVPTX extern __shared__ array types ? There are two flavors of __shared__ array types, static (non extern) and dynamic (extern).
```
fn kernel() {
   let mut a: #[shared] [f32; 16];
   // ^^ This array is shared by all threads in a thread-group
   // It's size is fixed at compile-time and it is the same for
   // all kernel invocations.

   let mut b: #[shared] [u8];
   // ^^ This array is shared by all threads in a thread-group.
   // It has a dynamic size that is constant during the invocation
   // of this kernel. Each kernel launch must set its size, but each time
   // this kernel is launched this array can have a different length. This
   // basically produces a pointer. The user is responsible for tracking
   // the size of these arrays, e.g., by passing it as an argument to the
   // kernels.
}
```
So it looks to me that this wouldn't interact with static __shared__ arrays because size_of_val would just return mem::size_of. However, the size of extern __shared__ arrays is not known, not at compile-time, and at least for nvptx not at run-time either: the user is in charge of passing the array size around as a kernel argument and it is a "common" idiom to pass a single integer from which multiple sizes are computed inside the kernel. So I assume size_of_val would need to result in an error for these.
gnzlbg at 2018-03-31 09:22:27
Field access being potentially aborting feels very sad.

On the other hand, the only way this can happen is when the struct type is uninhabitable, in which case the field access was dubious anyway. So this basically raises the old question of when is calling size_of_val or doing raw pointer lvalue access is valid.

I would prefer that to be documented somewhere - obviously, we want to access fields of e.g. uninitialized structs, as in e.g. RcFromSlice.

Ariel Ben-Yehuda at 2018-03-31 21:01:42
The recent DynSized RFC proposed
- T to mean T: Sized
- T: ?Sized to mean T with no trait bounds
- T: ?Sized + DynSized to mean T: DynSized
Where extern types would be !DynSized, but then only adding + DynSized to size_of_val in the new epoch, leaving it as a lint+panic for now. Since this is an extension of what's being proposed here and could be added later, is the idea still on the table?
Andrew Cann at 2018-04-02 09:53:01
Since we want to compile together crates that use different epochs/editions, opting into a new edition can only affect "superficial" crate-local aspects of the language like syntax, not public APIs.
Simon Sapin at 2018-04-02 10:06:20
@nikomatsakis

My basic concern is I feel a number of various issues are pushing us in the direction of more fundamental / opt-in traits, but the resistance to opt-in traits is such that we're throwing around ~~ad-hoc lints~~ ad-hoc solutions like lints instead. I get that ? is annoying to teach, but I'll take principled weirdness over banal but endless machinations. The grapple scares me more than the fall down this slippery slope.

I'll admit {size,align}_of_valisn't that interesting on it's own. But to show my cards, I was excited about contributing in part to this RFC because I finally had some issue by which to force the topic of DynSized in particular, and more special traits in general. I guess you all didn't take the bait :). Now, I suppose I'll ask whether, if we had a full menagerie already, would we still bother making {size,align}_of_val defined for !DynSized types. Relatedly, if we end up adding DynSized later, would be deprecate the {size,align}_of_val we have today? I realize "no" for the first and "yes" for the second aren't ironclad reasons to make DynSized now, but I'm still curious about the answer.

Even if we did say that everything must implement DynSized, then it seems like we are distinguishing

Mmm if all types must implement DynSized, then we're not distinguishing anything. What we are doing is providing a principled way of using an existing feature (the trait system) to allow users to right the requisite "hook". That alone is reason for a DynSized trait in my mind.

...a class of types (including at least extern type) for which said implementations unconditionally panic.

Surely you don't mean the salient attribute of !DynSized types is that querying the size panics? It's that they have no dynamically or statically known size. Panicking is just an enforcement mechanism with no intrinsic meaning.
John Ericson at 2018-04-02 22:42:19
opting into a new edition can only affect "superficial" crate-local aspects of the language like syntax, not public APIs.

We could deprecate and replace size_of_val then. Call it dynsize_of_val.
Andrew Cann at 2018-04-03 01:21:15
@canndrew (I am responding to two comments at once)
The recent DynSized RFC proposed
- T to mean T: Sized
- T: ?Sized to mean T with no trait bounds
- T: ?Sized + DynSized to mean T: DynSized The recent DynSized RFC proposed ... We could deprecate and replace size_of_val then. Call it dynsize_of_val.
I believe that this future could still be on the table. This is what I was trying to say in this comment when I wrote:

It seems that what we are deciding is actually relatively narrow:

Will we try to narrow the range of types on which you can invoke size_of_val?

I feel very strongly that we do not want T: ?Sized to actually mean T: DynSized. However, I could imagine that we introduce DynSized as an "ordinary" trait and introduce dynsize_of_val (or whatever) that requires it -- and then specify that size_of_val is implemented by using specialization to invoke dynsize_of_val when possible and aborting/packing otherwise (I lean more and more towards abort, personally).

Alternatively, thinking more about lints -- it is certainly plausible to lint on calls to size_of_val unless T: DynSized (one could even imagine generalizing this). That is important because we also do have to figure out the field access question. We can't deprecate field accesses -- and they are legal today knowing only that T: ?Sized (i.e., we do not require T: DynSized). But we could lint aggressively there, thus encouraging T: DynSized to proliferate.

Worth thinking over. But also not blocking further progress on extern type, I think.
Niko Matsakis at 2018-04-05 20:41:30
@Ericson2314

My basic concern is I feel a number of various issues are pushing us in the direction of more fundamental / opt-in traits, but the resistance to opt-in traits is such that were throwing around ad-hoc lints instead.

Can you be more explicit? It seems like this is one precise case where we are talking about lints, specifically because it is narrow and we don't see another way out of the backwards compatibility box, but in other cases where we had thought about adding "implicit" traits (notably, ?Move), I don't believe lints are on the table. Instead, we've found a way to add the desired functionality in a "non-infectious" fashion (using Pin). Are there other cases I'm overlooking?

That said, I do think there is a constant tension, one that Rust always has to walk: how to get the maximum bang for our static analysis buck, and I feel no shame about keeping lints as part of the toolbox.
Niko Matsakis at 2018-04-05 20:46:46
@nikomatsakis

Can you be more explicit?

Sorry, I meant "ad-hoc solutions like lints". edited the above accordingly.

I hadn't yet seen Pin. Glad there is a safe and total way out of that corner, but it too strikes me as a bit of a monkey patch; see the final comment https://github.com/rust-lang/rfcs/pull/2349#issuecomment-378990371 which makes one wonder whether all collections will need a Pin variant leading to an ecosystem split!

I realize there's a steep drop off in priority along [generators, extern types, custom DSTs, out pointers and other linear types]. But the fact that implicit traits keep coming up gives me pause to let them go: I now see them all as one problem and thus our current trajectory as many unrelated piecemeal solutions. Also, the observation (not mine, maybe in https://github.com/rust-lang/rfcs/issues/2255 ?) that more ?-traits probably makes them less confusing I find compelling.

Also, ?-traits work like Cargo features in that their the only general way to backwards-compatibly grow the language in a negative direction: reducing requirements rather than adding functionality, and I find that the more interesting direction for language evolution.

This all boils down to a difference in but opinion that's been around for years, and probably cannot be bridged. Your previous comment on positive DynSized gives me hope in this specific case. If we have far more DynSize than ?DynSize annotations in the end, I wonder what is achieved, but at least we can meaningfully speak about sizing.
John Ericson at 2018-04-05 22:32:45
OK, I just want to say that I am very strongly of the opinion that Rust should use built-in traits like DynSized to express the difference in capabilities between extern types and the dynamically-sized types that currently exist in Rust (i.e. trait objects and slices). All of the alternatives that I have seen – panicking, returning Option<usize> from size_of_val, post-monomorphisation lints – are less powerful, and the issues with ?-traits that people keep bringing up need to be tested and not just speculated about. We need to at least try doing things the builtin-traits way and see what it's like, and see what the ergonomic impact is like, and see if we can reduce it, before settling for something inferior.

Maybe I'm overreacting, I just got the sense from reading some of the comments in this thread that something might be done in order to get extern types out the door, that might put us in a backward-compatibility trap later on. Now that I have more time to work on Rust, I'm planning on writing an eRFC to add DST-specific builtin traits like DynSized and SizeFromMeta, so we can start experimenting with them and Custom DST.
Michael Hewson at 2018-04-06 04:57:24
@mikeyhew These alternatives are definitely less powerful, but as the maxim goes: always use the smallest tool for the job.

There are a host of global factors to consider when extending the language, especially when it comes to introducing a new fundamental distinction. The payoff in this case seems incredibly tiny. And we do have plenty of first-hand experience with ? traits in the form of Sized.

I wonder if you could spell out, in terms of practical impact, why you feel so strongly about built-in traits?
Aaron Turon at 2018-04-06 15:44:59
I'll start a list.
1. Opt-out traits are less impactful for those that don't care. Don't care about weird FFI types? Never write ?DynSized. If somebody else wants to use your crate for those, they can send the DynSize PR. C.F. with opt-in DynSize and deprecated {size,align}_of_val, now everyone needs to care if the new replacement methods are to get traction. This is the exact opposite of what @withoutboats said.
2. DynSize is a minor now, but seems like an important part of any custom DST proposal. Custom DSTs are very useful for things we care about.
3. Opt-out traits are like Cargo default features. They allow a completely different way of changing the library/language by reducing dependencies/assumptions instead of adding features. They are the only way to backwards compatibility do that we have, in fact.
  
  I am personally interested in this sort of thing. It's very similar to portability, for example. We want rust crates that don't or barely need a normal OS to also support weird platforms without annoying the crate author. It is an open "ecosystem sociology" question whether this can be pulled off. Similarly a bunch of us want truly unized types, custom DSTs, linear types, out pointers, and other weird things without pissing off regular uses. Opt-out traits, again, seem the best and only way to do that.
4. I strongly agree with whoever wrote that having more opt-out traits is good for pedagogy---it was a great point that I hadn't previously thought of at all. Right now Sized, being the one weird trait, isn't really part of a general pattern. DSTs, Sized, and opt-out traits are probably all one mess in most peoples head. Having more opt-out traits teases the concepts apart: who knows which opt-out trait you'll grok first, and now that can help you learn the others.
  
  I think it's illustrative that you wrote "built-in traits" above @aturon. We have many different types of magic traits today, from the most normal Copy (requires impl), to Send/Sync (implicit impl but not default bounds), and Sized (implicit impl and default bound). Making sure ever weird class has multiple examples and a dedicated names (better than old "OIBIT"! https://internals.rust-lang.org/t/pre-rfc-renaming-oibits-and-changing-their-declaration-syntax/3086/15) should clear things up.
John Ericson at 2018-04-06 16:07:29
DynSize is a minor now, but seems like an important part of any custom DST proposal. Custom DSTs are very useful for things we care about.

@Ericson2314 I've yet to see any custom DST proposal that involved any types whose size was completely unknown at runtime. Why is this so critical?
Taylor Cramer at 2018-04-06 16:17:36
@cramertj Custom DSTs are DynSized by definition. It's that implementing a trait is by far the most natural way to add the right hook. I want a repeat of Copy: Clone not Drop. Drop got it wrong because as all types (today) can be dropped, the question is when is the drop automatic and when does it require user code; I'd have preferred a Forget: Destroy.
John Ericson at 2018-04-06 16:19:42
@cramertj Also some custom types might have really expensive ways to calculate the size (C strings, for example). For performance-conscious users, it may be better to not implement DynSized and do all size look-ups by hand. IMO, all implicit operations being O(1) is a defensible if extreme position to take.

C.f. some people arguing similarly about lock guards being linear and needing to explicitly consume unlock in the past (not that i necessarily agree with that lock guard example).
John Ericson at 2018-04-06 16:23:39
1. I don't think "always use the smallest tool for the job" applies here. The decreased power of lints is directly worse for uses. C.f. non-null lints v.s. Option in other languages. Lints are easily lost amid other warnings, and the fact is only some users will care. This means while individual code bases might obey them, the ecosystem as a whole can not be trusted to uphold the invariants the lints try to maintain. This a real loss for fostering an ecosystem of DynSize abstractions, or whatever the niche feature is, as for such niche things, being able to sync up few and scattered programmers and form a community is all the more important.
2. Ecosystem-wide enforcement is also good for the "regular users don't need to care" goal. If some library happens to use truly unsized types, and the consumer is unaware, they could face some nasty unexpected packages. With ?DynSize they do get bothered with a compile time error they didn't expect, but that is much less nasty to deal with with than a run-time bug. If they don't want to learn ?DynSized, they can go use a different library; better to have the opportunity to do that up front than after you're tied to the library too deeply because it took a while to excise the {size,align}_of_val panic.
John Ericson at 2018-04-06 16:25:21
As useful as extern types would be for me, having extern types without all the proper language machinery to enforce their unsizedness would result in a partial solution that is marginally better than the current partial solution of using zero variant enums. Extern types don't even solve the real problems for me such as the inability to properly specify that a struct is opaque beyond the first few fields or that a struct ends in dynamically sized or unsized data yet has a thin pointer. I want a full comprehensive plan for how to get to a full solution to those problems. What I don't want is any partial solution being stabilized early without being part of the full comprehensive plan, because that just leads to Rust being locked into something sub par.
Peter Atashian at 2018-04-06 17:19:44
@aturon

I wonder if you could spell out, in terms of practical impact, why you feel so strongly about built-in traits?

I want to create safe data structures for DSTs, like the DSTVec data structure that I posted about on Reddit a while ago (probably over a year ago), which stores DSTs contiguously in memory to avoid boxing. It requires AlignFromMeta and either SizeFromMeta or SizeFromRef, and I'd like to be able to write those requirements as a trait bound.

A few months ago, I came up with an idea that avoids the ? altogether. I'm referring to the idea that if a Sized-family trait appears in the list of trait bounds, the default Sized bound is removed. I'd like to explore that by implementing it in tree and seeing what it's like to use it.

Like @Ericson2314, I don't think we want to pick the "smallest tool for the job" here, if "smallest" means the least powerful, least general, or least extensible. The Rust team has been pretty good about having a rigorous design process, and never just adding a language feature when the tools to implement it can be added instead, and the original feature possibly added as a syntactic sugar for something more expressive. In this case, the tools we are talking about are the Sized-family traits, and an extern type is really just a type that doesn't implement them.
Michael Hewson at 2018-04-06 20:34:24
@mikeyhew Overall I very much agree with all that. One thing is though:

I'm referring to the idea that if a Sized-family trait appears in the list of trait bounds, the default Sized bound is removed. I'd like to explore that by implementing it in tree and seeing what it's like to use it.

This would require us to design the entire hierarchy at once. Because otherwise, if we add another then now that one can't be disabled by the other older ones for backwards compatible. Better to have just one notion of ?-traits than a courser staircase thing I think.

(BTW the one notion can be thought of as just one type of ?-trait and a "flat" default bound Size + DynSized + ... such that any trait opted out also removes any other part of the default bound implying it.)
John Ericson at 2018-04-06 22:01:53
Note: @joshtriplett opened up https://github.com/rust-lang/rust/issues/49708 to discuss the specific question of "what should the behavior of size_of_val (and align_of_val) be when applied to extern types" -- that is, the specific question at hand (which obviously intersects larger questions around DynSized).

(I didn't see him announce that here.)
Niko Matsakis at 2018-04-10 15:23:33
So, the name currently decided on by @SimonSapin for the allocated memory type seems to be Opaque. I concur that Void is a dubious name, but I don't think Opaque is great, mainly due to its complete lack of descriptiveness. I propose one of the following:
- Blob
- Mem
- MemBlob
- MemChunk
- OpaqueMem – not my favourite, but at least slightly more explicit than just Opaque
I'd probably lean towards Mem, since it's the most pithy, but the others are okay too.
Alexander Regueiro at 2018-04-11 23:52:30
@alexreg You probably meant this for #49668 (tracking the GlobalAlloc trait) rather than this issue (tracking extern types in the language), but I personally don’t care much about the Opaque name: I only picked one so we could make progress. In my mind these names are pretty much all synonymous to Thing.
Simon Sapin at 2018-04-12 05:33:23
@SimonSapin Yes, I did. Not sure how I got here when I thought I clicked Alex Crichton's link, oops. Want me to repost my comment there, or leave it here (this issue is rather related anyway)?

Anyway, glad to hear you're not too bothered. I understand you just wanted to get that PR merged with a name that wasn't Void, so fair enough. Perhaps we can wait for a few people to voice there preferences, and you can pick one from the above, if that's okay with you? :-)
Alexander Regueiro at 2018-04-12 15:33:00
At this point I’d even prefer if someone else did the ~cat herding~ consensus gathering and picked one name (:
Simon Sapin at 2018-04-12 16:59:14
@SimonSapin @alexreg Can you please have the discussion about GlobalAlloc naming in that issue?
jethrogb at 2018-04-12 17:01:44
I just realized extern type can be used to easily & safely emulate single-inheritance hierarchies:
```
extern {
    pub type Base;
}

#[repr(transparent)]
pub struct Derived(Base);

impl Deref for Derived {
    type Target = Base;
    fn deref(&self) -> &Base { &self.0 }
}

impl Derived {
    pub unsafe fn unchecked_downcast(base: &Base) -> &Self {
        &*(base as *const Base as *const Self)
    }
}
```
Of course, using this requires using references in FFI, but that was already part of the plan for the usecase where I noticed this would be an option, i.e.: https://github.com/rust-lang/rust/blob/01dbfdaf4f45b68b49332b8785262a3a780d0a19/src/librustc_llvm/ffi.rs#L463-L479
Eduard-Mihai Burtescu at 2018-06-22 16:53:07
cc @rust-lang/lang Nominating for discussion this week, as to what's needed for potential stabilization. If someone has time to summarize the discussion so far ahead of the meeting, that'd be most helpful!
Aaron Turon at 2018-07-18 04:38:35
Another technique using this feature:
```
extern { type Opaque; }
struct InvariantOpaque<'a> {
    _marker: PhantomData<&'a mut &'a ()>,
    _opaque: Opaque,
}
pub struct Foo<'a>(InvariantOpaque<'a>);
```
This is sort of a hacky way of adding a lifetime parameter to an extern type (which we could also natively support, I suppose, but as you can see, it can be emulated; type parameters would also work).

What can we do with this?
- an "owned" pointer, akin to C++ unique_ptr<Foo<'a>>, is &'a mut Foo<'a>
  - the constructor can mark with 'a the lifetimes that foreign code would capture
  - if a constructor returns this type and a destructor takes it, the latter can't ever receive a reborrow, because borrows are necessarily &'b mut Foo<'a> with 'b strictly shorter than 'a, and 'a can't be shortedned because it's in an invariant position
  - therefore, &'a mut Foo<'a> can only be moved (when written with two 'as like that)
  - caveat 1: can't implement Drop on &'a mut Foo<'a>, need a wrapper struct
  - caveat 2: using it as a field in a wrapper that implements Drop means having to use &mut *(self.field as *mut _) because Drop impls can't move out their fields
- most functions can take &Foo (or maybe even &mut Foo, but that's your choice)
- if you need to refer to what the foreign object borrowed, use &Foo<'a>
  - e.g. fn foo_field(foo: &Foo<'a>) -> &'a Bar;
- if you need to refer to the foreign object's own lifetime, use &'a Foo
  - e.g. fn foo_iter(foo: &'a Foo) -> &'a mut FooIter<'a>;
Eduard-Mihai Burtescu at 2018-07-18 04:50:14
Extern types don't even solve the real problems for me such as the inability to properly specify that a struct is opaque beyond the first few fields or that a struct ends in dynamically sized or unsized data yet has a thin pointer. I want a full comprehensive plan for how to get to a full solution to those problems. What I don't want is any partial solution being stabilized early without being part of the full comprehensive plan, because that just leads to Rust being locked into something sub par.

@retep998 Have you tried creating a struct with an extern type as its last field? @irinagpopa has used the technique from my previous comment in #52461 with great success.

EDIT: an earlier example of success use of prefixes, this one even has data: https://github.com/rust-lang/rust/blob/f686885a14fff16ddf984b08fb0d9ded07e66f1c/src/librustc/ty/mod.rs#L594-L609
Eduard-Mihai Burtescu at 2018-07-18 04:55:38
What's the status of the compile-time detection prohibiting size_of_val on an extern type?
Josh Triplett at 2018-07-18 06:48:50
I was going to fix https://github.com/rust-lang-nursery/nomicon/issues/29 within the next few days by changing that section to recommend a ZST instead. Should I wait because this might get stabilized soon-ish? :D
Ralf Jung at 2018-07-18 07:03:40
@joshtriplett AFAIK no lint or panicking size_of_val solution has been implemented. If @aturon is fine with it (iff we want this as part of the edition), I could try implementing those.
Eduard-Mihai Burtescu at 2018-07-18 07:16:21
@nikomatsakis @oli-obk Do you have opinions on whether to have a check within the size_of_val safe wrapper for the intrinsic, or in the intrinsic itself? I think we can emit panics anywhere calls are involved, since we have the unwind edges and whatnot fully set up (TerminatorKind::Assert already works a bit like this), so we should be able to, with minimal work, handle the size_of_val intrinsic panicking inside codegen. But miri also needs to handle it, so maybe we should add a miri error kind and use that like Assert?
Eduard-Mihai Burtescu at 2018-07-18 07:22:07
I would have thought we'd just replace calls to the intrinsic with an assert false if the type is an extern type. So "inside the wrapper"?
Oli Scherer at 2018-07-18 08:40:25
We currently don't support repr(align(N)) on extern types (it's not prohibited but it's ignored -- maybe we should fix that), but it could possibly be useful. The alignment is a property of pointers, so it can be known even if the pointed-to type is opaque. For example, GCC lets you do struct __attribute__ ((aligned (8))) S;.

But if we support that, then it would make sense for ~align_of and~ align_of_val to return the declared alignment rather than panic (unlike dyn Trait, an extern type is not encompassing multiple different concrete types with different alignment requirements). I think it's backwards compatible to go from panics to this interpretation later, and it doesn't change the story for size_of_val, but @eddyb asked me to write this up for the record.
Hanna Kruppe at 2018-07-18 09:33:03
it's not prohibited but it's ignored -- maybe we should fix that

Yes, I’d even say this should be a blocker for stabilization.
Simon Sapin at 2018-07-18 09:46:47
Just to remind that &self.extern_type_field invokes the align_of_val intrinsic indirectly to calculate the field offset, so the assert(false) shouldn't be placed in the safe wrapper, but the intrinsic itself.
kennytm at 2018-07-18 10:08:51
@kennytm the field access was the reason I brought up that we shouldn't panic. The intrinsic and the wrapper are function calls, which means there's a target for unwinding. But we don't have the same kind of information for field accesses, so they would create serious problems by panicking. This is why I prefer returning a defined alignment.
Eduard-Mihai Burtescu at 2018-07-18 11:20:40

@eddyb unless the extern_type supports #[repr(align(n))] I don't think returning any fixed number from intrinsics::align_of_val() is safe 😄.

Without #[repr(align(n))], instead of panic an abort() is probably more appropriate for field access.

Together, the implementation could be:

fn size_and_align_of_dst(...) -> (ValueRef, ValueRef) {
    ...
    match t.sty {
        ty::TyForeign(_) => (C_usize(cx, 0), bx.trap()),
        // make sure intrinsics::align_of_val() will abort the program.
        // (size_of_val() can be anything, no one should call it anyway.)
        ...
    }
}

pub fn size_of_val<T: ?Sized>(val: &T) -> usize {
    unsafe {
        if intrinsics::is_extern_type::<T>() {   // if you need a panic
            panic!("extern type does not have a known size");
        }
        intrinsics::size_of_val(val) 
    }
}

kennytm at 2018-07-18 13:59:47

Should #[repr(align(n))] be required for extern types?
Simon Sapin at 2018-07-18 14:45:18
@kennytm It's safe in the sense that it can't break safe code! The only reason we need the alignment for DST fields is that you can, completely in safe code, coerce any &Foo<Bar> to &Foo<dyn Trait> (if defined struct Foo<T: ?Sized>), and then access that field, to call methods of Trait on it (for example).

But anything similar for extern type needs unsafe code and that would have to include some way to get the right alignment, whether it panics or not.

EDIT: there would be one counter-argument that custom DSTs might want dynamic alignment computed from a method that could panic, so that's something to keep in mind.
Eduard-Mihai Burtescu at 2018-07-18 14:51:33
@SimonSapin No.

If extern type is used like what it originally designed i.e. as FFI of opaque C structures, the alignment is indeed completely unknown, and there's no meaningful #[repr(align(n))] for it.
kennytm at 2018-07-18 15:00:27
Rust code can't move or construct an extern type (outside of unsafe code, which has to handle any size/alignment requirements itself), and can't ever read the inside of an extern type, so as far as Rust code is concerned, the alignment shouldn't matter. Just as with a C opaque struct, if you don't know the insides of the type then you can't declare anything that contains that type, ever. You can declare something that contains a pointer or reference to that type, but never the type itself.

The following code should not be allowed:
```
extern type Foo;
struct S {
    // ...
    foo: Foo,
    // ...
}
```
This should produce a compile-time error along the lines of "field foo has opaque extern type Foo", with an explanation that tells the user why they can't do that (because Rust doesn't know the size of Foo).

Rule of thumb: if you can't do it with a C struct Foo; then you shouldn't be able to do it with a Rust extern type Foo;.
Josh Triplett at 2018-07-18 17:41:54
Extern types are currently allowed in the tail position of structs, just as ?Sized types are also allowed. I don't think that produces any bugs.

That is, this is allowed right now:
```
extern { type Foo };

struct Bar {
    x: u32,
    foo: Foo,
}
```
I've used extern types in the past to create thin pointers to DSTs, which is a bad hack but it is what it is. For example, I was working on a skip list which had roughly this structure:
```
extern { type Lanes; }

struct Node<T> {
     data: T,
     lane_count: usize,
     lanes: Lanes
}
```
(Lanes was a [T] in reality, but I wanted to store the size inline).

AFAIK there's no technical reason this should not work, just as it works with normal DSTs, but maybe there's something I haven't understood properly.

EDIT: I misremembered my example originally and I've edited it to be correct.
srrrse at 2018-07-18 17:51:52
@joshtriplett That doesn't work for a generic ?Sized field (we've discussed before in https://github.com/rust-lang/rust/issues/43467#issuecomment-377644468)
kennytm at 2018-07-18 17:53:56
@joshtriplett As I said, the alignment is a property of the pointer and Rust code certainly has to handle pointers to extern types (raw pointers can be unaligned while not dereferenced, but we also have safe references). You can know nothing about the size or contents of a type, and still know that it must be aligned to, let's say, 16 byte. I don't know whether any C APIs actually communicate this information, but it's not an inherently nonsensical concept.

Now, if you're not in charge of creating such a value, you can safely assume a smaller alignment, but you're still losing information. Furthermore, if the extern type is created by unsafe Rust code, then although that code is ultimately responsible for ensuring correct alignment, it would still be more natural to put the alignment on the type in the usual way rather than e.g. storing it as a separate constant like const FOO_ALIGN: usize = 4;.

Rule of thumb: if you can't do it with a C struct Foo; then you shouldn't be able to do it with a Rust extern type Foo;.

Again note that one can attach __attribute__((aligned(N))) to incomplete types in common C compilers. I don't think this actually has any consequences since unaligned pointers are OK to create and pass around in C, but it's morally similar. Although I think in C++ there might be something you can do with references?
Hanna Kruppe at 2018-07-18 18:00:15
@rkruppe Rust code can't move an object of extern type, so it can't ever change a pointer to an extern type; it can only work with pointers it receives. So, it shouldn't ever need to care about alignment. I don't have any objection to being able to add a repr(align(N)) to an extern type if there's some value in doing so, but it certainly shouldn't be required.

@withoutboats An extern type is more (or, in a sense, less) than just ?Sized, hence the whole discussion aout ?DynSized. And one of the major differences is that you shouldn't be able to put an extern type inside a struct, even at the end, whereas you could potentially do so with a ?Sized type.
Josh Triplett at 2018-07-18 18:55:18
To clarify something: I'm not arguing that we couldn't, in the future, construct a broader self-consistent type system that allows you to do more things with extern type. I'm arguing that the current handling of extern type is problematic, and for the purposes of stabilizing enough to support opaque types for FFI usage, we should simply ban fields whose type is an extern type for now.
Josh Triplett at 2018-07-18 19:25:24
And one of the major differences is that you shouldn't be able to put an extern type inside a struct, even at the end, whereas you could potentially do so with a ?Sized type.

Err one of the original motivations was doing precisely this to represent FFI types with "some but not all" known fields.
John Ericson at 2018-07-18 19:57:03
And one of the major differences is that you shouldn't be able to put an extern type inside a struct, even at the end, whereas you could potentially do so with a ?Sized type.

This entirely rules out using extern type for generic parameters:
```
extern { type Lanes; }

struct S<T> {
     x: T,
}

type Foo = S<Lanes>;
```
Ralf Jung at 2018-07-18 20:09:20
@RalfJung Not necessarily; we could catch that when instantiated, while also allowing (for instance) struct S<T> { x: *T } to be instantiated as S<SomeExternType>.

@Ericson2314 That's a much broader use case than C opaque structures, and would require quite a bit more care to handle in a type-safe way. I'd like to avoid completely blocking the availability of opaque structures on that.
Josh Triplett at 2018-07-18 21:18:23
@joshtriplett well, truth be told. as long as one doesn't take a pointer to the final field (which is a useless thing to do anyways), there is no issue with final-field extern type. Broken record here, I think an unstable DynSized trait is the base way to stabilize this feature soundly while keeping open all possible futures (including panicking and no DynSized). It can be used to prevent the reference-taking.
John Ericson at 2018-07-18 21:39:02
@joshtriplett Please refer to https://github.com/rust-lang/rust/issues/43467#issuecomment-399509319, https://github.com/rust-lang/rust/issues/43467#issuecomment-405808005 and https://github.com/rust-lang/rust/issues/43467#issuecomment-405808732 for examples of idioms involving structs containing extern type tails that I would very much like Rust to officially adopt because they are miles ahead in terms of safety compared to existing solutions. If we can model know prefixes, C++ single inheritance and borrows correctly, why not do it?! Also, I expect/hope CStr to become an extern type (or wrapper around one) so we can also replace *const c_char with &CStr or even &'a CStr and remove an entire category of bugs/footguns.

Sadly fixing everything would require some weak form of runtime dependent typing, but we can at least make sure steps towards modelling foreign APIs in terms of safe Rust concepts, even without that.
Eduard-Mihai Burtescu at 2018-07-19 06:24:31
@joshtriplett

Not necessarily; we could catch that when instantiated, while also allowing (for instance) struct S<T> { x: *T } to be instantiated as S<SomeExternType>.

So you are suggesting monomorphization-time errors? That is the only possible implementation for this proposal, I think. I can't even remember if we have any monomorphization-time errors, but we sure should avoid adding more.
Ralf Jung at 2018-07-19 06:49:37
Summary of this thread so far:

Should we allow generic lifetime and type parameters on extern types?

This is mentioned in the initial post as an unresolved question, but no discussion has taken place on this. As mentioned in https://github.com/rust-lang/rust/issues/43467#issuecomment-405808005 you can add genericism in a different way.

std::ptr::null/null_mut

Currently these functions don't support generating null pointers for extern types because they require T: Sized. https://github.com/rust-lang/rust/issues/43467#issuecomment-321678621

extern type cannot support size_of_val and align_of_val.

There has been a lot of talk in this issue and also in https://github.com/rust-lang/rust/issues/49708. #49708 finished FCP, but it's not been resolved. The two main camps are “use traits” and “use panics at runtime supported by lints at compile-time”.

Since I have a very strong opinion on this matter I don't really want to try to summarize that entire discussion.

Should you be able to specify the alignment of an extern type?

There is some precedent for this in C. https://github.com/rust-lang/rust/issues/43467#issuecomment-405871235

Is this feature suitable for implementing c_void?

This is mentioned as one of the motivations in the RFC. Tracking in portability WG: https://github.com/rust-lang-nursery/portability-wg/issues/13

Remaining questions for this issue:
- Can this be done in a non-breaking way? See https://github.com/rust-lang/rfcs/pull/1783#issuecomment-400345303
- What does the LLVM type for c_void need to be? See https://github.com/rust-lang/rust/issues/43467#issuecomment-374011043 and following comments
Are all usecases covered?

@retep998 has some concerns about the feature not being useful enough in https://github.com/rust-lang/rust/issues/43467#issuecomment-379319435 but there are no concrete examples of this that don't work. These concerns might have been addressed in https://github.com/rust-lang/rust/issues/43467#issuecomment-405808732 .
jethrogb at 2018-07-19 18:53:25
@RalfJung We'd either need a monomorphization-time error, or we'd need to completely ban extern types in generics, or we'd need something like the ?DynSized trait, and that latter adds a huge amount of complexity and forces us to work out much more of the planned handling for custom DSTs.
Josh Triplett at 2018-07-19 19:20:34
Should we allow generic lifetime and type parameters on extern types?

As there is no place to spell out the variance of these parameters, my preference is "no".
```
// unknown if `'a` should be covariant or invariant.
extern { type Foo<'a>; }
```
kennytm at 2018-07-19 19:29:59
@kennytm Variance? With respect to what operation?
Alexander Regueiro at 2018-07-19 21:01:38

@alexreg With respect to Foo ...? I'm not sure what do you mean.

extern { 
    type Foo<'a>;
    fn create_a_static_foo() -> *const Foo<'static>;
};
fn create_a_shorter_foo<'a>() -> *const Foo<'a> {
    let foo = unsafe { create_a_static_foo() };
    foo  
//  ^~~ ok if 'a is covariant, error if 'a is invariant.
}

kennytm at 2018-07-19 21:28:28

@kennytm Okay, got it, thanks. The operation would be function type constructor (and consequently the type constructor *const Foo<'a>) in this case. Which is normal Rust is always covariant in the return type, as far as I know... Anyway, I'm not sure how treating extern function type constructors the same would be problematic.
Alexander Regueiro at 2018-07-19 21:45:54

A type itself can be co/contra/invariant in its lifetime parameter:

struct Covariant<'a> {
    r: &'a i32,
}
fn co<'a>(c: Covariant<'static>) -> Covariant<'a> { c }
// ERROR: fn contra<'a>(c: Covariant<'a>) -> Covariant<'static> { c }

struct Contravariant<'a> {
    f: fn(&'a i32),
}
// ERROR: fn co<'a>(c: Contravariant<'static>) -> Contravariant<'a> { c }
fn contra<'a>(c: Contravariant<'a>) -> Contravariant<'static> { c }

struct Invariant<'a> {
    m: &'a mut &'a (),
}
// ERROR: fn co<'a>(c: Invariant<'static>) -> Invariant<'a> { c }
// ERROR: fn contra<'a>(c: Invariant<'a>) -> Invariant<'static> { c }

Which behavior would an extern type with a lifetime parameter have? There's nothing to infer it from.

Russell Johnston at 2018-07-19 23:33:17

@rpjohnst I was effectively referring to the same thing. But it's still not quite invariant/covariant in itself. The operation/functor here is the type constructor, technically.
Alexander Regueiro at 2018-07-20 00:11:44
We usually say that Foo is co/contra/in-variant wrt 'a, where 'a is a parameter of Foo. Only user-defined types have variance from user-defined parameters directly, *const T, for example, is covariant wrt T regardless of what T might contain. Anyway the problem is basically the same as struct Foo<'a>; w/o using PhantomData. Both invariant and covariant are "good defaults" but invariant, the better one, cannot be relaxed at all.
Eduard-Mihai Burtescu at 2018-07-20 04:48:22
@eddyb Yes, but in computer science, technically covariance is with respect to an operation/functor, I must stress that. This is just "shorthand" for that.

I see what you mean with regards to defaults, however.
Alexander Regueiro at 2018-07-20 15:02:57
@alexreg sure, but you weren't right either, since you took *const Foo<'a> to be one type constructor, applied with 'a, whereas it's both, i.e. it's more like some (fictive) ConstPtr<Foo<'a>>. *const T aka ConstPtr<T> has a variance wrt T, Foo has its own (unrelated) variance wrt 'a, and they compose. The variance of their composition is deterministic so we usually talk about the variance of individual type constructors, instead of the variance of their composition.
Eduard-Mihai Burtescu at 2018-07-20 21:47:57
@alexreg I understood that perfectly well. But it is one type constructor – the fact it happens to be the composition of two more primitive ones is less important here. It would have just led to unnecessary verbosity.
Alexander Regueiro at 2018-07-20 23:06:15
Thanks for the summary, @jethrogb!

We discussed this in the lang meeting, and I felt the overwhelming concern was that the "right" answers to a bunch of those questions are intimately tied in to whatever the story for custom DSTs becomes, and that's not something soon, or something people want to wait for.

So the question becomes whether we can find a way to basically punt on all the hard questions for now.

Here's an initial proposal: Prohibit passing extern types to generics altogether (for now)

I think that leaves maximal flexibility for later, and while isn't terribly clean, it does have precedent in transmute. The important point, though, is that it doesn't prohibit passing &MyExternType or *const MyExternType to a generic. So the core FFI use should be unblocked. (Of course it's suboptimal that ptr::null and NonNullPtr wouldn't work, but 0 as *const MyExternType would, as would casting through NonZeroUsize, as workarounds.) Plus, the library author can hide their extern type inside a wrapper of some sort -- which they plausibly would be doing anyway to get Drop and such -- so users of the library wouldn't be exposed to the restriction at all.

Thoughts?
scottmcm at 2018-07-26 19:25:22
Prohibit passing extern types to generics altogether (for now)

How would that work with structs that contain extern types as the last field? I'm thinking of usecases such as https://github.com/rust-lang/rust/issues/43467#issuecomment-405808732 and https://github.com/rust-lang/rust/issues/43467#issuecomment-405808005
jethrogb at 2018-07-26 19:35:21
@jethrogb I would start with just it's prohibited for now.

I fully agree there are valuable scenarios there, but I want to find a safely-separable piece to get stabilized.
scottmcm at 2018-07-26 19:38:39
Not allowing passing extern type in generics would break implementing traits like PartialEq on the extern type itself, and using the blanket impls on references. For forwards-compat we can't allow impls that would overlap if extern type worked with generics so you pretty much have to impl on &FfiTy.
Eduard-Mihai Burtescu at 2018-07-26 19:43:31
Yeah, any blanket impls also wouldn't apply to extern types in that case, which could be frustrating.
Taylor Cramer at 2018-07-26 22:05:36
Not being able to use std::ptr::null for extern types means a severe ergonomic regression if I decided to use them in winapi, which means I would have zero interest in using extern types until that is resolved.
Peter Atashian at 2018-07-26 22:28:32
See #42847 about std::ptr::null of extern type.
kennytm at 2018-07-26 23:05:49
(@Ericson2314 is probably hoarse from repeating this so I'll mention instead that adding a DynSized trait and keeping it unstable should be functionally equivalent to "forbidding extern type in generics" on the stable channel, and may also be a simple way to implement it.)
Gábor Lehel at 2018-07-27 08:10:28
@glaebhoerl DynSized would then be a lang item I presume, and the compiler would look at it when type-checking generics?
Alexander Regueiro at 2018-08-06 15:53:02
I expect it would be an implicit bound on every type parameter like Sized (so yes, almost surely needs to be a lang item for this), except you couldn't then write ?DynSized because it's unstable.
Gábor Lehel at 2018-08-06 19:20:59
@glaebhoerl Makes sense. It might be a bit confusing to users though that it's just "unstable", since the intention is to permanently disallow it. I think stabilising it and explicitly checking to see T: DynSized makes more sense. I do like the idea of the DynSized trait in general though.
Alexander Regueiro at 2018-08-06 21:18:02
The thing I see with DynSized is that even that doesn't make ptr::null work, which seems to be the biggest wish for a better solution to solve.
scottmcm at 2018-08-09 19:20:33
Whether we use an unstable trait or not, we need to prohibit extern type from appearing in these positions:
- the value of a generic type parameter
- the value of an associated type
- the Self parameter for a trait
If using a trait, we also have to get the trait object situation correct, right? That is, we can assume that all dyn Trait types are DynSized or whatever it is called. (But that seems...true.)

Anyway, we discussed this in the @rust-lang/lang and reached no firm conclusions. =) Seems like we need a dedicated slot for discussion or something like that.

EDIT: Removed talk of auto traits, which was just me being confused. I was mostly trying to remember what the pain points are around new auto traits...
Niko Matsakis at 2018-08-09 19:30:59
Regardless of what solution is opted for, I think having a DynSized auto-trait to parallel the Sized auto-trait makes sense now.
Alexander Regueiro at 2018-08-09 20:46:59
@scottmcm why? Through in the Meta associated type (it's an unstable trait, we can always move it later), constrain it to be (), and we're set.
John Ericson at 2018-08-10 00:24:01
@Ericson2314 extern type won't implement DynSized, so making std::ptr::null accept T: DynSized<Meta = ()> won't make it available for extern type.

We'll need a separate trait to provide the Meta associated type.
<details><summary>A proposed hierarchy to address everything</summary>
</details>
kennytm at 2018-08-10 05:35:47
@kennytm You plan to add an Object trait? What are RegularSized, InlineSized? I presume Aligned is just for providing type-level information about alignment.
Alexander Regueiro at 2018-08-10 14:42:07
@alexreg The Object trait would be needed for the Meta associated type. (Another proposal uses the name Referent).

The hierarchy is just to show how extern type does not implement DynSized. (RegularSized (a.k.a. SizedFromMeta), InlineSized and Aligned are off-topic here, they are relevant for a full DST proposal.)
kennytm at 2018-08-10 15:41:12
Sorry about that. Indeed it does not.
John Ericson at 2018-08-11 04:46:25
Sorry about that, must've been me trying to navigate the page, and GitHub "helpfully" focusing on the Edit button without me looking at what it was doing. I don't even have a cat to blame it on!
Eduard-Mihai Burtescu at 2018-09-19 18:28:00
Is there coherent documentation somewhere of what the rules are for struct fields that are extern type? My assumption was that they are ruled out, because we cannot know their alignment and hence cannot know their offset. So miri bails out when it sees such a thing. Now we are seeing ICEs...
Ralf Jung at 2018-11-01 08:08:58
I don't think we've reached consensus about field access, at least not found in https://github.com/rust-lang/rust/issues/43467#issuecomment-406378830. IMO, it is fine to have extern type be the first (and thus the only) field of a struct since you don't need to know the alignment to get the offset. (Maybe need #[repr(transparent)] as I mentioned in https://github.com/rust-lang/rust/issues/43467#issuecomment-351220952). Using extern type as the second field and beyond should be prohibited.
kennytm at 2018-11-01 09:02:39
@kennytm What about if the first fields are zero sized types? If the struct is #[repr(transparent)] and only contains zero sized types plus a single extern type, then it should be equivalent (in terms of representation and computing accesses) as the plain extern type, shouldn't it? At least that's how I understood #[repr(transparent)] to work in situations like this.
Michael Bradshaw at 2018-11-01 13:52:46
@mjbshaw Yeah that could also work.
kennytm at 2018-11-01 14:29:42
Zero sized types have an alignment, too. If that alignment is not 1, then we are in the same situation as we were before, aren't we?
Oli Scherer at 2018-11-01 14:45:45
Yes, repr(transparent) only allows ZSTs with alignment 1 for this reason.
Hanna Kruppe at 2018-11-01 15:05:38
Mmm it's very important to model C++ stuff and other FFI things using extern types as a final field. Having the field should be fine, there just should be no way to access it.

CC @retep998
John Ericson at 2018-11-01 15:24:53
@Ericson2314 Could you provide an FFI example which the tail needs to be totally opaque? The following isn't valid C for instance:
```
struct Opaque;

struct WithOpaqueTail {
    int head;
    struct Opaque tail; // error: field has incomplete type 'struct Opaque'
};
```
(A thin slice like https://docs.microsoft.com/en-us/windows/desktop/api/winnt/ns-winnt-_token_groups doesn't count, it should better be served using Custom DST not hacked away using extern type)
kennytm at 2018-11-01 15:43:41
Well firstly as you know I think of custom DSTs as manually implementing DynSized, so it's nice to start with an opaque tail as a sanity check against the instance doing something funny. In the specific case of an array I rather use [_] and provide it's Meta somehow, but for the case of different structs sharing a prefix I think a purely opaque tail is fine.

And if I had no need to move something, I might not bother to implement DynSized, too.
John Ericson at 2018-11-01 17:37:36
@Ericson2314 but you could use a union for "different structs sharing a prefix"
kennytm at 2018-11-01 17:52:43
Note that in https://github.com/rust-lang/rust/issues/43467#issuecomment-405808732 I linked to https://github.com/rust-lang/rust/blob/f686885a14fff16ddf984b08fb0d9ded07e66f1c/src/librustc/ty/mod.rs#L594-L609 - the current version of that is https://github.com/rust-lang/rust/blob/a3f0f5107e48d47936a604325a1af325033901d0/src/librustc/ty/mod.rs#L599-L615

I really like this pattern, and I don't know what to replace it with, if we decide that extern type fields were not allowed at offsets greater than 0.
Eduard-Mihai Burtescu at 2018-11-03 20:31:11
Long-term the Right Way to encode that is ThinSlice with custom DSTs. But who knows when we'll get an accepted RFC for those.
Hanna Kruppe at 2018-11-03 20:37:08
@eddyb so you are suggesting miri should just assum extern types to have size 0 (i.e., to use layout.size even though they are unsized)? Will whoever later changes this stuff remember to also to it for miri?^^

My biggest concern here is not how to implement something in miri, but how to keep it in sync later.
Ralf Jung at 2018-11-04 09:42:32
@RalfJung The size doesn't matter for fields' offset, the alignment does. IMO we could panic in size_of_val while having align_of_val return the alignment, which maybe we could also allow controlling with #[repr(align(N))] on the extern type.

There's also the question of defaulting to byte alignment or always requiring #[repr(align(N))].
Eduard-Mihai Burtescu at 2018-11-04 13:38:35
You are right. So you are suggesting that miri assumes extern types to have the alignment given in their layout, but unknoweable size?

That does seem reasonable, but IMO it should be consistent with what LLVM codegen does so I'll hold off on implementing anything here.
Ralf Jung at 2018-11-04 13:45:19
Note that for fixing #55541, you only need to assume an alignment, you can leave the size unknown.
Eduard-Mihai Burtescu at 2018-11-04 14:17:12
Following up on that comment above, @eddyb made another remark at https://github.com/rust-lang/rust/pull/55672#discussion_r230590431:

You can't access that alignment without first having a reference, so having it too high would be a problem (the assumption might not be guaranteed by what you're FFI-ing to), but having it too low wouldn't break anything (other than field offsets).

IOW, the proposal is to make align_of_val work for extern type (and have it return the alignment annotated at the type, or 1 per default), but make size_of_val panic. That is enough to make field access never panic. It is also justified because without a size, we can never allocate such a type, and hence we only ever "consume" the alignment: When we get pointers to such types, we rely on them being sufficiently aligned, but we never create pointers to such type, so the fact that our alignment might be too low is not a problem.

Does that seems like a proposal worth considering?
Ralf Jung at 2018-11-05 10:18:20
@RalfJung Yes but

But we never create pointers to such type

But to be pendantic, in computing the field offset we are creating a pointer. If we were given the layout of the struct up front I'd say we would indeed be "rely[ing] on them being sufficiently aligned", but we are computing it ourselves from the information about the types of the fields.

As such, I'd say we should ask for an explicit #[repr(align(N))], as @eddyb floated. I want to make as few presumptions as possible in the no-information case. But if we put in core a Void for C, or OpaqueData, or any other "off the shelf helper", we should give it #[repr(align(1))].
John Ericson at 2018-11-05 19:31:53
in computing the field offset we are creating a pointer

Fair enough. However, we do so only by computing an offset on an already existing pointer.

But yes, in a case like (i32, ExternType), we might indeed be computing the wrong offset for the 2nd field, I have not considered that. In Rust itself that cannot be a problem, because you cannot do much with that pointer, but if you then feed it back to the other side of the extern interface things will go wrong.

Seems like there is no good way to support your "opaque slice" type after all, @eddyb. :/ Though TBH it seems more like a hack anyway.

A mandatory alignment annotation would be an option, but I suspect many people will be rather surprised by that requirement.
Ralf Jung at 2018-11-05 20:01:40
Seems like there is no good way to support your "opaque slice" type after all, @eddyb. :/ Though TBH it seems more like a hack anyway.

In @eddyb's case the opaque tail doesn't correspond to any type on the foreign side that can be pointed to directory, so the field offset wouldn't be done anyways, so it still works.

A mandatory alignment annotation would be an option, but I suspect many people will be rather surprised by that requirement.

Eh, this just means that size_of_val and align_of can panic. And if unstable DynSized ends up happening as a practical way to prevent the instantiation of generics with extern types, then it's nice and symmetrical and #[repr(align(N))] could be the stable way to write an impl of some unstable Align trait.
John Ericson at 2018-11-05 20:55:35
#[repr(align(N))] on the extern type

I don't object to that idea. However, note that the lack of #[repr(align(N))] doesn't mean, necessarily, 1; it more likely means "I don't know" or "the alignment isn't part of the ABI" or more generally "the alignment requirement is a function of more than just the type of the object".
Brian Smith at 2018-11-05 20:59:38
Eh, this just means that size_of_val and align_of can panic.

size_of_val sure, but align_of?

it more likely means "I don't know" or "the alignment isn't part of the ABI" or more generally "the alignment requirement is a function of more than just the type of the object".

So, as long as we make sure that struct fields have a manually specified alignment we are good? We could check that when structs are defined and on generic type parameters. That is, allow extern type without repr(align), but do not allow such "underspecified" extern type to ever occur in a struct. That means align_of_val can panic, but we know it will not when used for field access.

(We could also allow to write down the struct but then not allow to access the field, but that seems more confusing.)

I recall that there once was the proposal to not allow extern type as generic type parameters, but I do not remember why that was not pursued. Generally there have been tons of designs considered here, is there a place documenting why which desig was discarded? The RFC does not mention any of this.
Ralf Jung at 2018-11-06 08:05:51
I recall that there once was the proposal to not allow extern type as generic type parameters, but I do not remember why that was not pursued.

@RalfJung https://github.com/rust-lang/rust/issues/49708#issuecomment-380216336
kennytm at 2018-11-06 08:49:23
That is, allow extern type without repr(align), but do not allow such "underspecified" extern type to ever occur in a struct.

Yes

size_of_val sure, but align_of?

Yes. Struct alignment is static so it uses align_of not align_of_value. Without the restrictions on generic instantiation with extern type, it is still possible to end up with a alignment-less extern type in a struct field, so the only sane thing to is panic (or post monomorphism error).

Panics and post-monormorphism errors suck, so using more traits like DynSized and Aligned to statically prevent these bad things by construction, while not stabilizing them to avoid constraining our future selves, is very nice and expedient.
John Ericson at 2018-11-06 16:57:27

Comments from jethrogb and Ericson2314 mention that LLVM uses a special type opaque instead of Rust's current empty type, but I can't find any follow up on it. Rust still uses an empty type:

Rust

1.33.0-nightly 2019-01-04 f381a962550436f74dd6

extern {
    type nuffin_t;

    fn myalloc() -> *mut nuffin_t;
    fn myfree(_: *mut nuffin_t);
}

%"::nuffin_t" = type {}

declare %"::nuffin_t"* @myalloc() unnamed_addr #0
declare void @myfree(%"::nuffin_t"*) unnamed_addr #0

C

clang version 8.0.0 (trunk 350447)

struct nuffin_t;

struct nuffin_t *myalloc();
void myfree(struct nuffin_t*);

%struct.nuffin_t = type opaque

declare dso_local void @myfree(%struct.nuffin_t*) #1
declare dso_local %struct.nuffin_t* @myalloc(...) #1

Jake Goulding at 2019-01-05 15:25:42

Good catch!
John Ericson at 2019-01-05 15:42:11
@shepmaster I independently came across that (well, @RalfJung pointed it out) and I opened https://github.com/rust-lang/rust/pull/58271 last week, which I'm not sure how to fix.

If what I did is different from type opaque, that might be why it fails.

But also, note that type {} is just a tuple with no fields, i.e. LLVM's (), not an "empty type" in the Rust sense of enum Void {}.
Eduard-Mihai Burtescu at 2019-02-13 11:48:56
I've been trying to use extern type to model some variably-sized data that needs to co-operate with a C API.

I've come across an issue where I'd like to define a method which involves casting from various raw pointers, come of which are Sized and some of which are extern type. In trying to make the code common, I need to use T: ?Sized, but that then creates a problem, as it 'pulls in' all types, including fat pointers, which I don't want (the underlying C code assumes that all pointers are usize in size).

Is there any way to use the type system to differentiate extern type from just ?Sized (similar problems have bit me in the past - it would be nice to use the type system to restrict based on sizes or alignment patterns or repr(u16) or things being enums with specific layouts - but ho-hum).
Raphael Cohn at 2019-03-06 15:02:00
@raphaelcohn I believe the plan is to add a DynSized, so you could use T: DynSized to include basically all types except extern types. See https://github.com/rust-lang/rfcs/issues/2255 for more info. I'm not the most knowledgeable about this though.
Alexander Regueiro at 2019-03-06 15:52:54
@alexreg yeah so that's the opposite of what @raphaelcohn wants I think :)
jethrogb at 2019-03-06 18:05:59
https://github.com/rust-lang/rfcs/pull/2580 is another proposal in somewhat the same space as DynSized. With that implemented, you could use T: ?Sized + Pointee<Metadata=()> to restrict a type parameter to types that have thin pointers pointing to them, including extern types.
Simon Sapin at 2019-03-06 18:50:53
@jethrogb It's exactly what he wants, the way I read it. Or maybe he needs both. Regardless, negative bounds for auto traits exist, so e.g. !DynSized would work.
Alexander Regueiro at 2019-03-07 00:39:31
@alexreg You could write impl !Send for Stuff {} but that doesn't mean where Stuff: !Send is a valid bound.
kennytm at 2019-03-07 06:29:02
@alexreg, @jethrogb @SimonSapin @kennytm Thank you all. Essentially, I need a bound that says T: thin pointers. I think @SimonSapin's solution T: ?Sized + Pointee<Metadata=()> would work, but it does seem rather non-intuitive.
Raphael Cohn at 2019-03-07 12:46:59
Indeed, the RFC also proposes a trait alias pub trait Thin = Pointee<Metadata=()>; in the std::ptr module. But at this point it’s only a proposal.
Simon Sapin at 2019-03-07 15:33:44
@kennytm It's not? Well, it needs to be!
Alexander Regueiro at 2019-03-07 16:42:13
For the record, I think we need Pointee and DynSized. Thing + ?Sized still means you are responsible for coming up with the dynamic size and allocation from a (), which means panicking, which is no good. Likewise ?DynSized without Thin means you can do unsafe operations, but have no means to ensure the ABI is correct, so generic FFI stuff cannot be written. Both is wonderful, and will making custom DSTs so much easier to discuss because the unnecessary degrees of freedom don't exist.
John Ericson at 2019-03-07 18:29:39
And anything that lets me safely coerce a slice to an iovec would be most welcome...
Raphael Cohn at 2019-03-08 13:48:30
Are the "Unresolved questions" in the OP still blocking this from stabilization? Are there any other unresolved questions or issues that are also blocking this?
Michael Bradshaw at 2019-07-07 20:22:47
If this doesn't support generics, I'd expect people to hack it like this:
```
extern {
type Helper;
}

pub struct MyExternType<'a, T> {
 _phantom: core::marker::PhantomData<&'a T>,
_helper: Helper,
}
```
Since the suggested syntax doesn't support it directly, I think that the idea with marker is superior. Maybe worth revisiting.
Martin Habovštiak at 2020-04-12 11:09:02
@Kixunil Btw you probably want &'a mut &'a T (or *mut &'a T), to be invariant over those parameters, unless you really want variance (it's wrong if you use &MyExternType<'x, Y> for any mutation that involves types like &'x Y, although I'm not sure what all the conditions are - better make it invariant if you're not sure either).

But yeah, that's a good way to describe some FFI APIs, although without https://github.com/rust-lang/rfcs/issues/2770, it's more painful than it has to be.
Eduard-Mihai Burtescu at 2020-04-12 12:00:30
@eddyb oh, I was unclear, I didn't mean &'a T specifically, I meant "this is the syntax one would use to implement any variance needed using PhantomData".
Martin Habovštiak at 2020-04-12 12:10:21
One advantage extern { type Foo<'a, T>; } syntax would have, is that it could only be invariant. So it'd be "correct by default".
Eduard-Mihai Burtescu at 2020-04-12 12:13:05
Would it be possible to override it if one knows what he's doing though?
Martin Habovštiak at 2020-04-12 13:04:24
No, you'd need to do it like in your example if you wanted something weaker.
Eduard-Mihai Burtescu at 2020-04-12 13:12:06
I want to add a note to this discussion: the current implementation seems to be !Unpin. I think this is correct, but haven't seen that mentioned explicitly.
Florian Gilcher at 2020-12-17 10:23:20
Update on null and null_mut for extern types: https://github.com/rust-lang/rust/issues/42847#issuecomment-762437779
Simon Sapin at 2021-01-18 19:39:37
@SimonSapin in https://github.com/rust-lang/rust/issues/42847#issuecomment-762437779 rightly points out that Sized now should imply other things, and so no longer is a root of the trait hierarchy. I hope this can help dispel the resistance to DynSized.

resisting math is futile, accept Sized: Thin + DynSized.
John Ericson at 2021-01-18 19:51:35
You’re putting words in my mouth. That the trait resolver should be able to deduce Metadata == () from T: Sized is not an argument for adding a new opt-out ?Trait, nor does it alleviate the practical concerns with those proposals.
Simon Sapin at 2021-01-18 21:34:42
https://github.com/rust-lang/rfcs/pull/2984#issuecomment-762540405 what I wrote that I don't want to twist your words, I just see that chipping away at the argument against effectively.
John Ericson at 2021-01-19 01:40:29
I would like to raise an issue about this. extern type does not currently have a way to specify the type class of the extern type. This does not pose an issue in rust, but may affect the ability to use this with a C abi.

In ISO 9899 N1124, §6.2.5 Clause 26 defines the classes of pointer types which are required to be compatible. Notably, pointers in the following classes MUST have the same representation and alignment-requirements with others in the same class, and no further requirements are imposed:
- Pointers to void and pointers to character types (char, unsigned char, and signed char. i8 and u8 I believe are guaranteed to be in this class)
- Pointers to any structure types.
- Pointers to any union types (yes this is distinct from the first category)
- Pointers to any otherwise compatible types (IE. const/volatile qualified, signed/unsigned integer types)
This proposal seems to only allow one, presumably unified ABI (if not, there is no indication of which of the listed classes extern types are guaranteed to be included in). This is not generally compatible with the C abi. Unless rust further restricts compatible C platforms, there needs to be a way to select at least between the classes of struct and union. In a discussion on the Rust Programming Language Community [Discord] Server, a conversion made mention that the class of character types would also be a good idea (though this can be suplemented by a newtype arround u8 or i8, or a repr(u8)/repr(i8) enum). This would solve the c_void problem.
Connor Horman at 2021-01-20 14:27:56
@chorman0773 Are there any existing ABIs that differentiate between pointer types? Strictly speaking C doesn't have or define an ABI, only implementations of C's "abstract machine" have an ABI (i.e. OS and ISA). If there are ABIs that really do differentiate between pointer types then I suppose something like extern { struct Foo; union Bar; } would probably be sufficient.
Michael Bradshaw at 2021-01-20 14:47:53
@mjbshaw An option for void/character-like extern types would also be highly appreciated, at least on my part.

It's not strictly necessary to correctly interface with C due to *mut c_void being available, but it still makes for a fairly large ergonomics improvement when FFI-binding APIs that take/provide *void with effectively constrained lifetime. (e.g. an OOP API that holds user data pointers in destructible instances)

These parameters/return values could then be written as extern type references (which creates appropriate speed bumps when using the API), instead of adding the lifetime constraints to the /// documentation and hoping that everyone reads them thoroughly all the time.

(I'm mostly talking about myself here, though, since I'm a bit prone to these types of mistakes if I don't add the constraints from the get-go.)
Tamme Schichler at 2021-01-20 15:08:32
Are there any existing ABIs that differentiate between pointer types? Strictly speaking C doesn't have or define an ABI, only implementations of C's "abstract machine" have an ABI (i.e. OS and ISA). If there are ABIs that really do differentiate between pointer types then I suppose something like extern { struct Foo; union Bar; } would probably be sufficient.

I am not aware of any, though I am only familiar with the System V x86_64 abi. I also raised a similar issue in https://github.com/rust-lang/unsafe-coding-guidelines/issues/266, and one of the comments pointed out control-flow integrety schemes. I don't know that this is mandated in any ABI, though as it mentions, clang supports it with a software implementation (and it seems one of the santizers does check indirect function calls, though I don't see anything for direct calls to a free function).
Connor Horman at 2021-01-20 15:19:54
As an arbitrary end user that's picked up rust in the last year, can I say the concern about the teachability of ? bounds seems a bit arbitrary? It sounds like there were some larger design issues involved, but I strongly feel ?Movable would have been (probably still would be) much easier to figure out than the current Pin / !Unpin design, and I don't have any issue (usage wise) with the idea of something like ?DynSized showing up - morally it seems equivalent to an auto trait but less wordy in the common cases.

Otherwise, given the various use cases I've been reaching for something like this, and the discussion so far, here's my feelings:
- Generic type parameters at least are really valuable, but seem like they would be easy to wrap. But then, it seems we could just use a marker type and always wrap instead of a stable language feature?
- The idea of size/align_of_val panicking or aborting makes my skin crawl. This seems like a real minefield for combining crates in particular: consider upgrading crates A and B where A started internally using extern type or B added usage of size_of_val - now as a downstream user I have to try to get this figured out, where both can just push it into the other or me to "just not do that". It's nice when they're nice, but at least with say ?DynSized language crate A has to be clear that they are the one deliberately handing out a weird thing, and that it's a breaking change, while B can accept a PR adding support for taking ?DynSized as a clear general feature, while also being able to easily find out what the implications of that are.
- The thin pointer discussion seems a bit confusing: you want a trait on the extern type itself, rather than a wrapped pointer?
- Otherwise this seems to be getting stuck on field offsets behavior? That seems like a clear place to split out another feature that could depend on custom DSTs or whatever?
Simon Buchan at 2021-03-06 02:16:03
@simonbuchan please see rust-lang/rfcs#2255 for the concerns around ?Trait. Teachability is just one part of it, there are also more objective reasons such as backward compatibility.
kennytm at 2021-03-07 18:18:50
@kennytm yeah, it sounded like that, but I didn't have any particular info, and the teachability seemed to be what was most referenced in this thread. Thanks for the link, though!
Simon Buchan at 2021-03-07 22:00:57
@simonbuchan FWIW I am skeptical of any comparability issues being insurmountable. I left a comment in that other thread to that effect, as it looks like such a counter-point hadn't been raised before. I haven't used Pin or futures yet, but Pin does look spooky and sub-part from afar to me too.
John Ericson at 2021-03-08 03:41:00
I have a concern regarding lifetimes/PhantomData.

I sometimes need opaque types that conceptually borrow other types. For example, a C library I'm binding to has an opaque type called a Sprite, which can "borrow" a Texture.
```
pub struct Sprite<'s> {
    _opaque: [u8; 0],
    texture: PhantomData<&'s Texture>,
}
```
<details> <summary>How it works</summary>
To create a new instance, we do a C FFI call and wrap it in a special box type that calls the C destructor on Drop
```
pub fn new() -> SfBox<Sprite<'s>> {
        let sp = unsafe { ffi::sfSprite_create() };
        SfBox::new(sp as *mut Self).expect("Failed to create sprite")
}
```
Creating borrowed variants from FFI calls is also supported (This one is a mock because Sprite specifically doesn't do this, but other similar opaque types do)
```
impl<'s> SomeOtherType<'s> {
    pub fn get_sprite(&self) -> &Sprite<'s> {
        unsafe {
            let ptr = ffi::someOtherType_getSprite(self.raw());
            &*(ptr as *const Sprite)
        }
    }
}
```
The methods call C FFI functions, with a pointer cast
```
pub fn local_bounds(&self) -> FloatRect {
        unsafe { FloatRect::from_raw(ffi::sfSprite_getLocalBounds(self.raw())) }
}
pub(super) fn raw(&self) -> *const ffi::sfSprite {
        let ptr: *const Self = self;
        ptr as _
}
```
This pattern allows to easily create both owned and borrowed variants of Sprite from FFI calls, and keep lifetime information.
</details>
Since extern types don't support PhantomData, I'm not sure how this could be pulled off with them.
crumblingstatue at 2021-08-06 21:49:55

@crumblingstatue something like this should work:

extern {
  type Opaque;
}

pub struct Sprite<'s> {
    texture: PhantomData<&'s Texture>,
    _opaque: Opaque,
}

Ralf Jung at 2021-08-07 09:01:37

@RalfJung I'm running into trouble with trait objects

Is it possible to make &ExternTypeWrapper castable to &dyn Trait?

#![feature(extern_types)]
use core::marker::PhantomData;
extern {
  type Opaque;
}

pub struct Sprite<'s> {
    texture: PhantomData<&'s ()>,
    _opaque: Opaque,
}

trait Drawable {}

impl<'s> Drawable for Sprite<'s> {
    
}

fn draw(d: &dyn Drawable) {
    
}

fn main() {
    let s: &Sprite = unimplemented!();
    draw(s);
}

error[E0277]: the size for values of type `Opaque` cannot be known at compilation time
  --> src/main.rs:24:10
   |
24 |     draw(s);
   |          ^ doesn't have a size known at compile-time

crumblingstatue at 2021-08-07 11:24:46

Hm, I don't think that is possible -- for &dyn Trait, the actual dynamic size of the object is given in its vtable; that is simply not possible for Extern types.
Ralf Jung at 2021-08-07 11:34:53
Is it a soundness hole that this cast is possible with Sized "opaque" types? If not, then it should be considered that extern types are not a replacement for "manually enforced" opaque types in all cases, and some level of support should remain for them.
crumblingstatue at 2021-08-07 11:49:06
Presumably an opt-out syntax like trait Trait: ?DynSized {...} would allow dyn Trait that can be used e.g. behind references, but not heap-allocated, or passed by-value on the stack, etc.
Eduard-Mihai Burtescu at 2021-08-07 13:41:37
Smaller vtables without size/align also seem interesting for other reasons.
Ralf Jung at 2021-08-08 16:18:31
- In std's source, it is mentioned that LLVM expects i8* for C's void*. We'd need to continue to hack this for the two c_voids in std and libc. But perhaps this should be done across-the-board for all extern types?
Is there a separate issue for exploring this unresolved question, if it is still relevant with current LLVM? Could it be an option to codegen all extern types as i8 so that the type signatures of malloc etc. match what LLVM expects?
Matthias Schiffer at 2021-12-16 19:27:15
Note that LLVM is moving to opaque pointers (see https://llvm.org/docs/OpaquePointers.html) so I would expect it to be irrelevant in the not-too-distant future.
scottmcm at 2021-12-16 19:56:22
We discussed this in the lang team backlog bonanza today.

We'd like to make progress on this one, but were unsure exactly where things stood related to these around the big questions like whether pointers are thin and ptr::null works (did https://rust-lang.github.io/rfcs/2580-ptr-meta.html change anything here?), whether size_of_val would panic, and related.

Is anyone interested here in summarizing the state of things?
scottmcm at 2022-02-09 18:19:19
I didn’t realize it was a question whether pointers to extern types should be thin. What metadata would there be?

RFC 2580 was accepted saying that null, null_mut, and NonNull dangling should be extended to T: ?Sized + Thin but that’s not implemented yet. The "natural" way to do it would requires a similar language change in integer-to-pointer casts with as, which I imagine would require some logic in the compiler to reason about the thin-ness of pointers to some generic type parameter based on the presence of a Pointee<Metadata = ()> bound, like it does today with (potentially implicit) Sized bounds.

That change in as semantics was not parts of the RFC though, so T-lang should probably discuss it specifically.
Simon Sapin at 2022-02-10 08:39:37
I am not sure if this relates to the current iteration of the question, but discussions about custom DSTs in the past had extern types as a building block, with the user able to specify a custom Metatdata used to help define size_of_value and align_of_value.

...like it does today with (potentially implicit) Sized bounds.

Please, let's cross the implicit bound rubicon!
John Ericson at 2022-02-12 17:26:45
I think custom metadata would be compatible with extending null and friends to T: ?Sized + Thin. (with Thin = Pointee<Metadata = ()>). Those custom DSTs would not be Thin.

As to implicit bounds I’m only talking about the existing ones: T: Sized is assumed in most (all?) generic contexts unless T: ?Sized is specified. I’m not proposing adding any more. As far as I understand this is how as is legal here:
```
fn null<T>() -> *const T {
    0 as *const T
}
```
Simon Sapin at 2022-02-12 18:22:28
Ah, yours does work today, but
```
fn null<T: ?Sized>() -> *const T {
    0 as *const T
}
```
is does not. Great! That means the null, null_mut, and NonNull changes should be just fine, but also orthogonal to this issue, I think?
John Ericson at 2022-02-13 00:59:05
This is the tracking issue for extern types in general, so whether null and friends work with them is relevant. But yes it’s a sub-topic that can be discussed separately to reduce load on this already long thread. I’ve opened https://github.com/rust-lang/rust/issues/93959
Simon Sapin at 2022-02-13 10:19:00
questions like whether pointers are thin and ptr::null

https://github.com/rust-lang/rust/pull/94954 implements this part of the pointer metadata RFC
Simon Sapin at 2022-03-15 09:36:09
I haven't been involved in this so far but I'm going to attempt to summarize where this is in the hope that it allows some forward progress.

The original RFC kind of relied on the custom DSTs RFC which has been postponed, this means some of the workarounds mentioned may need to be made more permanent for this to be stabilised. Extern types have been implemented in #44295, this also flagged some questions not covered by the RFC. The other relevant piece of work is the pointer metadata RFC (tracking issue).

Some of this is copied from @jethrogb's summary above: https://github.com/rust-lang/rust/issues/43467#issuecomment-406378830

Should we allow generic lifetime and type parameters on extern types?

This appears to be entirely resolved by wrapping the extern type in a #[repr(transparent)] wrapper and adding some PhantomDatas as in: https://github.com/rust-lang/rust/issues/43467#issuecomment-405808005

That does assume you can put extern types in transparent structs, see later, if not then:
1. Do nothing, I haven't seen any particularly compelling arguments they're necessary.
2. Allow them and just assume everything's invariant.
3. Come up with some way of defining them and their variance (these feels beyond the scope of this feature).
What LLVM type should be used?

Extern types currently use type {} which is LLVM's equivalent of (). However it's desirable for c_void to be an extern type (see https://github.com/rust-lang-nursery/portability-wg/issues/13) but that must currently be an i8. This may all be made simpler by LLVM opaque pointers which will be required by LLVM 16 and is probably advisable for LLVM 15. Also see #59095.

std::ptr::null/null_mut

These functions did not work as T had to be sized, the pointer metadata RFC weakened this to T: ?Sized + Thin, so these now work correctly since #94954 was merged.

size_of_val (and partially align_of_val)

size_of_val / align_of_val can be called with extern types, and respectively return 0 and 1. This differs from the RFC, which specified that they should panic. This is essentially always wrong and not very Rust-y, a previous lang meeting suggested size_of_val should be changed to panic and a lint added to try and spot this at compile time. align_of_val should probably act similarly but has extra difficulties explained below.

Auto traits are not implemented by default.

Manual unsafe impl Sync for Foo is possible for all auto-traits except for Freeze. Types like CStrshould be extern types (or at least Thin DSTs) but don't have interior mutability. However, Freeze is private so it's not observable whether it's implemented and it's only used for optimisations so it's probably fine to ignore.

Trait objects cannot be built from extern types.

&Extern as &dyn Trait doesn't work because Extern is !Sized, however it is Thin so this could be allowed. Unfortunately this hits the unknown size & alignment problem below so may not be possible after all. Interestingly &Foo<Extern> as &Foo<dyn Trait> does work and gets the size & alignment wrong.

type class of the extern type

See https://github.com/rust-lang/rust/issues/43467#issuecomment-763643067. The C ABI allows for different address spaces (in LLVM terminology). I don't know whether Rust supports this anywhere though.

Unknown alignment of extern types

This is the big one and is hard to summarise, the RFC allows types like:
```
extern {
    type OpaqueTail;
}

#[repr(C)]
struct FfiStruct {
    data: u8,
    more_data: u32,
    tail: OpaqueTail,
}
```
Note I'm not clear that the above should be allowed, C rejects it for the reason I'm about to describe.

However the following code causes everything to blow up:
```
extern {
    type OpaqueTail;
}

#[derive(Debug)]
#[repr(C)]
struct Foo<T: ?Sized> {
    data: u8,
    tail: T,
}

fn foo_tail(foo: &Foo<OpaqueTail>) -> &OpaqueTail {
    &foo.tail // What offset is this field at?
}

fn hidden<T: Debug>(foo: &Foo<T>) {
    println!("{:?}", foo as &Foo<dyn Debug>) // extern type field access hidden behind trait object coercion
}
```
Currently rustc assumes that all extern types have alignment 1 but this is clearly wrong. There have been a few suggestions for how to fix this:
1. Post-monomorphisation error/lint - rustc will know when it's trying to layout a struct like this so could just throw a compile error, however there are good reasons to dislike post-monomorphisation errors in general and this would prevent the OpaqueTail pattern.
2. Allow #[repr(align(N))] on extern types - would be useful but doesn't solve the problem because there are opaque types where the alignment is unknown.
3. Prevent extern types being supplied as generic parameters, this could prevent legitimate use cases, may prevent blanket trait implementations from being applied, and may prevent you from implementing traits like PartialEq.
4. A DynSized trait this does approximately the same as the above but could allow ?DynSized bounds to selectively relax the restrictions. This has been discussed extensively before https://github.com/rust-lang/rust/pull/46108#issuecomment-360903211, https://github.com/rust-lang/rfcs/pull/2310, https://github.com/rust-lang/rfcs/issues/2255#issuecomment-361040229.
There is evidence to suggest something like OpaqueTail is wanted, see this rustc code but obtaining a reference the tail is only safe if you can #[repr(align(N))] the type (or field). Otherwise I feel like you actually want (a thing I just made up):
```
#[repr(C, unsized)]
struct Header {
    data: u8,
}
```
Where next?

The conclusion from a previous lang team meeting was that the correct answers to a bunch of the hard questions here is tied to custom DSTs and we should attempt to find the minimum possible thing to stabilise. However the alignment issues makes deciding what that minimum thing is difficult. Especially as custom DSTs may never land.
Jack Rickard at 2022-08-06 18:12:14
Nice summary!

Interestingly, if we had #[repr(unsized)] it would seemingly be redundant with extern { type Foo; }, as you could use #[repr(unsized)] struct Foo. That implies that it should be spelled extern { struct Foo { head: u32, } }.
Simon Buchan at 2022-08-06 21:31:11
It sounded to me like there was some interest from the lang team on trying out DynSize on a super unstable basis in the last few months.
John Ericson at 2022-08-07 01:02:31
I don't quite understand why giving extern types alignment 1/no alignment is wrong? It may be that the type Foo in your example doesn't report the "correct" alignment, but since the user can never create it manually, I assume that it would always be created by something that does know the required alignment.

Building upon your example, playground link (which doesn't compile).:
```
#[derive(Default, Debug)]
#[repr(C)]
struct Bar {
    data: u8,
    // 8 bits of padding here
    x: u16,
}

fn main() {
    let bar = Bar::default();
    let foo: &Foo<OpaqueTail> = unsafe { mem::transmute(&bar) };

    // Possible, but would just point to the padding, and effectively useless
    let tail = foo_tail(foo);

    // Not possible, since `OpaqueTail` doesn't implement `Debug`
    hidden(foo);
}
```
Note that &foo.tail is allowed, but you can't do anything useful with it without first converting it to another type. But maybe there's something I'm missing here?

Related, I opened a proposal over at the Unsafe Code Guidelines to change how extern types works wrt. references: https://github.com/rust-lang/unsafe-code-guidelines/issues/356
Mads Marquart at 2022-08-10 15:37:08
Hopefully this playground demonstrates the issue: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=b7350128f9f2869c7eda52d1574c78d9

Edit: Alternatively, here's a minor alteration that segfaults! https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=5b180f64c9c39a555114ca01bcf60d18
Jack Rickard at 2022-08-10 17:06:44
Yeah, though I don't think this is an issue since this kind of API doesn't exist in the real world? Or am I wrong here? I should think use_foo would usually take *const CHeader, and then the problem would be moot.

My point is, I can't see the reason we should disallow Rust from making references to opaque types, it's just that actually using them, as in your example, is impossible without further knowledge (e.g. maybe it is documented that CFoo has pointer alignment, and then one could include the padding and raise the alignment of Header such that it would work fine).
Mads Marquart at 2022-08-10 17:24:04
@madsmtm If the foreign type is the tail of the a struct, we want the field to be at a proper offset.

Also, it's just very bad luck to say "It's OK to lie here because it doesn't matter", thing change including the whatever underpins that rational today. I prefer not to make decision decision that will cause massive headaches for designs (including future me) tomorrow.
John Ericson at 2022-08-10 17:56:07
Yeah, it seems like for soundness we should disallow extern types to be used in non-transparent structs. But of course when generics are involved we can't really do that. So this is yet another way in which extern types cause problems when being used to instantiate generic (even ?Sized) types.
Ralf Jung at 2022-08-10 18:15:45
@madsmtm If the foreign type is the tail of the a struct, we want the field to be at a proper offset.

Also, it's just very bad luck to say "It's OK to lie here because it doesn't matter", thing change including the whatever underpins that rational today. I prefer not to make decision decision that will cause massive headaches for designs (including future me) tomorrow.

I guess my understanding of extern types were more "opaque blob of bytes of unknown length" rather than "corresponds to a specific C type, we just don't know the size of the type". Under the former, I think it is perfectly rational that the extern type would include any padding, but of course it isn't under the latter.

Note that I don't have any experience with language design, I'll believe you if you say the latter way of viewing extern types is correct!
Mads Marquart at 2022-08-10 18:39:37
The conclusion from a previous lang team meeting was that the correct answers to a bunch of the hard questions here is tied to custom DSTs and we should attempt to find the minimum possible thing to stabilise. However the alignment issues makes deciding what that minimum thing is difficult.

As someone who needs opaque pointers from time to time to interface with C, and who finds the workaround suggested in the Rustonomicon painfully ugly and error-prone, thank you for the summary, @Skepfyr.

I wonder to what extent trying to solve the harder problem of also allowing opaque tails is blocking progress on stabilizing a much simpler first solution that would for now only enable opaque pointers, and which might hopefully already cover 95% of the actual uses.

In particular, I personally would be delighted by a concept of an "thin opaque type" that would try to closely mirror the semantics of an incomplete struct in C and C++ as much as possible. In particular, it would
- enable thin pointers and references
- allow generics
- give a compile-time error after monomorphization of size_of and align_of
  - (If that's really undesired, a panic with a linter hint is probably fine, but I don't see why that would be an improvement.)
- give a compile-time error after monomorphization when used as a member of a non-transparent struct
Such a solution would be perfectly open to future extensions that would enable more programs to compile if certain hints for the type are given.

Post-monomorphisation error/lint - rustc will know when it's trying to layout a struct like this so could just throw a compile error, however there are good reasons to dislike post-monomorphisation errors in general and this would prevent the OpaqueTail pattern.

I agree that post-monomorphization errors feel un-Rusty. Unfortunately, that seems almost unavoidable to be fully compatible with C here, which is what this feature is all about.
Florian Häglsperger at 2023-02-12 09:54:14
I'm going to pretend that the following traits exist and that Sized isn't implied by default to make talking about all this easier:
```
trait DynAligned { fn align(&self) -> usize; }
trait DynSized: DynAligned { fn size(&self) -> usize; } // This is what ?Sized means.
trait Aligned: DynAligned { const ALIGN: usize; }
trait Sized: Aligned + DynSized { const SIZE: usize; } // Equivalent to the existing Sized trait.
```
Fundamentally some extern types need to be !DynAligned, this means that they cannot be placed in structs, but also that means that they don't satisfy any existing generic trait bounds (implicit or explicit) so can't be used in generics.

I'm not entirely convinced that what I said above is true, it's possible that all extern types have a alignment known at compile time, and #[repr(align(n))] solves half the above. Even then the types would be Aligned but not DynSized which means they don't quite satisfy ?Sized.

We could avoid the above by not having traits and just making any issues be post-monomorphisation errors. There are some of those already but I cannot see that being acceptable, what's the point of traits if we just ignore them sometimes? On the other hand, the compiler does understand size and alignment unlike most traits, and we could make everything "just work" with post-monomorphisation errors (it would mean a bunch of special casing in the compiler though).

I think that sorting all this out may require another RFC that supplants this one, given the kinds of questions that we're asking.
Jack Rickard at 2023-02-12 16:18:21
I've opened rust-lang/rfcs#3396 in a bid to fix this properly.
Jack Rickard at 2023-02-26 00:47:23
https://github.com/rust-lang/rust/issues/115709 is another issue caused by extern types looking like regular unsized types, but not having a dynamically fixed size.
Ralf Jung at 2023-09-09 14:18:22
I feel like I'm about to ask a very dumb question, but I haven't been able to find the answer in this thread or the original RFC: why are opaque types and unsized types being conflated into the same notation and forced to stabilize together?
1. Opaque types in FFI, the equivalent of a C forward declaration, are the natural fit for extern type. They have an unknown size and alignment, and therefore can only exist as references. They can't be struct fields because their alignment is unknown/unknowable[^alignment], and size_of_val() / align_of_val() cannot return meaningful results for them. These challenges don't seem to have easy answers, so stabilization is still some distance away.
2. Thin pointers to !Sized types, which can report their dynamic size through some type-specific mechanism. This seems like it should be pretty easy to separate out and stabilize, because there's not much difference from other Rust DST references (except that there's no metadata at all).
A lot of the inbound links to this feature / tracking issue are about category (2) -- the idiom of "fixed-size header plus blob of data" is quite common. It would be nice if better support for that pattern could be disentangled from extern type.

To throw out a probably oversimplified idea: allow impl !Sized on a type that would otherwise be Sized, with a mandatory method size_of_val:
```
#[repr(C)]
struct ThinSlice<T> {
    len: usize,
    values: [T; 0]
}
impl<T> !Sized for ThinSlice<T> {
    fn size_of_val(&self) -> usize {
        let pad = align_of::<T>().checked_sub(size_of::<usize>()).unwrap_or(0);
        size_of::<usize>() + (self.len * size_of::<T>()) + pad
    }
}
```
This notation for unsized types bypasses the known challenges with extern type:
- The alignment and field layout of ThinSlice is known at compile time, the same as current-day DSTs.
- core::mem::size_of_val() is guaranteed a piece of code to delegate to, which knows how to compute the value size.
  - Even if it panics, that's a decision of the trait author and not anything inherent to Rust.
- It's possible to impl !Send with feature(negative_impls) so the notation is familiar, but impl !Sized isn't currently allowed by negative_impls so it won't conflict with any existing code (stable or nightly).
  - The concept of of a mandatory method in a negative impl is new, but IMO pretty low on the weirdness meter.
- The requirement that the type otherwise be Sized means that it shouldn't conflict with future custom DSTs? I think?
- No ?DynSized or other new optional trait bounds -- any given type is still either Sized or !Sized.
[^alignment]: A forward-declared C struct is allowed to change its alignment between versions/implementations of the underlying library, which therefore is responsible for allocating and freeing it.
John Millikin at 2023-11-28 14:36:47
I think ThinSlice is a custom DST, just one with no/zero-size pointer metadata. Or at least, having a type customize its own size_of_val feels very much like custom DST territory. Maybe there’s a subset (with no metadata) of past custom DST proposals that could be accepted more easily than the fully general ones, but either way this seems unrelated to extern types.

(In the meantime, I’m happily using https://docs.rs/triomphe/latest/triomphe/struct.ThinArc.html on current stable Rust.)
Simon Sapin at 2023-11-28 19:11:46
OK, I'll file a separate RFC for the specific topic of !Sized thin pointers: https://github.com/rust-lang/rfcs/pull/3536
John Millikin at 2023-11-29 04:25:09
Any progress on this item? If Rust can solve this issue then a lot of low latency apps will be moved to Rust.
Ion at 2024-04-01 15:35:18
Breaking release of libc will be released soon. This will be very cool to implement extern types by then. Many libc types are good candidates for extern types, for example, FILE
Askar Safin at 2024-04-03 15:32:52
There's a summary of remaining issues here. The first step would probably be to update that summary; I am not sure how up-to-date it still is. For instance, size_of_val of extern types no longer returns 1, it now triggers an immediate abort. Similar for the alignment problem at the end of the summary that leads to making field access impossible; here it is on the playground. (I have no idea what the summary wants to do with that dyn Debug example, if T is sized then it can't be OpaqueTail and if T is not sized then the coercion to dyn does not work.)
Ralf Jung at 2024-04-03 15:43:25
As it happens, I've been thinking about this issue on and off for a while. I randomly wrote this blog post:

https://smallcultfollowing.com/babysteps/blog/2024/04/23/dynsized-unsized/

I admit this was based on my memory and maybe I'm out of date on the current thinking!

I suppose there is an alternative I didn't mention of just "let people call size_of_val on extern types", but I personally am no longer convinced that's a good option.
Niko Matsakis at 2024-04-23 21:43:40
https://smallcultfollowing.com/babysteps/blog/2024/04/23/dynsized-unsized/

Probably off topic for extern type, but is a thin CStr considered DynSized or Unsized or something in between? If we adapt rust-lang/rfcs#3396 there will be an additional MetaSized in the hierarchy[^1] which is implemented for [T] & dyn Trait but not CStr & extern type.

| type | size_of_val_raw | align_of_val_raw | |---|---|---| | Sized types | static constant | static constant | | [T] | dynamic, compute from metadata (safe) | static constant (align_of::<T>()) | | dyn Trait | dynamic, deref from metadata (unsafe) | dynamic, deref from metadata (unsafe) | | CStr | dynamic, must strlen() on pointee (slow & very unsafe) | static constant (1) | | extern type | unspecified | unspecified |

[^1]: Sized : MetaSized : DynSized(?) : Unsized(=Pointee?)
kennytm at 2024-04-24 00:07:42
https://smallcultfollowing.com/babysteps/blog/2024/04/23/dynsized-unsized/

Probably off topic for extern type, but is a thin CStr considered DynSized or Unsized or something in between?

a similar common case is a pointer to a C++ class with virtual methods, where it's a thin pointer and you can get the size/align by reading from the vtable, which the pointer to is stored in the main pointer's pointee
Jacob Lifshay at 2024-04-24 01:08:00
dyn Trait dynamic, deref from metadata (unsafe)

The safety invariant of raw wide pointers demands a valid vtable, so this is just as safe as size_of_val_raw on slices.

So, MetaSized would mean "metadata suffices to determine size (and even on raw pointers, the metadata safety invariant makes this safe)" whereas DynSized would need more arbitrary computation and so would not be safe on raw pointers in general.
Ralf Jung at 2024-04-24 06:03:29
[T] dynamic, compute from metadata (safe)

Actually this is not safe for raw pointers as it computing the size of the slice can overflow. I had forgotten about that.

So -- I don't quite see the motivation to distinguish MetaSized and DynSized; on raw pointers, computing the size is unsafe either way. But also I'm not sure if that's relevant for this issue.
Ralf Jung at 2024-04-24 06:12:28
@nikomatsakis thanks for sharing that blog post!

I find myself vehemently disagreeing with this point:

Just like T: Debug gives the function the extra capability of generating debug info, T: ?Sized feels to me like it gives the function an extra capability: the ability to be used on unsized types.

That's not giving the function an extra capability, but gives the caller an extra capability! Those are very different things. In fact they are the exact opposite of each other (in a precise, formal sense even). I think it would be extremely confusing to treat them like the same thing.

With unsafe we use the same keyword for a pair of duals (the act of adding a proof obligation and the act of discharging one). I think that is fairly widely regarded as a mistake as it is confusing. Now you are suggesting to use the same syntax T: Trait for a pair of duals (the act of adding a trait obligation and the act of removing one). I think that would be repeating the same mistake.

Ultimately, though, I’m not wedded to this idea, but I am interested in kicking off a discussion of how we can unblock extern types. I think by now we’ve no doubt covered the space pretty well and we should pick a direction and go for it (or else just give up on extern types).

There is a middle-ground, "min extern types": do not allow extern types (or types with extern type tail) to be used
- as arguments for size_of_val(_raw) and align_of_val(_raw)
- as types for the last field of a non-repr(transparent) struct (or, maybe allow declaring the type but do not allow place projections to the last field)
- to instantiate a generic parameter or an associated type
I don't know how much of the use-cases of extern type that would cover. *const ExternType would still be a normal type that can be used everywhere, including as instance of a generic type. The compiler's own List<T> type would also work in the variant that permits declaring structs with extern type tail but does not allow projecting to their last field.

The "do not use this to instantiate generic parameter" part of subtle, e.g. it also has to be taken into account during trait resolution: &ExternType does not implement Deref as we'd have to instantiate the generic parameter T in impl<T> Deref for &T to obtain that instance.

I assume this would probably be implemented internally by adding something like DynSized, but that trait would be entirely an internal implementation detail, we wouldn't have to figure out syntax to expose this to the user. Not sure if that's worth it...
Ralf Jung at 2024-04-24 06:34:26
I believe that preventing extern types to be used

to instantiate a generic parameter or an associated type

would be the kicker that makes them pretty unusable. To use them you have to pass them around behind references, and frequently you want to have owned references in some form which means you want SpecialBox<T> where T is one of the extern types in your C bindings. You could manually monomorphise those, but as you say you can't implement Deref on those types so using them will be a pain.

I will add that I haven't made much progress on rust-lang/rfcs#3396 partially because I've been busy with other things but partially because I'm less convinced that all the complexity in it is worth the benefits of having extern types, however I haven't seen any other good options.
Jack Rickard at 2024-04-24 10:08:17
You can't own an extern type, so what would that SpecialBox type look like? What would it do on drop?

In the past we have made good experience with "min" proposals, like "min const fn" or "min exhaustive patterns". "min extern types" wouldn't cover everything; the question is whether they cover sufficiently much that they'd be worth stabilizing as a stepping stone towards the full thing.
Ralf Jung at 2024-04-24 10:15:17
I'm imagining it looks something like:
```
struct SpecialBox<T: FfiDrop> {
  ptr: *mut T
}
impl<T: FfiDrop> Drop for SpecialBox<T> {
  fn drop(&mut self) {
    T::free(self.ptr)
  }
}
```
That setup would allow you to interact with C types which have their own free function, for example all OpenSSL types have dedicated free function, someone (in this thread) mentioned wanting a SwiftRc<T> that let them acccurately represent FFI types to Swift which are all internally reference counted.

I would love a "min" proposal for extern types, it feels too big for the functionality it's adding, however I've yet to be convinced that anyone would want to use something that prevented them from using them in generics. I guess you could probably get away with a bunch of hand-monomorphised code and making some things generic over *mut ExternType but it'll all be annoying to use.
Jack Rickard at 2024-04-24 10:34:32
You can't own an extern type, so what would that SpecialBox type look like? What would it do on drop?

Extern types represent an opaque handle to some resource, so there's typically some sort of free() function:
```
unsafe trait SpecialThingy {}
extern { fn free_special_thingy(ptr: *mut ()); }
struct SpecialBox<T: SpecialThingy>(*mut T);
impl<T: SpecialThingy> Drop for SpecialBox<T> {
  fn drop(&mut self) {
    unsafe { free_special_thingy(self.0.cast()) }
  }
}
```
Regarding the above blog post, I think it would be a mistake to focus overly much on sizes here. The defining feature of external types is that they're opaque identifiers, a sort of namespace for pointers. They're completely different from inhabited types, and the closest equivalent in Rust's type system today is a pointer to an uninhabited enum.

For example, it doesn't make sense to have type_of_val(&T) work on an extern type T, because external types don't have a known alignment and therefore can't be behind a reference. They can be behind a pointer, so they're a problem for size_of_val_raw(), but that function isn't stable anyway.

Another pattern that implies extern types need to work with generics is NonNull<T>, since Option<*const T> wouldn't have a niche optimization.

IMO an MVP for external types would make them pointer-only, and carve out the handling of "opaque tails" to a separate RFC.
John Millikin at 2024-04-24 10:45:45
If implementing Drop is the only thing SpecialBox does, the no-generics limitation does not seem that much of a problem to me. SpecialThingy could be a non-generic struct that implements Drop directly.

Disallowing NonNull<ExternType> however is unfortunate.
Simon Sapin at 2024-04-24 10:51:51
I'm not sure I agree with:

because external types don't have a known alignment and therefore can't be behind a reference

If you have a *const ExternType and know it points to a valid instance of the type for some lifetime 'a (and any other reference safety & validity requirements) why shouldn't you be able to convert it to a &ExternType?
Jack Rickard at 2024-04-24 10:52:53
If implementing Drop is the only thing SpecialBox does, the no-generics limitation does not seem that much of a problem to me. SpecialThingy could be a non-generic struct that implements Drop directly.

You definitely could, but you can't implement Deref, so you'd have to add a deref function, and you can't implement PartialEq or PartialOrd on the types directly (although you could implement them manually on all the pointer types you care about), and no blanket impls would apply to them.
Jack Rickard at 2024-04-24 10:56:38
If you have a *const ExternType and know it points to a valid instance of the type for some lifetime 'a (and any other reference safety & validity requirements) why shouldn't you be able to convert it to a &ExternType?

It's the "other reference safety & validity requirements" that's the problem. Consider the following C API:
```
struct UiWidget;
UiWidget* ui_button_new_with_label(const char *label);
void ui_widget_free(UiWidget *widget);
```
If you have a Rust type extern type Widget and you call the constructor function you can get a *mut Widget, but after that, what can you do with it? A Rust reference must be aligned and you don't know the alignment of UiWidget. A Rust reference must point to a valid object, and the pointer might have metadata stuffed into the high or low bits. There's nothing useful Rust can do with that value except pass it back into the original API functions.

If you were writing a binding then you'd need to define your own reference type, something like this:
```
pub struct ButtonMut<'a> {
  ptr: *mut Button,
  _ref: PhantomData<&'a ()>,
}
```
John Millikin at 2024-04-24 11:04:29
A Rust reference must be aligned and you don't know the alignment of UiWidget. A Rust reference must point to a valid object, and the pointer might have metadata stuffed into the high or low bits.

why not just say that rustc doesn't know what the alignment should be, so it just assumes that whatever unsafe code originally made the reference was correct and doesn't assume that their is any particular alignment...therefore having a reference to an extern type should be valid, rather than saying rust can't have references without known alignments even though it's perfectly capable of passing the address around correctly.

same thing for pointing to a valid object...there's nothing rustc can do with an extern type, so it just passes around the address and assumes that the code that made the reference is correct, this is similar to &().
Jacob Lifshay at 2024-04-24 11:37:58
Extern types represent an opaque handle to some resource, so there's typically some sort of free() function:

Note that with my proposed min_extern_type that you can still do something like
```
struct OwnedExternPtr(*mut ExternType);

impl Drop for OwnedExternPtr { ... }
```
OwnedExternPtr now is a normal sized type so it can be used in generics as usual.

Maybe you can even make that generic with a trait bound like *mut ExternType: FfiDrop.

Regarding the above blog post, I think it would be a mistake to focus overly much on sizes here. The defining feature of external types is that they're opaque identifiers, a sort of namespace for pointers. They're completely different from inhabited types, and the closest equivalent in Rust's type system today is a pointer to an uninhabited enum.

Uninhabited enums are something completely different from opaque handles. Uninhabited enums are the type system equivalent of the empty set. They represent the absence of a value. And they do have a known size. So I think this analogy is very wrong, dangerously so (we had soundness bugs in the past when people tried to model extern types with uninhabited enums).

IMO an MVP for external types would make them pointer-only, and carve out the handling of "opaque tails" to a separate RFC.

The no-generics proposal is fairly close to that -- but yes this would be another option. It wouldn't be quite enough for the compiler's own usage of extern type to model a "thin unsized list", but arguably that usage is a hack.

Disallowing NonNull<ExternType> however is unfortunate.

We could in principle do some lang item magic to allow extern types specifically for NonNull...

why not just say that rustc doesn't know what the alignment should be, so it just assumes that whatever unsafe code originally made the reference was correct and doesn't assume that their is any particular alignment...therefore having a reference to an extern type should be valid, rather than saying rust can't have references without known alignments even though it's perfectly capable of passing the address around correctly.

Indeed, that's exactly what would happen. There's no fundamental issue from the language definition side with references to extern types. In fact Miri already supports them.
Ralf Jung at 2024-04-24 11:52:55
as I previously proposed (sample usage), Box<T, A> can be changed to have impl Drop for Box<T, A> just call a trait instead of the usual drop_in_place and deallocation (that trait's impl for allocators does dropping and deallocation), this allows FFI code to just have that trait impl call whatever destroy/free/release FFI function you need.
Jacob Lifshay at 2024-04-24 12:00:35
as I previously proposed (sample usage), Box<T, A> can be changed to have impl Drop for Box<T, A> just call a trait instead of the usual drop_in_place and deallocation (that trait's impl for allocators does dropping and deallocation), this allows FFI code to just have that trait impl call whatever destroy/free/release FFI function you need.

That sounds like what you actually want is the ability to write impl Drop for ExternType.
Ralf Jung at 2024-04-24 12:09:07
That sounds like what you actually want is the ability to write impl Drop for ExternType.

that seems kinda problematic, since you can't actually pass a extern type around by value, references/pointers aren't a good substitute for Box. Also, the idea for having a trait for dropping boxes is also quite useful for automatic object pooling and other things that aren't related to extern types.
Jacob Lifshay at 2024-04-24 12:17:50

What would be the benefits of min_extern_types over:

mod hide_the_internals {
  struct ExternType;
  pub struct OwnedExternType(*mut ExternType);
  pub struct ExternTypeRefMut<'a>(*mut ExternType, PhantomData<&'a mut ExternType>);
  pub struct ExternTypeRef<'a>(*const ExternType, PhantomData<&'a ExternType>);
}

Given all the limitations to extern types then I'm not clear on how having at all would be better than just hiding a unit struct.

Jack Rickard at 2024-04-24 12:25:22

That's a fair question. Your proposal completely hides the extern type from the outside so it seems basically equivalent to the "only provide pointer types" variant of min-extern-type. My proposed variant would go a bit further and accommodate some of the things the compiler does with extern types. But the compiler actually implements a bunch of traits for structs with extern type tail as well, which probably means it relies on instantiating generic types with such an extern-type-tail struct.
Ralf Jung at 2024-04-24 12:37:52
A Rust reference must be aligned and you don't know the alignment of UiWidget. A Rust reference must point to a valid object, and the pointer might have metadata stuffed into the high or low bits.

why not just say that rustc doesn't know what the alignment should be, so it just assumes that whatever unsafe code originally made the reference was correct and doesn't assume that their is any particular alignment...therefore having a reference to an extern type should be valid, rather than saying rust can't have references without known alignments even though it's perfectly capable of passing the address around correctly.

Passing around the address (without additional semantics) is what *mut T (or NonNull<T>, or usize) is for. A usize, *mut (), *mut T, NonNull<T>, and &mut () all have the same representation in the registers, but rustc treats them differently.

Constructing a reference that is misaligned or dangling is undefined behavior, and (except in extremely narrow circumstances) there's no way to know whether a C opaque pointer would meet Rust's requirements. Could your FFI code handle the case where the call returns structs with different alignments at runtime?

@RalfJung I think the compiler's use of extern types to implement thin-pointer DSTs is probably unhelpful -- it continues to tangle up the two completely separate use cases:
1. Types defined in an external language and used to represent C opaque pointers. This is what the original RFC covers in the summary and motivation sections, and the only thing those types need to do is be pointed to without being dereferenceable. These types should not be usable with size_of_val() or align_of_val().
2. Types representing a memory region of dynamic size that is prefixed with some fixed-layout header, as used in RawList and discussed in RFC 3536. These types have known alignments, have sizes that can be peek'd, and are defined within Rust code (i.e. are not "external").
It's unfortunate that the original RFC enabled both patterns in such a way that the two features became conjoined, and I think the most likely path forward involves separating them.
John Millikin at 2024-04-24 12:46:11
@Skepfyr That design is one of the common ways to implement external types in stable, and it's mentioned in the RFC. The main downside is that code with access to struct ExternType; can accidentally use it "as a structure" -- it's not a fatal drawback, but it's a bit unfortunate.

It's possible that the subset of extern type that could be an MVP is essentially just the existing ZST structure pattern plus a bit of extra type-checking to enforce opacity.
John Millikin at 2024-04-24 12:52:53
Constructing a reference that is misaligned or dangling is undefined behavior, and (except in extremely narrow circumstances) there's no way to know whether a C opaque pointer would meet Rust's requirements. Could your FFI code handle the case where the call returns structs with different alignments at runtime?

Rust's requirements for extern types would be to not require alignment since we don't know it. Same for data validity. There's really no problem here. Undefined Behavior arises when the code violates the assumptions made by the compiler, but here the compiler cannot make any assumptions, and therefore the code cannot violate them. This is, in fact, already implemented correctly both in codegen and in Miri.

I think the compiler's use of extern types to implement thin-pointer DSTs is probably unhelpful -- it continues to tangle up the two completely separate use cases:

I tend to agree, what the compiler does here is a hack.

The main downside is that code with access to struct ExternType; can accidentally use it "as a structure" -- it's not a fatal drawback, but it's a bit unfortunate.

That's impossible in the design proposed by @Skepfyr as that struct is private to the module.
Ralf Jung at 2024-04-24 13:03:52
Rust's requirements for extern types would be to not require alignment since we don't know it. Same for data validity. There's really no problem here. Undefined Behavior arises when the code violates the assumptions made by the compiler, but here the compiler cannot make any assumptions, and therefore the code cannot violate them. This is, in fact, already implemented correctly both in codegen and in Miri.

I'm not trying to be snarky when I write this response, so please read literally: if extern types are compatible with Rust's reference semantics and the compiler already correctly emits code for them, then (1) why does size_of_val() cause a runtime panic and (2) why is stabilization blocked on incorrect handling of Box<T>?

It seems to me that the main blocker for extern type is that <T: ?Sized> makes assumptions about references that are guaranteed by the current semantics, and are broken by extern types. Given that extern types as described in the RFC could be restricted to pointers without breaking their main use case, and doing so would also fix the remaining known blockers, it's worth considering that as an approach.

That's impossible in the design proposed by @Skepfyr as that struct is private to the module.
```
mod hide_the_internals {
  struct ExternType;
  pub struct OwnedExternType(*mut ExternType);
  pub struct ExternTypeRefMut<'a>(*mut ExternType, PhantomData<&'a mut ExternType>);
  pub struct ExternTypeRef<'a>(*const ExternType, PhantomData<&'a ExternType>);

  fn do_something(x: &ExternType) { ... }
}
```
I know it's not the most critical failure mode, but considering that solving that specific drawback is the motivation for the RFC that this tracking issue exists for, I think it's worth at least keeping in mind.
John Millikin at 2024-04-24 13:15:15
why does size_of_val() cause a runtime panic

Because that's the most correct least dangerous semantics.

I was only talking about the runtime semantics. The type system clearly needs work. I never claimed the feature is ready.

We were talking about the question of alignment requirements of references, why are you now deflecting by talking about size_of_val? Your claim, as I understood it, was specifically that there's a problem defining the validity invariant for references (which typically involves alignment) as we don't know the alignment of these types. Two people now told you that this is in fact not a problem as we can just make the validity invariant not require any particular alignment for extern types.

why is stabilization blocked on incorrect handling of Box<T>?

I don't know what you are referring to here. Stabilization is blocked on the issue that the Rust type system does not understand the concept of a type that does not have a dynamically computable size.

mod hide_the_internals {

If you change other people's working code then it may not work any more. Not sure which point you are trying to make here. Usually when we evaluate the correctness of other people's code we only consider what can be done by calling that code as-if it was in another crate; we can't just access private fields or private types. It's trivial to break basically everything if you allow yourself to access private fields and private types.

But if we have e.g. a macro that generates the mod hide_the_internals then it won't be possible to do that.
Ralf Jung at 2024-04-24 13:39:08
We were talking about the question of alignment requirements of references, why are you now deflecting by talking about size_of_val?

My point is that size_of_val() (and align_of_val()) currently accept references to external types, but crash because those operations do not make sense for external types. When C code returns an opaque pointer, that pointer cannot be assumed to meet the requirements of Rust references that existing APIs depend on, so creating a reference to such a type should be considered undefined behavior.

A practical way to solve that problem is to prevent external types from being used as references, because if the only thing they're good for is their bit pattern then they're not actually references in the Rust sense.

why is stabilization blocked on incorrect handling of Box<T>?

I don't know what you are referring to here. Stabilization is blocked on the issue that the Rust type system does not understand the concept of a type that does not have a dynamically computable size.

I'm referring to https://github.com/rust-lang/rust/issues/115709, in which a Box<T> is allowed to be constructed for an extern type despite that being undefined behavior (because extern types do not have a size or alignment). And then the thing gets passed as a dereferenced T function parameter, which should also be undefined behavior because there's no way to move or copy such a type.

If you change other people's working code then it may not work any more. Not sure which point you are trying to make here.

But if we have a macro that generates the mod hide_the_internals then it won't be possible to do that.

I thought I was clear in my point, but in case not, the RFC for this tracking issue exactly describes the use of a ZST for extern types in the "motivation" section, and also describes why that is not the ideal solution.

Scrapping the extern type implementation and replacing it with ZSTs would be one possible approach, but I think it would be frustrating to the people who are subscribed to this tracking issue in the hopes of one day seeing this feature stabilize.
John Millikin at 2024-04-24 13:51:39

A practical way to solve that problem is to prevent external types from being used as references, because if the only thing they're good for is their bit pattern then they're not actually references in the Rust sense.

I have a use case that looks similar to this, that utilizes references to opaque types:

#![feature(extern_types)]

#[repr(C)]
#[derive(Debug)]
struct TextureSize {
    width: u32,
    height: u32,
}

extern "C" {
    type FfiSprite;
    
    fn ffi_sprite_create() -> *mut FfiSprite;
    fn ffi_sprite_get_texture(sprite: *const FfiSprite) -> *const Texture;
    
    type Texture;
    
    fn ffi_texture_get_size(texture: *const Texture) -> TextureSize;
}


struct Sprite<'texture> {
    handle: *mut FfiSprite,
    // The handle to the texture is on the FFI side
    _texture: core::marker::PhantomData<&'texture Texture>,
}

impl<'texture> Sprite<'texture> {
    fn new() -> Self { Self {
        handle: unsafe { ffi_sprite_create() },
        _texture: std::marker::PhantomData,
    }}
    fn texture(&self) -> &Texture {
        unsafe { &*ffi_sprite_get_texture(self.handle) }
    }
}

// Directly implement the opaque type
impl Texture {
    // Method by reference
    fn size(&self) -> TextureSize {
        unsafe { ffi_texture_get_size(self) }
    }
}

fn main() {
    let sprite = Sprite::new();
    let texture = sprite.texture();
    dbg!(texture.size());
}

Is this/should this be undefined behavior because you can't have a reference to an opaque type?

Here is a real life example in rust-sfml (using pseudo-opaque types similar to what bindgen generates, but the principle is the same): https://github.com/jeremyletang/rust-sfml/blob/3e275e06f408343c63133108e65009d1f4e6295c/src/graphics/texture.rs

Is this undefined behavior? In all my testing it worked fine.

crumblingstatue at 2024-04-24 15:00:55

I think there are usability problems with what I've described, reborrowing the Ref(Mut)? versions is painful, and it means that no-one else can name a *mut ExternType preventing them from using it directly (which I think might actually be a problem for -sys crates). I agree the fact that you can misuse it from within the module is bad, but we already have the module as the scope of unsafety so I think it's acceptable.

I agree we should consider getting the opaque tail stuff working a separate issue, it's incredibly painful due to alignment, and I haven't seen a working proposal yet.

@jmillikin Box<T> wouldn't be an issue though if we banned extern types from showing up in generic parameters, and it's not constructable anyway so it's of limited concern. On references, I think banning them from generics also solves this problem, as it prevents them from being passed to *_of_val, or any other API that could inspect the type behind the reference. A reference to an extern type does match Rust's existing semantics, if you assume it doesn't satisfy T: ?Sized, that's why rust-lang/rfcs#3396 works, it proposes to add another bound less restrictive than the current meaning of T: ?Sized.
Jack Rickard at 2024-04-24 15:03:21
My point is that size_of_val() (and align_of_val()) currently accept references to external types, but crash because those operations do not make sense for external types.

Yes. As I said, that's a type system issue. They crash in a safe way (a non-unwinding panic) so this is not unsound.

When C code returns an opaque pointer, that pointer cannot be assumed to meet the requirements of Rust references that existing APIs depend on, so creating a reference to such a type should be considered undefined behavior.

This does not follow, not at all. UB is a very big hammer we use to enable the compiler to optimize code better. "size_of_val would panic" is most definitely not an excuse for introducing UB. The consequences of accidental UB are way too severe to gratuitously use it here.

Ideally we will use the type system to prevent size_of_val on references or pointers to extern types. But if that doesn't work we can use a runtime check like we do right now, with a panic. That's still miles better than Undefined Behavior. There's no need to risk arbitrary memory corruption just because someone created a reference to an extern type.

I'm referring to https://github.com/rust-lang/rust/issues/115709, in which a Box<T> is allowed to be constructed for an extern type despite that being undefined behavior (because extern types do not have a size or alignment). And then the thing gets passed as a dereferenced T function parameter, which should also be undefined behavior because there's no way to move or copy such a type.

That issue isn't about Box, it is about unsized_fn_params, which (like size_of_val) assumes that all types are DynSized. There are indeed some gaps in the support for the combination of these two unstable features. Whatever type system solution is used to safeguard size_of_val against extern types should also be used to ensure unsized function parameters (and unsized locals) are dyn-sized.
Ralf Jung at 2024-04-24 15:12:17
@crumblingstatue

IIUC https://github.com/rust-lang/rust/issues/43467#issuecomment-2074173940 the proposed min_extern_type feature did not outright "ban" &Texture.

One big issue of &Texture was that align_of_val::<Texture>(_) is undefined, so a Rust program could not properly generate a valid &Texture reference[^1]. According to https://doc.rust-lang.org/reference/behavior-considered-undefined.html this was one of the UB scenarios:
- Producing an invalid value, … The following values are invalid (at their respective type):
  - A reference or Box<T> that is dangling, misaligned, or points to an invalid value.
[^1]: That said, the Texture type in the real-world code already produced an under-aligned structure: declare_opaque! results in an align-1 ZST, but the actual C++ definition of sf::Texture contains an Uint64 m_cacheId field meaning the actual alignment must be at least 8.

But given that Rust can never safely create a Texture/Box<Texture> value or deference a &Texture place this sentence can be relaxed or clarified for extern type (treating them effectively as align-1 behind a pointer, similar to the treatment of dyn Trait in LLVM IR).

@jmillikin

A practical way to solve that problem is to prevent external types from being used as references, because if the only thing they're good for is their bit pattern then they're not actually references in the Rust sense.

I don't see how banning &Texture is the practical solution. Using the min_extern_type restriction i.e.
1. Texture does not implement ?Sized, i.e. can't be substituted into any generic arguments or associated types
2. Texture can't be used as a struct member field (except #[repr(transparent)])
it can already prevent {size,align}_of_val{,_raw}::<Texture>() from being even instantiated, and Texture can't be used as a struct tail either, side-stepping the entire alignment/size question.

I'm not sure what "references in the Rust sense" encompasses. IMO at least calling methods given a &mut Texture / &Texture do make sense.
kennytm at 2024-04-24 16:34:22
@kennytm The problem with the proposed min_extern_types is the inability to use such a type as a generic parameter. It's sacrificing an important capability (extern type generics) to retain something unnecessary (extern type references).

Given the following sets of goals for an extern type T:
- Allow *const T and *mut T, which are required in FFI signatures.
- Allow NonNull<T> and Option<NonNull<T>>, for niche optimization.
- Forbid size_of_val::<T>(_) and align_of_val::<T>(_), as extern types don't have sizes or alignments.
- Forbid struct W(T), Option<T>, [T], and &[T], as a consequence of not having sizes or alignments.
... the proposal to forbid their use as references solves all four, and allowing references but forbidding generics would leave two unsolved.
John Millikin at 2024-04-25 00:07:04
- Forbid struct W(T), Option<T>, [T], and &[T], as a consequence of not having sizes or alignments.
the types Option<T>, [T], and [T; N] are forbidden anyway for extern types since extern types aren't Sized.
Jacob Lifshay at 2024-04-25 01:08:32
@jmillikin Forbidding &ExternType alone will not prevent calling <code>align_of_val<strong>_raw</strong></code> which takes a pointer, not a reference.

Meanwhile forbidding reference but not generic meant the following code is still possible:
```
// crate A
pub struct Tailed<T: ?Sized> {
    a: usize,
    b: T,
}

// crate B
extern { type Texture; }
type TailedTexture = crate_a::Tailed<Texture>;
```
The NonNull<T> case can be supported once we have sorted out the Sized hierarchy issue. ?Sized could become an alias of DynSized and the bound of NonNull<T> will be changed from ?Sized to Unsized.
kennytm at 2024-04-25 02:29:27
@kennytm I don't think size_of_val_raw() and align_of_val_raw() matter that much, because they're not stable and therefore their signatures can change to have new trait bounds or whatever.

I don't understand why the code you provided would be valid for extern type -- they don't have a size or alignment, so they can't be struct fields.
John Millikin at 2024-04-25 02:40:33
You can't forbid references without forbidding extern types in generics. Otherwise it's always possible to use generics to bypass the restriction.
Ralf Jung at 2024-04-25 05:47:41
I don't understand why the code you provided would be valid for extern type -- they don't have a size or alignment, so they can't be struct fields.

I think it should be valid for extern type, since that allows you to declare a struct that you can make references to that is unsized and is a header for whatever unknown larger struct is actually in memory. actually accessing the extern type field shouldn't be valid, unless the extern type has a known alignment, e.g.:
```
extern {
    #[repr(align(4))]
    type ExternTypeWithKnownAlignment;
}
```
Jacob Lifshay at 2024-04-25 05:50:00
@jmillikin

I don't think size_of_val_raw() and align_of_val_raw() matter that much matter that much, because they're not stable and therefore their signatures can change to have new trait bounds or whatever.

What really matters is the intrinsic they are being forwarded to (min_align_of_val) are actually used by the language to compute alignment of a struct tail field.

I don't understand why the code you provided would be valid for extern type -- they don't have a size or alignment, so they can't be struct fields.
1. If it is forbidden from "crate A", how could it know that T is an extern type and has unspecified size and alignment?
2. If it is forbidden from "crate B", how could it know that T is being used as a struct tail (given that all members of Tailed are private)?
The min_extern_type restriction is to forbid such code from "crate B" (since from "crate A"'s perspective, Tailed<[u8]> and Tailed<dyn Trait> must continue to work) by declaring ExternType does not implement ?Sized, simple.

@programmerjake

unless the extern type has a known alignment, e.g.:

While it sounds good in theory I doubt in practice if there is any actual scenario where the FFI type specified the exact alignment yet the size and all fields are implementation details. Usually you either have none or all of these 3 details.

Like for the <cstdio> type FILE*, either you just use whatever returned by fopen() and don't care about its alignment/size/fields, or you'll actually use the complete definition like
```
#[repr(C)]
struct FILE {
    _flags: c_int,
    _IO_read_ptr: *mut c_char,
    ...
}
```
I see no case that you specifically only want to know align_of::<FILE>() == align_of::<usize>() and nothing else.

The only exception might be dynamic-sized array, like an array tail like xxxx_t tail[0], or C-style string const char* / const wchar_t*, but I think they are a different kind of Custom DST that shouldn't be hacked through extern type.
kennytm at 2024-04-25 13:08:17
@RalfJung

That's not giving the function an extra capability, but gives the caller an extra capability! Those are very different things. In fact they are the exact opposite of each other (in a precise, formal sense even). I think it would be extremely confusing to treat them like the same thing.

I agree they are not the same (and my blog post said that). However, my point was that I do not think that understanding this distinction is as important as you do, at least at first, and I think that by the time it is, people will be able to handle it. In contrast, the ? operator puts that price up front -- it marks this bound as a very different thing -- in a way that I think can intimidate readers and isn't especially necessary for them to understand early on. Now, if ?Sized could scale to all the use cases we need, then maybe that'd be one thing, but it seems to me that it cannot (though maybe there are proposals I've missed, I feel like I have to do a bit of a review, I didn't realize how much conversation had been going on).

Let me explain this a different way. Early on in their Learning Rust journey, I think the main way that people will interact with Unsized or DynSized bounds is when calling library APIs -- i.e., reading them in documentation, not authoring them. They might not realize that Unsized or DynSized is in fact saying that this function accepts a broader set of types (unlike Debug, which narrows it). But so long as they can tell that their types implement those traits, do they care? I don't think so.

Maybe they then go to author an impl that needs T: Unsized to be maximally applicable. My experience in authoring such impls is that I often forget the T: ?Sized initially. I write the code. It type checks. Then I go to use it, and I have to add some bounds. This is ... surprisingly similar to what happens with ordinary bounds. Of course the "compiler complains, add bound, recompile" cycle is happening when I use the impl, not when I author the impl, but the overall interplay feels similar. Especially because there is a kind of reverse thing that sometimes plays out, where I add bounds to write the code, then find out that the types I want to use don't support those bounds (sometimes for good reason), and I have to go back and refactor the code not to require so many bounds.

All that said, I think we would really benefit from doing some experimentation here. It's often hard to judge how something will feel after using it for a while. My hunch is that Unsized bounds will feel pretty natural, but I'd feel better if we had a nightly implementation to play with. And, as I said, I'd like to review the other ideas. I'm open to something better. But mostly I don't want us to delay forever and ever.
Niko Matsakis at 2024-04-26 14:59:29
(Has anybody written lints to detect when a type parameter could have been ?Sized?)
Niko Matsakis at 2024-04-26 15:00:40
I have a concern that drop glue for extern types should not be instantiated, meanwhile it currently resolves with empty drop_in_place. Same goes for needs_drop as it should always return true instead of false, just like with dyn Trait.

P.S. It should probably become some sort of a post-monomorphization (or linker) error, since this feature usually relies on the linker finding symbols anyway. I could try implement that. Perhaps this could also be done to size_of_val? So I would say calling drop_in_place most probably indicates a bug in the code. I wonder what crater run would say about that.
Daria Sukhonina at 2024-04-27 13:03:32
@zetanumbers Could you give a code example of what your talking about? I can't imagine a scenario where drop glue would be emitted for an extern types, because you can't ever have an owned extern type in Rust. Similarly, I'd expect needs_dropto return false, because you can't implement Drop for extern types (which means I'd expect drop_in_place to do nothing).
Jack Rickard at 2024-04-27 15:28:13
@zetanumbers Could you give a code example of what your talking about? I can't imagine a scenario where drop glue would be emitted for an extern types, because you can't ever have an owned extern type in Rust. Similarly, I'd expect needs_dropto return false, because you can't implement Drop for extern types (which means I'd expect drop_in_place to do nothing).

As you have said, so that drop_in_place::<ForeignType> is instantiated with empty drop glue.

I'd expect needs_drop to return false, because you can't implement Drop for extern types

You cannot implement Drop for a number of types including dyn Trait for which needs_drop would return true.

I understand you don't see the difference for yourself, I am just looking which behavior would people prefer if they care.
Daria Sukhonina at 2024-04-27 15:56:02
You cannot implement Drop for a number of types including dyn Trait for which needs_drop would return true.

dyn Trait forwards Drop to the actual underlying type. So not being able to implement Drop here is an entirely different matter.

Extern types do not have any implicit drop glue, so there's no good reason for needs_drop to return true here. needs_drop() == false just means that the drop shim does nothing, and indeed drop_in_place does nothing for these types, so all works out.

Maybe we want to allow impl Drop for ExternType in the future, but that's a separate discussion and the current behavior is forward-compatible with that.
Ralf Jung at 2024-04-27 16:17:47
I am worried if such behavior may affect support for unforgettable types. Extern types seem like very much raw feature for unsafe code only, which is not inherently bad, but I wonder if usage from safe code is planned.

Also to note if unforgettable types are indeed only meaningful when they have a lifetime, as I believe it would be without accounting other hypothetical features, then extern type should probably not be allowed to have lifetime arguments.
Daria Sukhonina at 2024-04-27 16:30:50
I don't see how extern types are particularly interesting here. In terms of needs_drop and drop_in_place, I think extern type Foo acts exactly the same as struct Foo. Currently, you can't implement Drop for an extern type but as Ralf says it's not clear that restriction will stay forever, and if it does drop it will continue to look like struct Foo: needs_drop will return true and drop_in_place will call the drop function.
Jack Rickard at 2024-04-27 17:11:21
Since no one seems to have asked this question yet:

The current "idiom" for opaque types is having a _opaque: [u8; 0] field. This approach supports having PhantomData fields for supporting things like adding lifetime annotations on the type. I make very heavy use of this in rust-sfml:
```
#[repr(C)]
pub struct Shader<'texture> {
    _opaque: [u8; 0],
    // Keeps track of the lifetime of the texture the Shader borrows,
    // but of which Rust has no idea otherwise, since
    // the field that "borrows" the texture is on the C side
    _texture: PhantomData<&'texture Texture>,
}
```
How would I go about this using extern types? If extern types cannot support this, then it's not really a universal replacement for the _opaque: [u8; 0] field "workaround".
crumblingstatue at 2024-10-24 01:50:43

How would I go about this using extern types?

I would assume by doing something like this:

extern {
    type ShaderExtTy;
}

#[repr(transparent)]
pub struct Shader<'texture> {
    inner: ShaderExtTy,
    _phantom: PhantomData<&'texture Texture>,
}

Jacob Lifshay at 2024-10-24 05:09:42

@programmerjake Perhaps... Although it would have to be

#[repr(transparent)]
pub struct Shader<'texture> {
    _phantom: std::marker::PhantomData<&'texture Texture>,
    inner: ShaderExtTy,
}

Since only the last field can be unsized.

crumblingstatue at 2024-10-24 11:48:22

The third unresolved question shouldn't be a problem anymore, since LLVM now uses opaque pointer types.
CatsAreFluffy at 2024-12-02 18:21:41
The third unresolved question shouldn't be a problem anymore, since LLVM now uses opaque pointer types.

it is probably still a problem for rustc_codegen_gcc since libgccjit has separate pointer types for each pointee type.
Jacob Lifshay at 2024-12-02 19:15:44
Hopefully, GCC isn't quite so sensitive about which pointee type needs to be picked. (I forgot why it was important for LLVM for void*.) But anyway given that the GCC backend is not complete yet, I don't think we should block this RFC on that -- it's "just" one more item on the list of things whose implementation is still incomplete.
Ralf Jung at 2024-12-02 19:20:33

Tracking issue for RFC 1861: Extern types

Should we allow generic lifetime and type parameters on extern types?

`std::ptr::null`/`null_mut`

`extern type` cannot support `size_of_val` and `align_of_val`.

Should you be able to specify the alignment of an `extern type`?

Is this feature suitable for implementing `c_void`?

Are all usecases covered?

Rust

C

Should we allow generic lifetime and type parameters on extern types?

What LLVM type should be used?

std::ptr::null/null_mut

`size_of_val` (and partially `align_of_val`)

Auto traits are not implemented by default.

Trait objects cannot be built from extern types.

type class of the extern type

Unknown alignment of extern types

Where next?

Should we allow generic lifetime and type parameters on extern types?

std::ptr::null/null_mut

extern type cannot support size_of_val and align_of_val.

Should you be able to specify the alignment of an extern type?

Is this feature suitable for implementing c_void?

Are all usecases covered?

Rust

C

Should we allow generic lifetime and type parameters on extern types?

What LLVM type should be used?

std::ptr::null/null_mut

size_of_val (and partially align_of_val)

Auto traits are not implemented by default.

Trait objects cannot be built from extern types.

type class of the extern type

Unknown alignment of extern types

Where next?

`std::ptr::null`/`null_mut`

`extern type` cannot support `size_of_val` and `align_of_val`.

Should you be able to specify the alignment of an `extern type`?

Is this feature suitable for implementing `c_void`?

`size_of_val` (and partially `align_of_val`)