Tracking issue: declarative macros 2.0

Opened by Nick Cameron at 2025-02-07 11:41:18

Tracking issue for declarative macros 2.0 (aka macro aka decl_macro aka macros-by-example).

RFC: https://github.com/rust-lang/rfcs/blob/master/text/1584-macros.md

RFC PR: https://github.com/rust-lang/rfcs/pull/1584

cc @rust-lang/compiler

Tasks

[ ] Complete the draft hygiene prototype and RFC.
- [x] Hygiene for items, explicit imports, lexical scopes, and module scopes.
- [x] Hygiene for globs and trait methods (w.r.t. extension trait candidates in scope for a method call).
- [x] Hygiene for type directed name resolutions (i.e. methods, associated types, and fields).
- [x] Support macro in blocks as well as modules.
- [x] Implement inter-crate hygiene, except for nested macros.
- [ ] Implement "hygiene bending" for when users want a name from macro def to "escape".
- [ ] Implement inter-crate hygiene for nested macros (pending macro def encoding).
- [ ] Make private_in_public hygienic.
- [ ] Make unsafe and lints hygienic (if appropriate).
[x] Add variant ast::ItemKind::MacroDef for macro_rules! items and macro items (PR #40220).
[ ] Encode macros in the crate metadata using TokenStream, not String.
[x] Fix #30476 (private_in_public details).
[x] Fix span issues #30506 and #39450 (PR #40597).
[x] Land macro behind a feature gate (PR #40847).
[ ] Future-proof matchers (e.g. $e:expr) by employing a simpler, more general grammar.
[ ] Allow fragments (e.g. $e where $e:expr) to be parsed in more contexts (c.f. #26361).
[ ] Decide whether we want macro invocations in identifier positions and/or eager expansion.
[ ] Remove $:meta matcher (#49629)

Tracking issue for declarative macros 2.0 (aka macro aka decl_macro aka macros-by-example).

RFC: https://github.com/rust-lang/rfcs/blob/master/text/1584-macros.md

RFC PR: https://github.com/rust-lang/rfcs/pull/1584

cc @rust-lang/compiler

About tracking issues

Tracking issues are used to record the overall progress of implementation. They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions. A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature. Instead, open a dedicated issue for the specific matter and add the relevant feature gate label. Discussion comments will get marked as off-topic or deleted. Repeated discussions on the tracking issue may lead to the tracking issue getting locked.

Tasks
- [ ] Complete the draft hygiene prototype and RFC.
  - [x] Hygiene for items, explicit imports, lexical scopes, and module scopes.
  - [x] Hygiene for globs and trait methods (w.r.t. extension trait candidates in scope for a method call).
  - [x] Hygiene for type directed name resolutions (i.e. methods, associated types, and fields).
  - [x] Support macro in blocks as well as modules.
  - [x] Implement inter-crate hygiene, except for nested macros.
  - [ ] Implement "hygiene bending" for when users want a name from macro def to "escape".
    - Examples: https://github.com/rust-lang/rust/issues/91249, https://github.com/rust-lang/rust/issues/46342
  - [ ] Implement inter-crate hygiene for nested macros (pending macro def encoding).
  - [ ] Make private_in_public hygienic?
  - [ ] Make unsafe and lints hygienic (if appropriate)?
- [x] Add variant ast::ItemKind::MacroDef for macro_rules! items and macro items (PR #40220).
- [x] Encode macros in the crate metadata using TokenStream, not String.
- [x] Fix #30476 (private_in_public details).
- [x] Fix span issues #30506 and #39450 (PR #40597).
- [x] Land macro behind a feature gate (PR #40847).
- [ ] Future-proof matchers (e.g. $e:expr) by employing a simpler, more general grammar.
- [ ] Allow fragments (e.g. $e where $e:expr) to be parsed in more contexts (c.f. #26361).
- [ ] Decide whether we want macro invocations in identifier positions and/or eager expansion.
- [x] ~Remove $:meta matcher (#49629)~
Potentially blocking issues:
- https://github.com/rust-lang/rust/issues/71614
Vadim Petrochenkov at 2020-04-17 20:30:13
@nrc Typo in Issue Name: "issuse"
Christopher Serr at 2017-01-30 20:37:31
Tasks

(dtolnay edit: moved the checklist up to the OP)
Jeffrey Seyfried at 2017-02-07 01:05:01
We need to RFC a whole bunch of stuff here. In particular, I would like to propose some new syntax for declaring macros and we should RFC the changes to matchers.
Nick Cameron at 2017-02-07 04:55:52

Can the hygiene RFC mention pattern hygiene? This in particular scares me:

// Unsuspecting user's code
#[allow(non_camel_case_types)]
struct i(i64);

macro_rules! ignorant_macro {
    () => {
        let i = 0;
        println!("{}", i);
    };
}

fn main() {
    // oh no!
    ignorant_macro!();
}

Tim at 2017-03-10 20:34:46

@tikue I'm not sure patterns need special treatment with respect to hygiene.

For example, on the hygiene prototype,

#[allow(non_camel_case_types)]
struct i(i64);

macro ignorant_macro() {
    let i = 0; // ERROR: let bindings cannot shadow tuple structs
    println!("{}", i); 
}

fn main() {
    ignorant_macro!(); // NOTE: in this macro invocation
}

This makes sense to me since let i = 0; is shadowing a tuple struct. In particular, if let i = 0; were removed then the following use of i would resolve to the tuple struct, no matter where ignorant_macro is used.

Note the symmetry to this example:

#[allow(non_camel_case_types)]
struct i(i64);

fn ignorant_fn() {
    let i = 0; // ERROR: let bindings cannot shadow tuple structs
    println!("{}", i); 
}

If the tuple struct i isn't in scope at macro ignorant_macro() { ... }, then the let i = 0; does not shadow it and there is no error. For example, the following compiles on the hygiene prototype:

mod foo {
    pub macro ignorant_macro() {
        let i = 0;
        println!("{}", i); // Without `let i = 0;`, there would be no `i` in scope here.
    }
}

// Unsuspecting user's code
#[allow(non_camel_case_types)]
struct i(i64);

fn main() {
    foo::ignorant_macro!();
}

Jeffrey Seyfried at 2017-03-13 00:14:45

Has there been any progress on Macros 2.0 lately?
Alexander Regueiro at 2017-08-17 01:23:04
Note for those who haven't seen yet: macros 2.0 is apparently slated to be stable later this year, according to the proposed roadmap (rust-lang/rfcs#2314)...

On the one hand, that's pretty exciting :tada:!

On the other hand, I was really surprised that the feature is so close to being done and so little is known about it by the broader community... The RFC is really vague. The unstable book only has a link to this issue. This issue(s) in the issue tracker mostly have detailed technical discussions. And I can't find that much info anywhere about what's changed or how stuff works.

I don't mean to complain, and I really truly appreciate all the hard work by those implementing, but I would also appreciate more transparency on this.
mark at 2018-01-30 17:13:53
@marcbowes Yeah, I'm kind of worried too. It seems like there's quite a lot of work left for stabilisation this year. I offered to work on opt-out hygiene for identifiers myself, but have received no response yet... Some greater transparency would be nice, as you say.
Alexander Regueiro at 2018-01-30 17:22:32
Macros 2.0 are not even in the RFC stage yet - the @jseyfried's RFC linked in the issue was never submitted, there's only very high level RFC 1584. The implementation is purely experimental and mostly unspecified, and large part of it (not related to hygiene) is reused from macro_rules! without fixing problems that macros 2.0 were supposed to fix.

Some greater transparency would be nice

The problem is that no work happen right now, so there's nothing to reveal :( @nrc is busy, @jseyfried is busy, I worked on macros 2.0 a bit and would really like to dig into them more, but I'm busy too, unfortunately. We need someone to adopt the feature, support and extend it, and gain expert-level knowledge of it, otherwise we will end up breaking hygiene and syntax details left and right after stabilization.
Vadim Petrochenkov at 2018-01-30 17:38:37
@petrochenkov Ah, fair enough. I mean, that's a shame, but it makes sense at least. I guess this is an open call to anyone who might be able to take ownership of this feature, since no one comes to mind? I could still have a go at a little sub-feature, but I'm certainly in no position to take ownership of this.
Alexander Regueiro at 2018-01-30 18:31:29
@alexreg

I could still have a go at a little sub-feature

Yes, please do! Hygiene opt-out is one of the primary missing parts and implementing it would be useful in any case.

IIRC, two questions will need to be decided on during implementation:
- Syntax for "unhygienic" identifiers (@jseyfried tentatively suggested #ident in https://github.com/rust-lang/rust/pull/40847)
- What exactly hygienic context the identifier introduced with #ident in a macro m will have - context of m's invocation? context after expanding all macros ("no hygiene")? something else?
Vadim Petrochenkov at 2018-01-30 21:01:54
Syntax for "unhygienic" identifiers (@jseyfried tentatively suggested #ident in #40847)

Yeah, this was the plan. :-)

What exactly hygienic context the identifier introduced with #ident in a macro m will have - context of m's invocation? context after expanding all macros ("no hygiene")? something else?

What's your inclination? I'm leaning towards the context of m's invocation, but curious to hear your thoughts...
Alexander Regueiro at 2018-01-30 21:58:02
@alexreg

I'm leaning towards the context of m's invocation

I think this is what should be implemented first, just because this is a more conservative alternative.

On the other hand, it makes writing internal helper macros harder, e.g.
```
macro m_helper() {
    struct #S;
}

macro m() {
    m_helper!(); // `S` has this context
    let s = S; // OK
}

fn main() {
    m!(); // `S` is not accessible here
    let s = S; // ERROR
}
```
Vadim Petrochenkov at 2018-01-30 22:31:34
@petrochenkov Yeah, good point. I wonder if adding syntax like m_helper!#() (reuse the current context for the invocation) would help with that, or if there's a more elegant way that covers all use cases without too much pain?
Alexander Regueiro at 2018-01-30 22:42:04
At that point you might start providing utility macros like lift!(m_helper!(...)) (move all tokens generated by m_helper!(...) up one context, essentially pretending foo! was called by main in the example above):
```
macro m_helper() {
    struct #S;
    struct T;
}

macro m() {
    lift!(m_helper!()); // Sets caller context of `m_helper!` to caller context of `m!`.
    let s = S; // Not OK: `S` is in callers context.
    let t = T; // Not OK: `T` is in `m_helper!` context.
}

fn main() {
    m!();
    let s = S; // OK: `S` is in `main` context.
}
```
As a bonus, I think this would be possible to implement with some minor extensions to the proc macro API (mostly getting and setting the parent of an arbitrary scope/span, rather than only having access to the def and call site scopes).
Edward Pierzchalski at 2018-01-31 00:07:20
@pierzchalski Yeah, that's not a bad idea at all. Thoughts, @jseyfried / @petrochenkov?
Alexander Regueiro at 2018-02-03 03:24:13
@alexreg Actually, I just realised this is the use-case I was looking for for call_from in the proc macro RFC I put up.
Edward Pierzchalski at 2018-02-03 03:44:43
@pierzchalski I'll give that RFC a read tomorrow. Anyway, I certainly won't be including this lift macro or similar into my RFC; only the basic hygiene opt-out syntax. If the lift macro could go in another crate eventually, that would be ideal.

Incidentally, you seem to have a good knowledge of macro expansion and hygiene. Could I tempt you to contribute to https://github.com/rust-lang-nursery/rustc-guide/issues/15? :-)
Alexander Regueiro at 2018-02-03 04:25:19
@alexreg Unfortunately my understanding of expansion and hygiene is a bit abstract - you'll notice the reference-level explanation in the RFC was rather thin! But if the RFC is accepted and I end up implementing it then I'd definitely like to document what I discover along the way.
Edward Pierzchalski at 2018-02-05 14:58:02
I filed #49629 to consider dropping support for #[$m:meta] in favor of #[$($meta:tt)*] now that attributes are allowed to contain an arbitrary token stream.
David Tolnay at 2018-04-03 17:02:01
Stylistic question: why is it macro foo() and not macro foo!(), like the way it's called?
Claudia Meadows at 2018-08-30 10:43:34
I believe the question mark isn't considered part of the name.
mark at 2018-08-30 12:30:58
@mark-i-m You mean exclamation point, not question mark, right?

Also, in the docs, it usually refers to macros including the exclamation point as if it were part of the name, such as println! or vec!. As a concrete example, here's one page in the book that uses it exclusively.
Claudia Meadows at 2018-08-31 02:45:36
Oh, yes, I meant exclamation mark. I might be wrong, but I believe the compiler itself doesn't count the exclamation mark as party of the ident.
mark at 2018-08-31 03:12:46
@mark-i-m I would expect the compiler not to, but I'm speaking of the language itself, not the implementation. I'm suggesting matching what people think, not what computers process.
Claudia Meadows at 2018-08-31 03:15:21
That said, I don't see any reason why we cannot do something different from the internal representation.

If we did want to make ! part of the name, we would also want to do it in macro imports in the 2018 edition.
mark at 2018-08-31 03:17:47
@mark-i-m If you change how you do it in imports, you could even integrate them into use, to avoid the need to deal with separate syntax altogether:
```
use mod::foo::some_macro!;
use mod::foo::{some_fn, some_macro!};
```
Even as recently as a couple days ago, I had to google how to import a macro from a peer module, and this would make it a million times simpler. It also wouldn't require any special attribute or whatever - it'd just work.

Similarly, exporting macros could be simply pub macro foo!() or similar. Alternatively, because ! could serve as a delimiter, you could remove macro altogether and just do pub fn foo!($a: ident) { ... }, or you could reserve macro foo! for macros with multiple syntax rules (like vec!), and let fn foo! be for ones with only a single value (like how try! could be written if this RFC gets implemented.
Claudia Meadows at 2018-08-31 04:45:02
@isiahmeadows you can do use foo::bar::some_macro in the 2018 edition. It is part of the module/macro system changes being stabilized.

Long term IIUC, the plan is to get rid of the attributes altogether (currently, you still need them to export macros).

I strongly prefer not to conflate functions and macros. They are very different.
mark at 2018-09-03 15:05:16
use mod::foo::{some_fn, some_macro!};

The ! suffix is not technically necessary here, but since this may be the only thing pointing out that it is indeed a macro, seems like a good idea (if feasible and not too late — probably is too late).

But of course if foo and foo! are two distinct identifiers, they should not both be allowed to exist in the same scope.
Diggory Hardy at 2018-09-06 08:16:42
Macros live in another namespace. You can have a macro foo and a function foo and a type foo all coexisting.
mark at 2018-09-06 14:27:22
This makes me wonder. If I have a crate mycrate defining function foo and procedural macro foo, then if I do use mycrate::foo, which one is imported into scope? both? (I'm starting from the idea that some day in the hopefully near future the -derive crates and the not--derive crates will be merge-able)

Adding the ! as something to remove the ambiguity would likely help (in addition to being more explicit about what happens, which is always a good thing).

Assuming it's still time to do it, though… otherwise, add the ! suffix as a possibility and a warning to push using it?
Léo Gaspard at 2018-09-09 06:27:56
@Ekleog yes, both (along with potentially a module foo and type foo as well).
Nemo157 at 2018-09-09 07:27:41
If I have a crate mycrate defining function foo and procedural macro foo

You cannot do that today:
```
error: `proc-macro` crate types cannot export any items other than functions tagged with `#[proc_macro_derive]` currently
 --> src/lib.rs:1:1
  |
1 | pub fn some_public_function() {}
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error
```
However in an hypothetical future Rust where that restriction is lifted somehow, or if a non-proc-macro crate re-exports a proc macro and also a function of the same name, then yes use will import both.
Simon Sapin at 2018-09-10 13:58:11
Should small proposals for Macros2.0 be brought up here or in a separate issue?
Sebastian Malton at 2018-10-12 15:02:08
What the status of this?
Eric Shimizu Karbstein at 2019-07-13 15:21:17
What the status of this?

Bugs in macros 2.0 are occasionally fixed, but the feature is not on the 2019 roadmap.
Vadim Petrochenkov at 2019-07-13 15:32:25
I'd like to clarify the rules about visibility and propose a mechanism for breaking hygiene in a controlled way.

I believe visibility should be resolved relative to where the macro is defined. For example, imagine a module like this:
```
fn foo<T>(arg: T) {...}
pub macro call_foo($arg:expr) => {
    foo($arg)
}
```
With the current rules, expanding call_foo! in a different module doesn't work because foo isn't public. It should be should be fine because foo is visible where call_foo! is defined.

As for breaking hygiene, I think some additional flexibility would be very useful, and I think some solution needs to be introduced at the same time as the new hygiene rules, because otherwise it becomes impossible to break hygiene in ways that macros can depend on now (e.g. when defining a new type). I propose a built-in macro whose argument is an identifier that will appear verbatim in the expansion of a macro where it's used. Let's call it verbatim! for now (although I don't think that's a great name for it). It could, for example, be used to define an "anaphoric" map whose argument is an implied closure with a fixed argument name:
```
pub macro map_it($iter:expr, $body:expr) {
    $iter.map(|verbatim!(it)| $body)
}
```
It could also be used to translate an old macro that defines a type in the current module:
```
// before:
macro_rules! define_foo {
    () => {
        struct Foo {...}
    }
}

// after:
pub macro define_foo() => {
    struct verbatim!(Foo) {...}
}
```
Special names like self and Self, which are treated as identifiers for the purpose of macro expansion, should be implicitly verbatim. With the current rules, this doesn't work because $body can't refer to self:
```
macro_rules! paranoid_method {
    ($name:ident, $body:expr) => {
        pub fn $name(&mut self) {
            self.verify_preconditions();
            $body;
            self.verify_postconditions();
        }
    }
}
```
Currently the only workaround is to pass self as an additional argument to the macro, which is pretty silly since passing any other identifier would produce a syntactically invalid expansion.

One last thing on my wishlist is to extend verbatim! to support multiple arguments, which are concatenated together to produce a new verbatim identifier. This could be used to do something like define a pair of related methods:
```
/// Defines a pair of conversion methods, `as_$name` and
/// `into_$name`, where `into_$name` consumes `self` and 
/// `as_$name` clones `self`.
macro conversions($name:ident, $ty:ty, $body:expr) => {
    pub fn verbatim!(into_, $name)(self) -> $ty { $body }
    pub fn verbatim!(as_, $name)(&self) -> $ty {
        self.clone().verbatim!(into_, $name)()
    }
}
```
John Williams at 2020-02-14 21:44:00
I'd like to clarify the rules about visibility and propose a mechanism for breaking hygiene in a controlled way.

I believe visibility should be resolved relative to where the macro is defined. For example, imagine a module like this:
```
fn foo<T>(arg: T) {...}
pub macro call_foo($arg:expr) => {
    foo($arg)
}
```
With the current rules, expanding call_foo! in a different module doesn't work because foo isn't public. It should be should be fine because foo is visible where call_foo! is defined.
Could you give an example of this? This seems to work: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=a3e26640f0fb170b9a5a620250155f1c

You might need an absolute path $crate::path::to::foo($arg), but I'm not sure if this applies to decl macros or only macro_rules.
Shane Pearman at 2020-02-14 21:57:56
It could, for example, be used to define an "anaphoric" map

I know it's just meant as an example, but anaphoric macros were a bad idea in Common Lisp, and I believe they're a bad idea here. The issue with them is that they don't scale: imagine that you end up in a situation where you have a nested usage of map_it. What does it refer to? Of course here it's easy enough to figure out in this contrived example, but at scale it causes ambiguity in the mind of the programmer.

verbatim!(it)

I would like to note that that requires macros to be expandable to identifiers. I don't believe they currently have that capacity, so that would need to be added as well. As for the name, I believe this is one instance where another cue from Lisps might help. While they use ~ (Closure) or , (CL) for interpolation, we could similarly use a sigil for identifier interpolation (leaving the default case to be verbatim instead). I'm not sure how that would interact with identifier concatenation though.
jjpe at 2020-02-14 22:04:49

Weather or not anaphoric macros are allowed, the ability to create new identifiers from ones passed into the macro seems highly desirable. Consider the following case with macro_rules!:

macro_rules! example {
    ($($a:ident),*) => {
        $(
            // assume each $a is an identifier for a HashMap<K,V>, where each 
            // K and V can have a different type
            let side_effect = $a.remove("some_key");
            ... // do some other stuff
        ),*
        ...
        $( 
            // do some more things with each side effect created earlier, 
            // like putting them in a tuple
        ),*
    };
}

This doesn't work because side_effect has to have a fixed name, so it'd be shadowed each 'iteration'. (Achieving something like this is possible, but it's hacky and involves a lot of boilerplate).

Being able to do something like one of these:

let side_effect_${a} = ...
let verbatim!(side_effect_, $a) = ...

would be really nice.

Maya at 2020-03-04 18:31:23

Doesn't side_effect_$ already successfully tokenize as side_effect_ $?
Eduard-Mihai Burtescu at 2020-03-12 15:29:20
Some status update (copypasted from Zulip https://rust-lang.zulipchat.com/#narrow/stream/213817-t-lang/topic/decl.20macro.20syntax/near/194354354):

macro items have multiple components to figure out before stabilizing, syntactic and semantic:
- Span::def_site, pretty far away, requires implementing cross-crate hygiene at least, and then formalizing stuff in hygiene.rs more carefully, in application to type-relative paths in particular.
- Syntax of the macro's left side (macro "parameters"), requires major design work to figure out future-compatibility (currently done with FIRST and FOLLOW sets) and figuring out how to match or not match all arms at the same time.
- Syntax of the macro's right side (macro body), requires a syntax for hygiene opt-out perhaps, maybe simplifying the use of repetitions (https://github.com/rust-lang/rust/issues/61053#issuecomment-506950747), but otherwise seems ok, it's just an arbitrary token stream.
- Surface syntax of the macro. The last year I almost wrote and RFC to set the top-level syntax macro single_arm() {} + macro multiple_arms { (lhs1) {rhs1} (lhs2) {rhs2} } in stone, but them recalled that people wanted the macros want to control what delimiters they are invoked with (e.g. restrict vec![] to only use square brackets), and that added more questions to the surface syntax, and I didn't write anything.
Vadim Petrochenkov at 2020-04-17 20:32:55
One more issue is expanding fragments like $e:expr as token streams rather than AST pieces (this is also mentioned in the top comment and links to https://github.com/rust-lang/rust/issues/26361), but this is equally applicable to macro_rules where it should be doable in a (almost) backward-compatible way.
Vadim Petrochenkov at 2020-04-17 20:34:58
Do macros 2.0 address the self both is and isn't an identifier inconsistencies described here?: https://danielkeep.github.io/tlborm/book/mbe-min-non-identifier-identifiers.html

Or the confusing behavior of macro_rules invoking other macro_rules macros not behaving the same as calling directly?: https://danielkeep.github.io/tlborm/book/mbe-min-captures-and-expansion-redux.html
jgarvin at 2020-06-27 18:44:14
Do macros 2.0 address the self both is and isn't an identifier inconsistencies described here?

self is always an identifier, keywords are a (reserved) subset of identifiers. I don't think macros 2.0 change anything here.

Or the confusing behavior of macro_rules invoking other macro_rules macros not behaving the same as calling directly?

This is not decided and needs design.
Vadim Petrochenkov at 2020-06-27 19:23:06
@petrochenkov are you saying the the author is mistaken, or that rust has changed since it was written? A number of confusing examples are provided at the link.
jgarvin at 2020-06-27 19:29:31
@jgarvin Neither. The author is right in the sense that all the examples are correct etc., but he uses a different definition of "identifier" than the language and that's probably the source of the confusion. (Anyway, this is pretty off-topic for this issue.)
Vadim Petrochenkov at 2020-06-27 19:54:48
Does macros 2.0 allow to parse generic parameters definition? With current macro_rules! the best approximation is < $( $lt:tt $( : $clt:tt $(+ $dlt:tt )* )? ),+ $(,)?>, but it does not cover all possible cases.
Warlock at 2020-09-05 02:43:27
Does anybody happen to know if there are any plans to put "macros 2.0" into the 2021 roadmap?
Denis at 2020-09-20 12:09:17
Given the ruat 2021 posts I've read so far and my vague knowledge of the state of things, it seems unlikely. There is still some design work needed and there doesn't seem to be anyone interested in pushing it over the finish line atm.
mark at 2020-09-20 12:57:46
This is maybe out of scope of this discussion, but now that Rust has (or will soon have) constant functions and miri enabling it to run rust code at compile-time, wouldn't it be possible to support in-crate procedural macros which don't work essentialy as a compiler plugin? This seems like something I personally would expect a new macro system to enable.
iqbigbang at 2021-02-25 13:02:06

Private decl macro shows up in generated docs (rustc 1.56.0-nightly (0035d9dce 2021-08-16))

// src/lib.rs
#![feature(decl_macro)]
#[derive(Clone, PartialEq)]
pub struct Foo;
macro macro1 ($foo:ty) {
  impl Eq for $foo { }
}
macro_rules! macro2 {
  ($foo:ty) => {
    impl Copy for $foo { }
  }
}
macro1!(Foo);
macro2!(Foo);

macro1 is shown in generated cargo doc, macro2 is not.

Shane Pearman at 2021-08-17 17:35:08

Before being reminded that Span::def_site() on the proc macro side is what hygiene is tracked under, I came up with this silly little snippet:
```
macro_rules! foo_impl {
    // ...
}

macro_rules! define_foo_with_hygiene {
    ($hygienic_crate:ident, $hygienic_foo_impl:ident) => {
        #[macro_export]
        macro_rules! foo {
            ($($inputs:tt)*) => {
                $hygienic_foo_impl!($hygienic_crate; $($inputs)*);
            };
        }
    };
}

macro define_foo() {
    define_foo_with_hygiene!($crate, foo_impl);
}
define_foo!();
```
The idea there is that by using $crate from inside a macro (as opposed to macro_rules!), the "hygiene 2.0" aspect was captured, and private parts of $crate can now be accessed through it (in a kind of objcap-y way, heh). (Similarly, passing foo_impl from the macro would allow foo_impl! to remain unexported)

And you would only need to stabilize "macro" IDENT "(" ")" "{" TOKEN_TREE* "}" syntax, completely obscuring any further changes to MBE input LHS pattern syntax, parsing/choice, or RHS expansion.

But if a proc macro can use Span::def_site() to achieve the same thing, this trick is probably less useful.
Eduard-Mihai Burtescu at 2022-02-22 02:27:11
Should metavariables be prevented from being named self or as keywords? I feel like metavariable keywords could be used for future expansions to macros, like associated macros with $self.
Daniel at 2022-10-04 03:37:21
yeah, I agree, no keywords should probably be valid names for metavariables, in case we want to make any of them special somehow
danielrab at 2022-10-07 21:04:24
These suggestions might be out of scope for this discussion, but it beats the RFC process, so
- hygiene could be (mostly) achieved by having macros (mangle) prefix functions with h_ and normal functions be (mangle) prefixed uh_
- If you don't want hygiene, you could use a keyword like unhygienic to prefix with uh_
```
unhygienic macro count {}
macro count2 {}

count!();
unhygienic { count2!(); }

// hygiene bending
macro count3 {
() => { unhygienic { let x = 4 } }
}
```
To bring up the out of scope part,
- Proc macros can be preprocessed, removing the need for a separate crate
- You can proc use a crate to bring it into scope during preprocessing
eap314 at 2022-11-12 20:00:18
```
* hygiene could be (mostly) achieved by having macros (mangle) prefix functions with h_ and normal functions be (mangle) 
```
As far as I understand it would not work for #[no_mangle] functions. If this is correct, then it means the suggestion just does not work.
Warlock at 2022-11-13 22:59:15
When this will be stablized :)
Yonggang Luo at 2023-08-02 19:07:54
Presumably it will be stabilised after all of the requisite tasks in the issue description are completed.
Clar Fon at 2023-08-03 07:01:46
I might not fully grasp the nuances of macros 2.0, but these are the benefits I anticipate:

1 - I should be able to use crates, types, and functions that I define in my library directly without needing $crate (suggesting deprecation of $crate). 2 - Every item created within the macro should expand; after all, that's the core purpose of macros.

This reasoning is sound:

If you desire certain types or functions to remain private, simply define them outside the macro where they remain accessible to the macro. If an item is dynamic and changes based on token input, it still needs to be expanded into the scope. Given that it's a distinct type, it shouldn't just disappear. The types or variables introduced during expansion shouldn't conflict with tokens, even if their names are identical. Think of library-defined code as being in red, and tokens in blue; they are distinct entities. The user decides where the macro expands. They can enclose it using a mod { macro!() } or restrict its scope using a block { macro!() }. At its core, macros operate on this principle: Library code + User code (Tokens) = Output Expansion. I'm concerned that we might be overcomplicating the concept of hygiene.
cybersoulK at 2023-08-09 07:50:03
From my previous points, I believe the confusion we're facing stems from how to handle situations where Library code intertwines with User code. It's naturally expected for them to merge within macros, since, by definition, a macro represents the library code's blueprint being replicated into the user's domain. Perhaps there's a need for enhanced tools to manage instances when two textual terms conflict, rather than sidestepping the issue and introducing myriad limitations to macros.

yea, i don't see why this is hard to do, if the macro has a chance of word collision.
```
mod macro_mod {
macro!();
}

pub use macro_mod::Type as NewNamedType;
```
i would not be against if the macro 2.0 used mod by default when expanding in open scopes, such as
```
use crate::macro;
macro!();
pub use macro::Type as NewNamedType;

use crate::macro as macro_renamed;
macro_renamed!();
pub use macro_renamed::*;

//automatically created macro mod and macro_renamed mod.
```
But then there must be a differentiation of macros that contain function calls and variable assignments, and should be expanded directly, such as:
```
match ... {
   A(name) => macro!(name),
}
```
cybersoulK at 2023-08-09 08:52:26
@cybersoulK do everyone here a favor please, and don't multipost. Everyone who is subscribed to this gets a separate email about each post, and the signal:noise ratio is dropping real fast ATM.

To be clear: posting is fine, but maybe don't drive people up the wall with constant useless notifications.
jjpe at 2023-08-09 09:12:20
Has there been any progress on Macros 2.0 lately?
邓云升 at 2023-08-22 00:02:52
No, but there will hopefully be a general increase in knowledge in what the issues are, as the proposal for a macros working group has been accepted. The WG will be created shortly!
Jacob Pratt at 2023-08-22 00:08:53
It could be interesting to allow a way to reverse parsing to match from right to left. This can be faked using an accumulator and recursively popping tokens until you get a match on a single one, but this is a pretty messy pattern. A more simple way to create a reverse TT muncher would make it much easier to apply precedence to contents, such as when evaluating math.
```
// Works
macro first {
    ($first:tt $($rest:tt)*) => { println!("first item is {}", $first) };
}

// Fails to compile (multiple parsing options)
macro last {
    #[reverse] // maybe this could make it work
    ($($rest:tt)* $last:tt) => { println!("last item is {}", $last) }
}
```
Edit: an alternative to this would just be adjusting the rules for multiple parsing such that it tries right to left if it gets stuck. I think this would accomplish the same thing without needing an annotation.
Trevor Gross at 2023-09-13 23:23:06
The main issue with RTL parsing is that, at least for now, parsing doesn't actually store the tokens being returned in a way that lets you iterate backwards. You could allow this and do it without allocating if you permit parsing the tokens twice (once to find the end of the macro invocation, the second time to do the actual parsing), but it's still a performance penalty to allow parsing from the other direction.

Of course, the usual way, recursively expanding an intermediate macro in reverse order, is probably slower than offering a dedicated method that allows this. If this were done, it would be nice to also have proc_macro::TokenStream implement DoubleEndedIterator to reflect that, and allow proc macros to also take advantage of that.
Clar Fon at 2023-09-15 06:58:04
Here is to hoping two big issues are getting solved (hygiene bending and nesting macros).

I tried solving macro hygiene but one pre-requisite was learning how other languages do it. Learning Racket is where I lost my motivation.
Ygg01 at 2023-09-15 09:17:49
Here is to hoping two big issues are getting solved (hygiene bending and nesting macros).

I tried solving macro hygiene but one pre-requisite was learning how other languages do it. Learning Racket is where I lost my motivation.

The issue about hygiene is "Tracking issue for Span::def_site()", this issue is blocked on it. In general, declarative macros are a syntactic sugar for procedural macros, so the "procedural macros 2.0" need to be implemented first.

Racket needs to be learned just enough to understand https://users.cs.utah.edu/plt/scope-sets/ in detail, to be able to possibly tweak it to match Rust realities.
Vadim Petrochenkov at 2023-09-20 06:14:28
one usecase of macros where rust does worse than C is local unhyginic macros used in the definition of a single array or function.

some way to bind specific variables to to those from the outer scope, or to disable macro hygine for certain identifies, would be appreciated.

perhaps i need to call a function 10 times, and each time i need to pass 7 different arguments, 6 of which are always the same local variables, and only the 7th changes. C could easily manage this with a define and undef, but in rust, you would have to pass all the local variables to each macro call, which wouldn't actually simplify the code at all.
lolbinarycat at 2024-05-11 16:37:35
which wouldn't actually simplify the code at all.

That is true for this version of "simplify":

Simpler code: shorter code.

But it is not true for the following version of "simplify":

Simpler code: code that is easier to understand and maintain.

I don't think locally disabling hygiene helps at all, if what we seek is simpler code in the second sense.
Félix Fischer at 2024-05-11 16:44:47
one usecase of macros where rust does worse than C is local unhyginic macros used in the definition of a single array or function.

I think "worse" here is generally better. I've read a pretty bad C codebase once, which is full of unhygine macros that means a lot of sloppy auto copy-and-paste. IDEs could not even help me to analyze the code. I really appreciate Rust didn't just choose the simple way to do macros.

perhaps i need to call a function 10 times, and each time i need to pass 7 different arguments, 6 of which are always the same local variables, and only the 7th changes.

Make a closure please.
ifsheldon at 2024-05-11 16:50:39
Make a closure please.

A closure can't do everything a macro can. For example it can't return from the parent function.
Thayne McCombs at 2024-05-11 18:32:40
perhaps i need to call a function 10 times, and each time i need to pass 7 different arguments, 6 of which are always the same local variables, and only the 7th changes.

I think what you're describing is partial application which is something that can already be achieved with existing crates. Combining this with the Try operator could achieve the early-return functionality described above as well.
Maya at 2024-05-11 23:25:29
it turns out the existing macro_rules! macros can already access identifiers from their parent scope, so this type of local macro actually already works, for when you need to do something that partial application can't do.
lolbinarycat at 2024-05-13 01:04:13
Tracking issues are not intended as places for discussions. This issue already has over 100 comments and lots of subscribers. Please open separate issues and link them, instead of having the discussion here.
Oli Scherer at 2024-05-20 07:44:12

🚀 I found a way to evaluate concat_idents (and concat and few other built-in macros) before evaluating other macro, which takes concat_idents as an argument! I. e. I found a way to evaluate a!(concat_idents!(...)) such way, that concat_idents evaluates before a. Answer is crate https://crates.io/crates/with_builtin_macros !!! Thanks, @danielhenrymantilla ! In other words, with_builtin_macros is paste, but not only for concat_idents, but also for concat and some other macros.

And in other words, with_builtin_macros allows one to achieve eager evaluation of macros in limited way.

Also, with_builtin_macros allows one to use concat_idents when defining new identifier.

Also, https://crates.io/crates/with_builtin_macros allows one to use concat_idents in stable Rust.

// (This code was not tested, may contain typos)

fn concat_idents!(a, b) () {} // Doesn't work

with_builtin_macros::with_eager_expansions! {
  fn #{ concat_idents!(a, b) } () {} // Works! Even on stable!
}

macro_rules! this_macro_accepts_ident {
  ($a:ident) => {}
}

// Doesn't work, because "this_macro_accepts_ident" evaluates before "concat_idents"
this_macro_accepts_ident!(concat_idents!(a, b));

with_builtin_macros::with_eager_expansions! {
  this_macro_accepts_ident!(#{ concat_idents!(a, b) }); // Works! Even on stable!
}

macro_rules! this_macro_accepts_literal {
  ($a:literal) => {}
}

// Doesn't work.
// Moreover, you cannot solve this problem using #[feature(macro_metavar_expr_concat)],
// because ${concat(...)} produces identifier, not string literal!!!
// Same applies to "paste"! "paste::paste!" deals with identifiers, not strings. So, with_builtin_macros is the only way!!!
this_macro_accepts_literal!(concat!("a", "b"));

with_builtin_macros::with_eager_expansions! {
  this_macro_accepts_literal!(#{ concat!("a", "b") }); // Works! Even on stable!
}

Askar Safin at 2024-10-13 19:11:12

@safinaskar you have posted the same comment on three different tracking issues related to macros - providing stable alternatives can be helpful, but once on the most relevant issue is enough (or better yet, something like a URLO or blog post). The suggestion also isn't at all related to this tracking issue's feature.
Trevor Gross at 2024-10-13 19:18:24
you have posted the same comment on three different tracking issues related to macros

Yes. Because I spent a lot of time searching for solution. And so I posted it everywhere, so everybody will be able to easily find it

providing stable alternatives can be helpful

In some cases with_builtin_macros is not just "alternative", but the only solution. E. g. when you need to eagerly evaluate something!(concat!(a, b))

But okay, I will post to one issue only next time
Askar Safin at 2024-10-13 19:31:13
@rustbot label A-hygiene
Askar Safin at 2025-02-07 10:22:54
@rustbot label +A-macros +WG-macros +A-macros-2.0
Askar Safin at 2025-02-07 10:36:19

Tasks

About tracking issues

Tasks

Tasks