Tracking issue for #[bench] and benchmarking support

Opened by Alex Crichton at 2023-05-08 16:09:22

This is a tracking issue for the #[bench] attribute and its stability in the compiler. Currently it is not possible to use this from stable Rust as it requires extern crate test which is itself not stable.

This is a tracking issue for the #[bench] attribute and its stability in the compiler. Currently it is not possible to use this from stable Rust as it requires extern crate test which is itself not stable.

Core APIs for benchmarking:
- #[bench], which means the function should take a &mut Bencher argument and will be run as part of regular tests (and specifically benchmarked with --bench passed to the test binary).
```
crate test {
    mod bench {
        #[derive(Clone)]
        struct Bencher { ... }

        impl Bencher {
            fn iter<T, F>(&mut self, inner: F)
            where
                F: FnMut() -> T;
        }       
    }
}
```
Mark Rousskov at 2022-03-02 22:59:17
Nominating for discussion in 1.6, and cc https://internals.rust-lang.org/t/bench-status/2122/10, a thread about this.

I've currently extracted libtest and I would personally like to deprecate #[bench] with a little bit of extra support from perhaps Cargo or the compiler. We'd basically be saying "let's have benchmark support develop externally first" before moving it back in to the compiler.
Alex Crichton at 2015-11-04 01:43:53
I like bench being as easy to use as it currently is, but would like more functionality.

IMO it would be nice to be able to provide custom providers for --test bineries so you could compile rustc … --test and run binary --bench, binary --quickcheck, binary --criterion etc for whatever testing and benching libraries you’re using.
Simonas Kazlauskas at 2015-11-04 17:16:03
Yeah my feeling here is that I wouldn't want to stabilize anything in the compiler itself unless it looks like a generic "custom test framework" infrastructure, but that's unfortunately a big undertaking that may take awhile. As a result my personal preference is to deprecate the support in the meantime for technically-the-same-if-not-quite-as-easy-to-use support on crates.io
Alex Crichton at 2015-11-04 17:33:19
The libs team decided to punt on this for 1.6
Alex Crichton at 2015-11-05 18:00:29
This looks like it might be a good replacement for the current #[bench] mechanism: https://github.com/japaric/criterion.rs.

I agree with Alex that deprecation is the best resolution.
Brian Smith at 2016-02-01 19:43:09
This just came up on IRC and should be a consideration for anything we do here. Allowing for "unbenchmarked" parts of an iteration. The specific case this came up for was benchmarking an in-place sorting algorithm. Since performance is going to be different for sorted vs. unsorted lists, you need to clone the list each iteration which can throw off the results.
James Miller at 2016-04-12 03:20:02
I'm sorry, but this is still required in nightly 1.20. If I write:
```
#[bench]
fn bench_blur_image(b: &mut Bencher) {}
```
Rust complains that it can't find Bencher. You have to use #![feature(test)] and extern crate test still. Why is this not stabilized or that Bencher could be found automatically? Just asking.
Felix Schütt at 2017-09-15 07:34:28
@fschutt That is correct, benchmarking with #[bench] is unstable on only available in Nightly at the moment. This issue is about tracking its stabilization.
Simon Sapin at 2017-09-15 11:18:13
What's the current status of this given the recent discussion here? Is there another ongoing discussion somewhere else?
Alkis Evlogimenos at 2017-10-23 14:20:18
https://github.com/rust-lang/rfcs/pull/2287
Manish Goregaokar at 2018-01-11 07:33:34
Latest development seems to be https://github.com/rust-lang/rfcs/pull/2318
Pierre Krieger at 2018-10-19 12:35:59
Triage: no major movement recently
Steve Klabnik at 2019-03-25 14:15:49
This has been discussed numerous times across various RFCs and internals threads over the last ~7 years (and probably before then, too), and the latest 'next step' towards stabilization here seems to be the custom test frameworks RFC (tracked in https://github.com/rust-lang/rust/issues/50297). That RFC and implementation seem unlikely to be pushed over the finish line -- even for a working, usable nightly implementation -- on any relatively short timescale, so I would like to suggest that we discuss whether the existing bench API could be minimally stabilized essentially as-is (listed in the issue description).

It has a number of extensions that users have asked for historically, some already implemented:
- Non-time data collection (e.g., memory, throughput -- in bytes/time instead of time/op)
- Customizable data reporting, aggregation, etc.
- Customizable number of iterations
- Customizable "parameter" to vary (e.g., to evaluate scalability of some algorithm)
The majority of these seem relatively feasible to either add to libtest or develop out of tree (e.g., in crates like criterion), but also look to me like extensible design atop the very simple interface we have today which has -- largely -- worked fine for the majority of basic use cases. A relatively fast skim through of the various RFCs to me suggest that the main objection to just stabilizing in place has always been more general designs or concerns about a forever stable piece of the language. Ultimately, it seems to me that a few functions and an attribute (which could have a different meaning or multiple meanings in the future) are not that painful to commit to.

For example, we could, I think, easily imagine that #[bench] here is basically the equivalent of #[ignore]. The main difference is the argument passed to the function, but it seems like the minimal API being stabilized here could be provided by other frameworks, or at least doesn't really hurt them. The surface area is pretty minimal, after all.

The stabilization would mean stable access to the test crate, which I believe today is entirely unstable, so one alternative would be to move Bencher to std::test, re-exporting it in test.
Mark Rousskov at 2022-03-02 23:10:19
@Mark-Simulacrum as far as I can see the only part that the language has to support is preventing optimizations - stabilizing black_box. Everything else can be built on top using external crates. Am I missing something?
Martin Habovštiak at 2022-03-03 09:30:04
Like #[test], #[bench] has special powers not available to #[proc_macro_attribute]: all functions in a crate with these attributes are "collected" and then some code is generated at the crate top-level to call them all.
Simon Sapin at 2022-03-03 10:55:35
black_box is separately tracked by https://github.com/rust-lang/rust/issues/64102 these days and is not necessary for a baseline stabilization (the closure return value is already passed through a black box, IIRC).

FWIW, I don't personally see bench as distinct from test there -- they both share nearly all of the same infrastructure in the compiler and libtest. If we wanted to make test the only special attribute and require #[test] #[bench] or #[test(bench)] or something on all benchmarks in the stable variant, that would probably be fine (just seems like needless churn to me).
Mark Rousskov at 2022-03-03 14:46:34
Given that the custom test frameworks experiment has been closed, how does @rust-lang/lang and @rust-lang/libs-api feel about either:
- Stabilize the #[bench] attribute and a minimal Bencher API as-is (perhaps in a different module than test::bench?)
- Or deprecate built-in benchmarking entirely.
  - cargo bench would continue to run benches/*.rs to support external tools like criterion.
  - Possible Cargo.toml changes for a future edition: make harness = false the default in [[benches]] sections, make bench = false the default in other "target" sections ([lib], [[bin]], [[example]], and [[test]]).
Simon Sapin at 2023-05-08 16:09:22