libtest Shouldn't Panic On Unexpected Command-line Arg

Opened by Brook Heisler at 2023-03-09 20:25:03

Right now, if the user passes a command-line argument to a test or benchmark executable that isn't recognized, it will produce an error like the following:

thread 'main' panicked at '"Unrecognized option: \'example\'."', /checkout/src/libtest/lib.rs:278:26
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: bench failed

There are two problems with this. First, it would be nice to give a cleaner error message, since this is an expected case. Second and more importantly, the panic halts the cargo bench or cargo test run without running other executables.

I've attached an example project showing this. The fake benchmark harness in benches/example.rs checks for the argument --example. However, the libtest harness for the tests in the src directory is launched first and panics on seeing the same argument. This means that the naive cargo bench -- --example doesn't work as one would expect.

command-line-example.zip

This is a problem for me because it means I can't add command-line arguments to Criterion.rs benchmarks without an extra hassle for users unless libtest also accepts those same arguments.

Note on the second part (accepting arguments), it is not so great to have any test runner silently accept unknown arguments. Does it not work for you to run the specific benchmark only, with the argument?
1. Like this: cargo bench --bench example -- --example
2. If benches/example is actually the only real benchmark runner in the package, then just disable bench building for the lib crate itself ([lib]\n bench = false)
bluss at 2018-01-07 14:09:07
I didn't know it was possible to disable benchmarks for the lib crate. That's useful to know, thanks.

I think the issue with both of these options is that they impose an ease-of-use burden on the end user. Either they have to specify one specific benchmark, or they can no longer use benchmarks in the lib crate at all.

I suppose these limitations are acceptable, at least for now. A better solution would be to make it possible to use alternate benchmarking harnesses in the lib crate itself (without also removing the test harness), which would also solve a lot of other limitations. That would probably require an RFC, yes?
Brook Heisler at 2018-01-07 15:34:30

As-reported, this has been fixed for a while. Benches and tests don't panic if passed unexpected arguments, and the provided example prints this:

   Compiling command-line-example v0.1.0 (/tmp/command-line-example)
    Finished bench [optimized] target(s) in 0.52s
     Running unittests src/lib.rs (target/release/deps/command_line_example-41f53ad111ac3615)
error: Unrecognized option: 'example'
error: bench failed, to rerun pass `--lib`

Ben Kimock at 2022-10-09 17:26:04

@saethlin From a criterion.rs user perspective this issue still isn't fully resolved as far as I can see, even though it is technically no longer a panic. Criterion offers for instance --verbose and running cargo bench -- --verbose should work. However in the most basic "hello world" setup it fails with:
```
error: Unrecognized option: 'verbose'
error: bench failed, to rerun pass `--bin foo`
```
Rerunning with --bin foo is exactly what doesn't work, because that's not the name of benchmark binary, but rather the package name that doesn't contain a criterion.rs benchmark. I.e. following this recommendation just produces the same error, which isn't really the expected UX.

The work-around of introducing a [lib] or [[bin]] section containing bench = false also isn't great, because it requires to re-specify the name property, which isn't very DRY in the case where the binary name should simply follow the package name. In a cargo workspaces project, it also requires to adapt all sub Cargo.toml's to disable bench argparsing for all bins/libs.
Fabian Keller at 2023-03-09 20:25:03