Support death tests in libtest

0b9f69d
Opened by gnzlbg at 2024-05-04 00:09:40

I would like to be able to mark a test as a death test of some kind, like, running this code should trigger an abort (instead of just an unwind), an exit with a particular exit code, or an assert (in case assert doesn't unwind if, e.g. panic=abort).

This requires two fundamental ingredients, libtests needs to support:

  • spawning test in different processes (that is, process spawning/setup and teardown)
  • marking tests as death tests (and maybe indicate the type).

A way to mark a test as a death test is important, since in the case in which panic != abort one still wants all the non death tests to happen within the same process for speed, but libtest stills need to spawn different processes for the death tests only (so it needs a way to tell these apart).

For crates in which panic == abort, it would be nice if one could turn all tests into death tests at the crate level.

This issue is tangentially related to: https://github.com/rust-lang/rfcs/pull/1513

Google Test is probably the most famous unit-testing framework that supports these although in C++ other frameworks do as well. It usually takes 3k LOC C++ code to implement a "portable" process spawning/set up/tear down mechanism for spawning death tests, so it is quite a bit of machinery that might better belong outside of rustc.

  1. In https://github.com/rust-lang/rust/issues/43788 I had two possible solutions to the panic=abort problem, although only the former solves the "death test" problem

    • Instead of starting a thread for all tests we could instead start a process. This process could be a re-execution of the executable itself and we'd just do that once per test. The downside of this is that it may be super slow.
    • We could continue to start a thread for each test but override the panic handler for the process. This panic handler would do "test related things" and provide an opportunity for a "clean exit". The downside of this is that you wouldn't get a set of all tests that failed, just the first few (maybe)

    Alex Crichton at 2017-08-10 18:26:33

  2. I think we definitively should address the "death test" problem – there are a lot of situations where programs call abort() directs (e.g. an external C library like libsodium or a custom memory allocator).

    About the performance: wouldn't it be possible to limit the re-execution to "death tests" or [should_panic] tests in panic = abort scenarios? And if those tests are slow – then so be it... I mean those tests are relatively rare and if the overhead is communicated clearly, I don't see any problem with it (unless we have a better alternative).

    Keziah Biermann at 2019-01-20 04:36:29

  3. It's pretty easy to write your own custom_test_framework today that runs each test on its own process. If you need "death tests" for some of your tests, the easiest way would be to use such a test framework only for those, while continuing to use libtest for the rest.

    gnzlbg at 2019-01-20 08:13:07

  4. I'm working on a change to libtest that will implement the first of @alexcrichton's solutions above, specifically to support panic=abort. It should be pretty easy to extend to allow proper death tests, though.

    So far I have a working prototype that's around 150 lines of implementation code. My intention is to make it "opt-in," so you have to either compile with a special flag or supply an argument on the command line. (Not sure how I'm going to go about this yet. Ideally for me, we can supply an option while building libtest to enable this for all tests, because we want panic=abort in our whole build.)

    cc @rust-lang/libs, WDYT? Would this be a welcome addition to libtest?

    Tyler Mandry at 2019-08-17 00:18:36

  5. I would personally at least welcome a change to libtest to support panic=abort mode in libtest. Ideally this would all be automatically inferred in that the test crate would automatically switch to process-per-test if linked into an executable as panic=abort, continuing to use threads by default otherwise. Having a runtime switch to use one over the other also makes sense to me though!

    Alex Crichton at 2019-08-19 13:58:30

  6. Ideally this would all be automatically inferred in that the test crate would automatically switch to process-per-test if linked into an executable as panic=abort

    I agree that this would be ideal. Any suggestions on how to do it?

    The best way I can think of off the top of my head is choosing a mode at the time of building libtest. I'm not sure how to detect that a specific binary is linked as panic=abort.

    Tyler Mandry at 2019-08-19 22:12:26

  7. Choosing when libtest is built is possible but probably not gonna work out well because we ship a prebuilt libtest which is always compiled as panic=unwind. I think though that fn main() for the test harness is always synthesized at the end crate, so we could synthesize a main function which passes a boolean to libtest. That'd all be unstable and not really usable for other crates, so no new extra surface area to stabilize!

    Alex Crichton at 2019-08-19 22:20:07

  8. The fn main() that's synthesized basically just calls the test_runner.

    Currently, the compiler selects libtest's test_main_static as the test_runner. We could build another entrypoint called test_main_static_abort (or something less gross), and when the compiler sets the test_runner it inspects the panic mode and chooses accordingly.

    djrenren at 2019-08-27 05:27:41

  9. I think for panic=abort we don't want to group tests at all by default. Because this would only report the first test failure as opposed to all. I think this is a good thing to put behind a runtime flag, for if you really need the performance.

    djrenren at 2019-08-27 05:36:53

  10. "No, Mr. Bond! I expect you to die."

    Jubilee at 2024-05-04 00:09:40