Add --slowtest mode to test harness for finding the slowest test

Elixir has this. It would be pretty useful for speeding up large testsuites.

Probably should exist as a --slowtest N parameter to the test harness, which cargo test passes down.

Doubt this needs an RFC?

1. Hey @Manishearth, I'd like to do this;

   I have a few questions (new to the compiler):

   • Is this only shown if all tests pass, or does it just ignore failing tests?
   • Does this overwrite a regular test binary? Or produce a separate binary the way bench or build does?
   • I see where in libtest I can get the metrics info and add the functionality. Would I need to propagate these changes elsewhere too?
   • I assume we do this for unit and integration tests; do doc tests need it too?
   • How would you recommend testing this?

   (This is all assuming it doesn't in fact need an RFC)

   Thanks!

   LeopoldArkham at 2017-10-06 11:51:26

2. Probably ignore failing tests. But only when all tests pass is also fine by me.

   Does this overwrite a regular test binary?

   We can build this instrumentation into the regular binary and have it take the --slowest N argument that turns on measurement and reporting. I think. Making it a part of the test-binary-build step instead is fine too.

   I see where in libtest I can get the metrics info and add the functionality. Would I need to propagate these changes elsewhere too?

   Eventually we'd need cargo changes. I'm not fully clear on how the testing stuff works to know exactly what you need to change; I understand there's libtest and also the codegen done for test binaries.

   I assume we do this for unit and integration tests; do doc tests need it too?

   Your choice, that can be added later.

   How would you recommend testing this?

   Unsure, perhaps ask in #rust-internals. We can also figure this out later.

   (I'm not well right now so I can't dig up these answers for you unfortunately; I'm able to type but reading is hard)

   Manish Goregaokar at 2017-10-06 16:58:02