Compile ignored tests rather than skipping

2f5e20a
Opened by klutzy/defunct at 2024-10-17 02:53:13

Some tests have ignore-test then become old - we can't build them now. If they are skipped due to unexpected assertion failure, then it's better to compile and skip running. Same arguments for ignore-<target>.

(We may introduce ignore-compile if needed?)

  1. Ignored tests:

    Lack of support in codegen for closures/generics are always inline:

    • [ ] src/test/codegen-units/item-collection/cross-crate-closures.rs

    • [ ] src/test/codegen-units/item-collection/non-generic-closures.rs

    • [ ] src/test/codegen-units/partitioning/methods-are-with-self-type.rs

    This compile successfully, but it should fail:

    • [ ] src/test/ui/lint/dead-code/closure-bang.rs

    Mark Rousskov at 2020-04-30 14:04:45

  2. My idea is to add a way to specify why a test is ignored (causes ice, fails to compile, fails to run). Test runner could compile such tests and check whether they still compile / cause ice / fail to compile / fail to run and warn if the something changed.

    Piotr Jawniak at 2014-06-30 10:27:18

  3. Triage: I'm not aware of any changes in this area.

    Steve Klabnik at 2016-02-02 18:26:44

  4. Attempted to triage ignored tests. Summary below, also filed https://github.com/rust-lang/rust/pull/41629 to unignore or update the ignore statements on a few tests to indicate that they are auxiliary files.

    Nightly-only:

    • [ ] src/test/rustdoc/issue-27362.rs

    Currently fail, undiagnosed reasons:

    • [ ] src/test/codegen-units/item-collection/cross-crate-closures.rs
    • [ ] src/test/codegen-units/item-collection/non-generic-closures.rs
    • [ ] src/test/codegen-units/partitioning/methods-are-with-self-type.rs
    • [ ] src/test/run-pass/issue-31702.rs

    I don't even know what this tried to test, but I expect it's gone by now (delete?):

    • [x] src/test/parse-fail/regions-out-of-scope-slice.rs

    This compile successfully, but it should fail:

    • [ ] src/test/compile-fail/dead-code-closure-bang.rs

    Test claims that it has different output on 32/64 bit (#23926):

    • [ ] src/test/compile-fail/huge-array-simple.rs

    Unsized enums aren't supported:

    • [ ] src/test/compile-fail/issue-17025.rs

    This does not appear to be a compile-fail test:

    • [ ] src/test/compile-fail/macro-expanded-include/foo/mod.rs

    This should be a UI-test, probably -- compile-fail can't test this I think:

    • [ ] src/test/compile-fail/fully-qualified-type-name3.rs

    No associated stderr file for undiagnosed reasons:

    • [ ] src/test/ui/codemap_tests/two_files_data.rs

    Mark Rousskov at 2017-04-29 12:27:05

  5. I don't even know what this tried to test, but I expect it's gone by now (delete?):

    * [ ]  `src/test/parse-fail/regions-out-of-scope-slice.rs`

    I did some research into where this came from; wow was that a rabbit hole.

    Here's the commit where that test came from: https://github.com/rust-lang/rust/commit/2db4259b3531e0cfa1f6df2ff95e30a4c871a105#diff-d7743dac25dc59daac6583aab0dc1bb8

    It has a lot of interesting stuff (beyond just the look back into very old Rust syntax), like a hard-coded blk region for the current block (!).

    From its inception, it was tagged an "xfail-test" (as in, ignore-test). The reason for it being ignored, according to https://github.com/rust-lang/rust/commit/f8bd95589fceb893e5244f8f94e010a8b30f6529#diff-884e9fc8b00b17708aa78ee7e6645c6b , is that "blk" wasn't supported in the front-end. (Though I think the type checker was still doing something special for it, as shown here : https://github.com/rust-lang/rust/blob/f8bd95589fceb893e5244f8f94e010a8b30f6529/src/librustc/middle/typeck/check.rs#L580)

    So, yeah, we should just kill that test.

    Felix S Klock II at 2019-01-25 14:58:02

  6. I've filed https://github.com/rust-lang/rust/pull/71711 which further updates some things here and put an updated description of what's left as ignored tests in the issue description. I'm personally leaning towards closing this at this point as I think the remaining tests are sufficiently uninteresting to not be worth having a meta tracking issue like this for.

    Mark Rousskov at 2020-04-30 14:56:13

  7. May I ask what the state of this issue is? What has to be done? (I'm a newbie in the rust-eco) Can I help here?

    TornaxO7 at 2023-03-10 20:44:23

  8. Oh goodness, this issue is ancient. It looks like the current header command for this is ignore-test, which is used in a bunch of different tests:

    ; rg ignore-test tests | wc -l
65

    @TornaxO7 if you want to help, I think the steps would look something like this:

    1. Categorize why the tests are ignored.
      • In particular, a lot have "not a test" comments, so they can be moved to auxiliary folders to avoid compiletest trying to build them.
    2. Decide how to fix them. A lot have more specific issues associated with them and those you can probably just ignore. The rest will be on a case-by-case basis; you can ask on Zulip if you get stuck. I think all ignored tests should either have an associated issue or be removed all together.

    jyn at 2023-05-24 02:30:16

  9. (I know this issue is ancient, but)

    After categorizing and possibly making decisions about how to fix them, we should have compiletest enforce that ignore-test must be given a reason and not just a //@ ignore-test last modified 10 years ago without any context. In general, we should (eventually) enforce that any ignore-* directives must come with a reason justifying the ignore.

    许杰友 Jieyou Xu (Joe) at 2024-06-11 10:01:53

  10. compilestest triage: current ignore-test (it is possible that some conditional test running directives produce a combination that leads to tests that never run in CI)

    tests\debuginfo\drop-locations.rs
2://@ ignore-test: #128971

tests\rustdoc\macro-document-private-duplicate.rs
1://@ ignore-test (fails spuriously, see issue #89228)

tests\rustdoc\inline_cross\assoc-const-equality.rs
3://@ ignore-test (FIXME: #125092)

tests\ui\explicit-tail-calls\drop-order.rs
2://@ ignore-test: tail calls are not implemented in rustc_codegen_ssa yet, so this causes 🧊

tests\run-make\branch-protection-check-IBT\_rmake.rs
8://@ ignore-test

tests\run-make\branch-protection-check-IBT\Makefile
10:# ignore-test

tests\ui\generic-const-items\recursive.rs
3://@ ignore-test

tests\ui\issues\issue-59756.rs
2://@ ignore-test (rustfix needs multiple suggestions)

tests\ui\issues\issue-49298.rs
9://@ ignore-test (#54987)

tests\ui\lint\dead-code\self-assign.rs
4://@ ignore-test FIXME(81658, 83171)

tests\ui\match\issue-27021.rs
7://@ ignore-test (#54987)

tests\ui\match\issue-26996.rs
7://@ ignore-test (#54987)

tests\ui\lint\unknown-lints\other.rs
1://@ ignore-test

tests\ui\panics\panic-short-backtrace-windows-x86_64.rs
3://@ ignore-test (#92000)

tests\ui\traits\next-solver\object-soundness-requires-generalization.rs
2://@ ignore-test

tests\ui\precondition-checks\write_bytes.rs
5://@ ignore-test

tests\ui\precondition-checks\write.rs
5://@ ignore-test

tests\ui\precondition-checks\read.rs
5://@ ignore-test

    许杰友 Jieyou Xu (Joe) at 2024-10-17 02:53:13