[STDCXX-683] implement notion of expected failures in the test suite

• Type: New Feature
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 4.2.0
• Fix Version/s: 4.2.2
• Component/s: Test Driver, Test Harness
• Severity: Usability
• Assignee: Martin Sebor
• Reporter: Martin Sebor
• Created: 18/Dec/07 23:58
• Updated: 12/Sep/08 00:33

Description

Tests (or examples) that fail for known reasons that we haven't been able to deal with should be distinguished from failures that haven't been analyzed yet. For example, an example program that fails to compile on an older target platform because of a compiler bug that we can't find a simple/elegant workaround should be flagged as such in the test results. Similarly, a test that fails one or more assertions due to compiler or libc bugs on a specific platform (or a set of platforms) that we are unable to work around should be reported as such.

This is important in order to reduce the currently fairly large number of unexpected failures and to be able to make changes without having to worry about regressions as much.

Issue Links

• is depended upon by: STDCXX-702 move all buildlog parsing from xbuildgen to xcomp.awk