[HARMONY-4842] [bti] Unreliable status calculation for snapshot testing results

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: build - test - ci
• Assignee: Stepan Mishura
• Reporter: Alexey Varlamov
• Created: 24/Sep/07 09:06
• Updated: 29/Nov/07 11:30

Description

The snapshot status page [1] wrongly accounts some suites PASSED. It happens when suite is expected to generate JUnit-style report and some tests crash or cannot compile and build run aborts after running only part of suite. In this case those few successfully run tests are published and suite status is shown green on the page [1].
This in particular explains wrong data for DRLVM tests on Windows [2], which in reality is caused by compilation errors in cunit tests.

The most reliable way would be to generate status.txt always and rely on it when calculating status, while look for JUnit statistics only to get pass rate.

[1] http://people.apache.org/~mloenko/snapshot_testing/script/snapshots_summary.html
[2] http://people.apache.org/~mloenko/snapshot_testing/script/r577501/index.html

Issue Links

• relates to: HARMONY-4350 [site]Script for automatic generation of talbe with snapshots info (testing results, download path, etc)