[SOLR-12028] BadApple and AwaitsFix annotations usage

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Tests
• Assignee: Erick Erickson
• Reporter: Erick Erickson
• Created: 25/Feb/18 04:33
• Updated: 18/Jul/20 14:23
• Resolved: 18/Jul/20 14:23

Description

There's a long discussion of this topic at SOLR-12016. Here's a summary:

• BadApple annotations are used for tests that intermittently fail, say < 30% of the time. Tests that fail more often shold be moved to AwaitsFix. This is, of course, a judgement call

• AwaitsFix annotations are used for tests that, for some reason, the problem can't be fixed immediately. Likely reasons are third-party dependencies, extreme difficulty tracking down, dependency on another JIRA etc.

Jenkins jobs will typically run with BadApple disabled to cut down on noise. Periodically Jenkins jobs will be run with BadApples enabled so BadApple tests won't be lost and reports can be generated. Tests that run with BadApples disabled that fail require immediate attention.

The default for developers is that BadApple is enabled.

If you are working on one of these tests and cannot get the test to fail locally, it is perfectly acceptable to comment the annotation out. You should let the dev list know that this is deliberate.

This JIRA is a placeholder for BadApple tests to point to between the times they're identified as BadApple and they're either fixed or changed to AwaitsFix or assigned their own JIRA.

I've assigned this to myself to track so I don't lose track of it. No one person will fix all of these issues, this will be an ongoing technical debt cleanup effort.

Issue Links

• is related to: SOLR-14636 Provide a reference implementation for SolrCloud that is stable and fast.
• relates to: SOLR-12877 reduce test failures due to NullPointerException
• relates to: SOLR-10107 CdcrReplicationDistributedZkTest fails far too often and is an extremely expensive test, even when compared to other nightlies.
• relates to: SOLR-10191 HdfsChaosMonkeyNothingIsSafeTest needs to be hardened.
• relates to: SOLR-7736 Add a test for ZkController.publishAndWaitForDownStates
• relates to: SOLR-11134 TestLTRReRankingPipeline.testDifferentTopN test fails
• supercedes: SOLR-5775 Disable constantly failing solr tests