[SOLR-12801] Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.

• Type: Task
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Assignee: Mark Miller
• Reporter: Mark Miller
• Created: 24/Sep/18 16:03
• Updated: 24/Oct/19 00:54

Description

A single issue to counteract the single issue adding tons of annotations, the continued addition of new flakey tests, and the continued addition of flakiness to existing tests.

Lots more to come.

Issue Links

• causes: SOLR-13310 facet.threads is using the updateExecutor
• causes: SOLR-13300 Reproducing seed in master for DistributedFacetExistsSmallTest
• contains: LUCENE-8541 Fix ant beast to not overwrite junit xml results for each beast.iters iteration.
• contains: LUCENE-8545 Precommit could check for any tests in the latest commit and do a cursory beasting.
• contains: LUCENE-8547 Add a build target that beasts all of the tests modified in the last commit.
• contains: LUCENE-8546 Fix ant beast to fail and succeed based on whether beasting actually fails or succeeds.
• incorporates: SOLR-13796 Fix Solr Test Performance
• is related to: SOLR-13051 Improve TimeRoutedAlias preemptive create to play nice with tests
• is related to: SOLR-13867 Make Solrcloud stable and performant and capable of having passing tests.
• relates to: SOLR-13021 You cannot safely create and delete the same collection.
• relates to: SOLR-12933 Fix SolrCloud distributed commit.
• relates to: SOLR-13036 JettySolrRunner's port retry is broken
• relates to: SOLR-11872 Refactor test infra to work with a managed SolrClient; ditch TestHarness
• relates to: SOLR-12313 Removing TestInjection#waitForInSyncWithLeader
• relates to: SOLR-10229 See what it would take to shift many of our one-off schemas used for testing to managed schema and construct them as part of the tests
• links to: GitHub Pull Request #486

Sub-Tasks

• Fix all tests in org.apache.solr.search and begin a defense of them.: Open
• Fix all tests in org.apache.solr.update and begin a defense of them.: Open
• Introduce a new set of Mock objects that cover key parts of the system and are easy to use.: Open
• Resurrect regular test beasting reports.: Open
• Separate Solr unit tests and integration tests.: Open
• ant test (without badapples=false) should pass easily for developers.: Open
• Harden LeaderTragicEventTest.: Open
• Harden ChaosMonkeySafeLeaderWithPullReplicasTest: Open
• Harden HttpSolrCallGetCoreTest: Open
• Harden SaslZkACLProviderTest.: Open
• Testing of TLOG Replicas needs to be re-instated, may be hiding bugs: Open
• Need reliable example (Test) of how to use TestInjection.failReplicaRequests: Open
• when TestInjection.nonGracefullClose causes a TestShutdownFailError, test is garunteed tofail due to leaked objects (causes failures in (Hdfs)RestartWhileUpdatingTest): Open
• restore ability for BaseDistributedSearchTestCase subclasses to use null keys in "handle" when comparing responses: Open