[SOLR-9144] TestReplicationHandler.doTestStressReplication(): timed out waiting for collection1 startAt time to exceed: [point-in-time]

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 6.0, 6.1, 7.0
• Component/s: replication (java)
• Assignee: Unassigned
• Reporter: Steven Rowe
• Created: 21/May/16 20:23
• Updated: 24/May/16 23:56

Description

I found this while doing other testing - reproduces for me only on the branch_6_0 branch, including at tag releases/lucene-solr/6.0.0:

  2> NOTE: reproduce with: ant test  -Dtestcase=TestReplicationHandler -Dtests.method=doTestStressReplication -Dtests.seed=251666D68D49C05B -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=sr-Latn-ME -Dtests.timezone=Pacific/Niue -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[14:10:25.387] FAILURE 34.1s | TestReplicationHandler.doTestStressReplication <<<
   > Throwable #1: java.lang.AssertionError: timed out waiting for collection1 startAt time to exceed: Sat May 21 07:10:28 NUT 2016
   >    at __randomizedtesting.SeedInfo.seed([251666D68D49C05B:FEBD66108861A9E8]:0)
   >    at org.junit.Assert.fail(Assert.java:93)
   >    at org.apache.solr.handler.TestReplicationHandler.watchCoreStartAt(TestReplicationHandler.java:1499)
   >    at org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:851)
   >    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.lang.reflect.Method.invoke(Method.java:498)
   >    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   >    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
   >    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
   >    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
   >    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   >    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   >    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   >    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   >    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   >    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   >    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
   >    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
   >    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
   >    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
   >    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
   >    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   >    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   >    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   >    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   >    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   >    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   >    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   >    at java.lang.Thread.run(Thread.java:745)

I don't know if it's relevant, but there are several CorruptIndexExceptions logged as warnings, e.g.:

  2> 6505 WARN  (qtp998957110-74) [    x:collection1] o.a.s.h.ReplicationHandler Could not read checksum from index file: _0_MockRandom_0.sd
  2> org.apache.lucene.index.CorruptIndexException: misplaced codec footer (file truncated?): length=8 but footerLength==16 (resource=MMapIndexInput(path="/Users/sarowe/git/lucene-solr/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_251666D68D49C05B-002/solr-instance-001/collection1/data/index/_0_MockRandom_0.sd"))
  2>    at org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:430)
  2>    at org.apache.solr.handler.ReplicationHandler.getFileList(ReplicationHandler.java:564)
  2>    at org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:252)

I'll attach the tests-failures.txt from a run with this seed.