Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9144

TestReplicationHandler.doTestStressReplication(): timed out waiting for collection1 startAt time to exceed: [point-in-time]

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 6.0, 6.1, 7.0
    • Fix Version/s: None
    • Component/s: replication (java)
    • Labels:
      None

      Description

      I found this while doing other testing - reproduces for me only on the branch_6_0 branch, including at tag releases/lucene-solr/6.0.0:

        2> NOTE: reproduce with: ant test  -Dtestcase=TestReplicationHandler -Dtests.method=doTestStressReplication -Dtests.seed=251666D68D49C05B -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=sr-Latn-ME -Dtests.timezone=Pacific/Niue -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
      [14:10:25.387] FAILURE 34.1s | TestReplicationHandler.doTestStressReplication <<<
         > Throwable #1: java.lang.AssertionError: timed out waiting for collection1 startAt time to exceed: Sat May 21 07:10:28 NUT 2016
         >    at __randomizedtesting.SeedInfo.seed([251666D68D49C05B:FEBD66108861A9E8]:0)
         >    at org.junit.Assert.fail(Assert.java:93)
         >    at org.apache.solr.handler.TestReplicationHandler.watchCoreStartAt(TestReplicationHandler.java:1499)
         >    at org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:851)
         >    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         >    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         >    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         >    at java.lang.reflect.Method.invoke(Method.java:498)
         >    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
         >    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
         >    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
         >    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
         >    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
         >    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
         >    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
         >    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
         >    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
         >    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
         >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
         >    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
         >    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
         >    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
         >    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
         >    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
         >    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
         >    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
         >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
         >    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
         >    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
         >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
         >    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
         >    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
         >    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
         >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
         >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
         >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
         >    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
         >    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
         >    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
         >    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
         >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
         >    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
         >    at java.lang.Thread.run(Thread.java:745)
      

      I don't know if it's relevant, but there are several CorruptIndexExceptions logged as warnings, e.g.:

        2> 6505 WARN  (qtp998957110-74) [    x:collection1] o.a.s.h.ReplicationHandler Could not read checksum from index file: _0_MockRandom_0.sd
        2> org.apache.lucene.index.CorruptIndexException: misplaced codec footer (file truncated?): length=8 but footerLength==16 (resource=MMapIndexInput(path="/Users/sarowe/git/lucene-solr/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_251666D68D49C05B-002/solr-instance-001/collection1/data/index/_0_MockRandom_0.sd"))
        2>    at org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:430)
        2>    at org.apache.solr.handler.ReplicationHandler.getFileList(ReplicationHandler.java:564)
        2>    at org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:252)
      

      I'll attach the tests-failures.txt from a run with this seed.

        Attachments

        1. Lucene-Solr-Tests-master_1157_log.txt
          3.27 MB
          Chris M. Hostetter
        2. tests-failures.txt
          314 kB
          Steven Rowe

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              sarowe Steven Rowe
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated: