Lucene - Core
  1. Lucene - Core
  2. LUCENE-4459

TestWeakIdentityMap.testConcurrentHashMap fails periodically in jenkins

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.6.1, 4.0-BETA
    • Fix Version/s: 4.1, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      There is either a bug, a test bug, or a jvm bug.

      I dont care which one it is, but lets fix the intermittent fail or disable the test.

      1. LUCENE-4459.patch
        4 kB
        Uwe Schindler
      2. LUCENE-4459.patch
        2 kB
        Uwe Schindler

        Activity

        Hide
        Michael McCandless added a comment -

        +1

        This test has also tripped on both Uwe's build box
        (http://markmail.org/thread/yrg3rnorefq5jqq6) and Simon's
        (http://markmail.org/thread/qcjegwszimqrgs3y). It also fails
        quite easily for me when running Zing (free for open-source
        dev...).

        Show
        Michael McCandless added a comment - +1 This test has also tripped on both Uwe's build box ( http://markmail.org/thread/yrg3rnorefq5jqq6 ) and Simon's ( http://markmail.org/thread/qcjegwszimqrgs3y ). It also fails quite easily for me when running Zing (free for open-source dev...).
        Hide
        Michael McCandless added a comment -

        Here's the exc when it fails:

        [junit4:junit4] Suite: org.apache.lucene.util.TestWeakIdentityMap
        [junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestWeakIdentityMap -Dtests.method=testConcurrentHashMap -Dtests.seed=3BF33FCBB5027789 -Dtests.slow=true -Dtests.locale=es_UY -Dtests.timezone=Europe/Volgograd -Dtests.file.encoding=ISO-8859-1
        [junit4:junit4] FAILURE 1.26s J0 | TestWeakIdentityMap.testConcurrentHashMap <<<
        [junit4:junit4]    > Throwable #1: java.lang.AssertionError: iteratorSize(0)>=newSize(1)
        [junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([3BF33FCBB5027789:BA829794CBA28EFB]:0)
        [junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
        [junit4:junit4]    > 	at org.junit.Assert.assertTrue(Assert.java:43)
        [junit4:junit4]    > 	at org.apache.lucene.util.TestWeakIdentityMap.testConcurrentHashMap(TestWeakIdentityMap.java:234)
        [junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        [junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        [junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        [junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:597)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        [junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        [junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        [junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:662)
        [junit4:junit4]   2> NOTE: test params are: codec=Lucene3x, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=es_UY, timezone=Europe/Volgograd
        [junit4:junit4]   2> NOTE: Windows Vista 6.0 x86/Sun Microsystems Inc. 1.6.0_23 (32-bit)/cpus=4,threads=1,free=12923248,total=16318464
        [junit4:junit4]   2> NOTE: All tests run in this JVM: [TestWeakIdentityMap]
        [junit4:junit4] Completed on J0 in 1.92s, 2 tests, 1 failure <<< FAILURES!
        
        Show
        Michael McCandless added a comment - Here's the exc when it fails: [junit4:junit4] Suite: org.apache.lucene.util.TestWeakIdentityMap [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestWeakIdentityMap -Dtests.method=testConcurrentHashMap -Dtests.seed=3BF33FCBB5027789 -Dtests.slow=true -Dtests.locale=es_UY -Dtests.timezone=Europe/Volgograd -Dtests.file.encoding=ISO-8859-1 [junit4:junit4] FAILURE 1.26s J0 | TestWeakIdentityMap.testConcurrentHashMap <<< [junit4:junit4] > Throwable #1: java.lang.AssertionError: iteratorSize(0)>=newSize(1) [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([3BF33FCBB5027789:BA829794CBA28EFB]:0) [junit4:junit4] > at org.junit.Assert.fail(Assert.java:93) [junit4:junit4] > at org.junit.Assert.assertTrue(Assert.java:43) [junit4:junit4] > at org.apache.lucene.util.TestWeakIdentityMap.testConcurrentHashMap(TestWeakIdentityMap.java:234) [junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [junit4:junit4] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:597) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787) [junit4:junit4] > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) [junit4:junit4] > at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) [junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4:junit4] > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) [junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4:junit4] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) [junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4:junit4] > at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2> NOTE: test params are: codec=Lucene3x, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=es_UY, timezone=Europe/Volgograd [junit4:junit4] 2> NOTE: Windows Vista 6.0 x86/Sun Microsystems Inc. 1.6.0_23 (32-bit)/cpus=4,threads=1,free=12923248,total=16318464 [junit4:junit4] 2> NOTE: All tests run in this JVM: [TestWeakIdentityMap] [junit4:junit4] Completed on J0 in 1.92s, 2 tests, 1 failure <<< FAILURES!
        Hide
        Uwe Schindler added a comment -

        Don't disable the test, we can maybe disable this part of the assert. I will take a look.

        Show
        Uwe Schindler added a comment - Don't disable the test, we can maybe disable this part of the assert. I will take a look.
        Hide
        Uwe Schindler added a comment -

        There are only a few cases when this can happen and one can reproduce the exact bug behaviour:
        It looks like in some cases, WeakReferences are already cleared before they appear in the ReferenceQueue (which is perfectly fine). In that case, the iterator will filter out those removed values, but the size() call later will still report them.

        This is somehow a "bug" in the test (wrong assumption: the ReferenceQueue will contain the items before they are removed). But we can do better (and make the test work). We keep the backing map up-to-date by quickly removing the GCed values in the iterator.

        I will attach a patch with some minor cleanups to make the code look like FilterIterator.

        Show
        Uwe Schindler added a comment - There are only a few cases when this can happen and one can reproduce the exact bug behaviour: It looks like in some cases, WeakReferences are already cleared before they appear in the ReferenceQueue (which is perfectly fine). In that case, the iterator will filter out those removed values, but the size() call later will still report them. This is somehow a "bug" in the test (wrong assumption: the ReferenceQueue will contain the items before they are removed). But we can do better (and make the test work). We keep the backing map up-to-date by quickly removing the GCed values in the iterator. I will attach a patch with some minor cleanups to make the code look like FilterIterator.
        Hide
        Uwe Schindler added a comment -

        Patch.

        This is not a bug, so no need to backport to 4.0 or 3.6. It is just an improvement, whcih makes the test pass always!

        Show
        Uwe Schindler added a comment - Patch. This is not a bug, so no need to backport to 4.0 or 3.6. It is just an improvement, whcih makes the test pass always!
        Hide
        Uwe Schindler added a comment -

        I added a test improvement to verify the map size before the iterator, too.

        Show
        Uwe Schindler added a comment - I added a test improvement to verify the map size before the iterator, too.
        Hide
        Uwe Schindler added a comment -

        Mike tested this with Zing VM and was not able to reproduce this anymore (which was easy with Zing).

        I will commit now.

        Show
        Uwe Schindler added a comment - Mike tested this with Zing VM and was not able to reproduce this anymore (which was easy with Zing). I will commit now.
        Hide
        Uwe Schindler added a comment -

        Committed to trunk and 4.x revision 1394291, 1394293

        Show
        Uwe Schindler added a comment - Committed to trunk and 4.x revision 1394291, 1394293
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Uwe Schindler
        http://svn.apache.org/viewvc?view=revision&revision=1394293

        Merged revision(s) 1394291 from lucene/dev/trunk:
        LUCENE-4459: Improve WeakIdentityMap.keyIterator() to remove GCed keys from backing map early instead of waiting for reap(). This makes test failures in TestWeakIdentityMap disappear, too.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Uwe Schindler http://svn.apache.org/viewvc?view=revision&revision=1394293 Merged revision(s) 1394291 from lucene/dev/trunk: LUCENE-4459 : Improve WeakIdentityMap.keyIterator() to remove GCed keys from backing map early instead of waiting for reap(). This makes test failures in TestWeakIdentityMap disappear, too.

          People

          • Assignee:
            Uwe Schindler
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development