Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7476

Fix transient failure in JapaneseNumberFilter run from TestFactories

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 6.2.1
    • Fix Version/s: 6.3, 7.0
    • Component/s: core/other
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Repeatedly running TestFactories show this test to fail ~10% of the time.

      I believe the fix is trivial and related to loosing the state of the underlying input stream when testing some analyzer life cycle flows.

      1. LUCENE-7476.patch
        10 kB
        Michael McCandless
      2. LUCENE-7476.patch
        9 kB
        Michael McCandless
      3. LUCENE-7476.patch
        1 kB
        Andy Hind

        Activity

        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Closing after 6.3.0 release.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.3.0 release.
        Hide
        mikemccand Michael McCandless added a comment -

        Thank you Andy Hind!

        Show
        mikemccand Michael McCandless added a comment - Thank you Andy Hind !
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 257ea3423f5b03495e08a8aa4263f9520deb662b in lucene-solr's branch refs/heads/master from Mike McCandless
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=257ea34 ]

        LUCENE-7476: JapaneseNumberFilter should not invoke incrementToken on its input after it's exhausted

        Show
        jira-bot ASF subversion and git services added a comment - Commit 257ea3423f5b03495e08a8aa4263f9520deb662b in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=257ea34 ] LUCENE-7476 : JapaneseNumberFilter should not invoke incrementToken on its input after it's exhausted
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit e585c84f244f4b8c8752152e9f232eba192041dc in lucene-solr's branch refs/heads/branch_6x from Mike McCandless
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e585c84 ]

        LUCENE-7476: JapaneseNumberFilter should not invoke incrementToken on its input after it's exhausted

        Show
        jira-bot ASF subversion and git services added a comment - Commit e585c84f244f4b8c8752152e9f232eba192041dc in lucene-solr's branch refs/heads/branch_6x from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e585c84 ] LUCENE-7476 : JapaneseNumberFilter should not invoke incrementToken on its input after it's exhausted
        Hide
        mikemccand Michael McCandless added a comment -

        Here's your patch (w/ some small code style edits) combined with my patch (copying over TestFactories.java) ... I think it's ready.

        Show
        mikemccand Michael McCandless added a comment - Here's your patch (w/ some small code style edits) combined with my patch (copying over TestFactories.java ) ... I think it's ready.
        Hide
        mikemccand Michael McCandless added a comment -

        And your original patch indeed fixes the issue.

        The problems is the incrementToken that JapaneseNumberFilter does lower down (in the while loop): in this case, today, the preceding tokenizer can return false while JapaneseNumberFilter returns true, causing a future call to incrementToken to illegally call incrementToken from the preceding tokenizer.

        Show
        mikemccand Michael McCandless added a comment - And your original patch indeed fixes the issue. The problems is the incrementToken that JapaneseNumberFilter does lower down (in the while loop): in this case, today, the preceding tokenizer can return false while JapaneseNumberFilter returns true, causing a future call to incrementToken to illegally call incrementToken from the preceding tokenizer.
        Hide
        mikemccand Michael McCandless added a comment -

        OK thanks Andy Hind, I think this is in fact a real bug!

        The reason it happens in Eclipse but not ant is because when TestFactories runs from Eclipse, it apparently includes the JapaneseNumberFilter in the CLASSPATH but ant does not, which is annoying. (There is a TODO about this in TestFactories).

        I'm attaching a patch (based on 6.x) that copies TestFactories.java into the kuromoji module, and then I can reproduce this failure:

        .TEST FAIL: useCharFilter=false text='alari/power.html 301'
        ENOTE: reproduce with: ant test  -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=4139A5350B1C10BE -Dtests.slow=true -Dtests.linedocsfile=/lucenedata/hudson.enwiki.random.lines.txt.fixed -Dtests.locale=cs -Dtests.timezone=America/Paramaribo -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=6, maxDocsPerChunk=1004, blockSize=452), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=6, blockSize=452)), sim=ClassicSimilarity, locale=cs, timezone=America/Paramaribo
        NOTE: Linux 4.4.0-38-generic amd64/Oracle Corporation 1.8.0_92 (64-bit)/cpus=8,threads=1,free=415218864,total=514850816
        NOTE: All tests run in this JVM: [TestFactories]
        
        Time: 2.298
        There was 1 failure:
        1) test(org.apache.lucene.analysis.ja.TestFactories)
        java.lang.IllegalStateException: incrementToken() called while in wrong state: INCREMENT_FALSE
        	at __randomizedtesting.SeedInfo.seed([4139A5350B1C10BE:C96D9AEFA5E07D46]:0)
        	at org.apache.lucene.analysis.MockTokenizer.fail(MockTokenizer.java:125)
        	at org.apache.lucene.analysis.MockTokenizer.incrementToken(MockTokenizer.java:136)
        	at org.apache.lucene.analysis.ja.JapaneseNumberFilter.incrementToken(JapaneseNumberFilter.java:152)
        	at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:716)
        	at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:627)
        	at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:525)
        	at org.apache.lucene.analysis.ja.TestFactories.doTestTokenFilter(TestFactories.java:104)
        	at org.apache.lucene.analysis.ja.TestFactories.test(TestFactories.java:57)
        
        Show
        mikemccand Michael McCandless added a comment - OK thanks Andy Hind , I think this is in fact a real bug! The reason it happens in Eclipse but not ant is because when TestFactories runs from Eclipse, it apparently includes the JapaneseNumberFilter in the CLASSPATH but ant does not, which is annoying. (There is a TODO about this in TestFactories ). I'm attaching a patch (based on 6.x) that copies TestFactories.java into the kuromoji module, and then I can reproduce this failure: .TEST FAIL: useCharFilter=false text='alari/power.html 301' ENOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=4139A5350B1C10BE -Dtests.slow=true -Dtests.linedocsfile=/lucenedata/hudson.enwiki.random.lines.txt.fixed -Dtests.locale=cs -Dtests.timezone=America/Paramaribo -Dtests.asserts=true -Dtests.file.encoding=UTF-8 NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=6, maxDocsPerChunk=1004, blockSize=452), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=6, blockSize=452)), sim=ClassicSimilarity, locale=cs, timezone=America/Paramaribo NOTE: Linux 4.4.0-38-generic amd64/Oracle Corporation 1.8.0_92 (64-bit)/cpus=8,threads=1,free=415218864,total=514850816 NOTE: All tests run in this JVM: [TestFactories] Time: 2.298 There was 1 failure: 1) test(org.apache.lucene.analysis.ja.TestFactories) java.lang.IllegalStateException: incrementToken() called while in wrong state: INCREMENT_FALSE at __randomizedtesting.SeedInfo.seed([4139A5350B1C10BE:C96D9AEFA5E07D46]:0) at org.apache.lucene.analysis.MockTokenizer.fail(MockTokenizer.java:125) at org.apache.lucene.analysis.MockTokenizer.incrementToken(MockTokenizer.java:136) at org.apache.lucene.analysis.ja.JapaneseNumberFilter.incrementToken(JapaneseNumberFilter.java:152) at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:716) at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:627) at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:525) at org.apache.lucene.analysis.ja.TestFactories.doTestTokenFilter(TestFactories.java:104) at org.apache.lucene.analysis.ja.TestFactories.test(TestFactories.java:57)
        Hide
        andyhind Andy Hind added a comment - - edited

        Running the tests 100 times via ant produces no issue. This seems to be an eclipse configuration issue.

        ant test  -Dtestcase=TestFactories -Dtests.method=test  -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        
        Show
        andyhind Andy Hind added a comment - - edited Running the tests 100 times via ant produces no issue. This seems to be an eclipse configuration issue. ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.asserts= true -Dtests.file.encoding=UTF-8
        Hide
        andyhind Andy Hind added a comment -

        I spotted this running org.apache.lucene.analysis.core.TestFactories with @Repeat (iterations = 100) from eclipse
        I just got 9 failures running this again. It is odd that I do not see them in the build failures.

        I believe the 9 fails are all the same

        java.lang.IllegalStateException: incrementToken() called while in wrong state: INCREMENT_FALSE
        	at __randomizedtesting.SeedInfo.seed([18C3960FB72D4F07:2AB7AA6A139D55E3]:0)
        	at org.apache.lucene.analysis.MockTokenizer.fail(MockTokenizer.java:125)
        	at org.apache.lucene.analysis.MockTokenizer.incrementToken(MockTokenizer.java:136)
        	at org.apache.lucene.analysis.ja.JapaneseNumberFilter.incrementToken(JapaneseNumberFilter.java:152)
        	at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:716)
        	at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:627)
        	at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:525)
        	at org.apache.lucene.analysis.core.TestFactories.doTestTokenFilter(TestFactories.java:108)
        	at org.apache.lucene.analysis.core.TestFactories.test(TestFactories.java:61)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:498)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        	at java.lang.Thread.run(Thread.java:745)
        
        

        The log gives:

        TEST FAIL: useCharFilter=false text='1'
        NOTE: reproduce with: ant test  -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        TEST FAIL: useCharFilter=false text='2'
        NOTE: reproduce with: ant test  -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        TEST FAIL: useCharFilter=false text='\ufe0f\ufe06\ufe03\ufe04\ufe0c\ufe0f\ufe08\ufe06\ufe04 7'
        NOTE: reproduce with: ant test  -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        TEST FAIL: useCharFilter=true text='\u7e08\uda09\udd65\u0cc2= \u04cd\u3bc3\uc780lW 39'
        NOTE: reproduce with: ant test  -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        TEST FAIL: useCharFilter=true text='ell'emendamento n. 8'
        NOTE: reproduce with: ant test  -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        TEST FAIL: useCharFilter=true text='uaoqfm \ue274\ud98f\udce3\ud771\ufd82\ub336 8'
        NOTE: reproduce with: ant test  -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        TEST FAIL: useCharFilter=false text='\u06c1L 1'
        NOTE: reproduce with: ant test  -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        TEST FAIL: useCharFilter=true text='R\u06b0\u4410\u03a1\u03d4 205'
        NOTE: reproduce with: ant test  -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        TEST FAIL: useCharFilter=true text='etaan huomenna klo 1'
        NOTE: reproduce with: ant test  -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        NOTE: leaving temporary files on disk at: /tmp/lucene.analysis.core.TestFactories_18C3960FB72D4F07-001
        NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=17439, maxDocsPerChunk=83, blockSize=212), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=17439, blockSize=212)), sim=ClassicSimilarity, locale=uk, timezone=Atlantic/Jan_Mayen
        NOTE: Linux 4.4.0-38-generic amd64/Oracle Corporation 1.8.0_91 (64-bit)/cpus=4,threads=1,free=34383216,total=359661568
        NOTE: All tests run in this JVM: [TestFactories]
        
        Show
        andyhind Andy Hind added a comment - I spotted this running org.apache.lucene.analysis.core.TestFactories with @Repeat (iterations = 100) from eclipse I just got 9 failures running this again. It is odd that I do not see them in the build failures. I believe the 9 fails are all the same java.lang.IllegalStateException: incrementToken() called while in wrong state: INCREMENT_FALSE at __randomizedtesting.SeedInfo.seed([18C3960FB72D4F07:2AB7AA6A139D55E3]:0) at org.apache.lucene.analysis.MockTokenizer.fail(MockTokenizer.java:125) at org.apache.lucene.analysis.MockTokenizer.incrementToken(MockTokenizer.java:136) at org.apache.lucene.analysis.ja.JapaneseNumberFilter.incrementToken(JapaneseNumberFilter.java:152) at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:716) at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:627) at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:525) at org.apache.lucene.analysis.core.TestFactories.doTestTokenFilter(TestFactories.java:108) at org.apache.lucene.analysis.core.TestFactories.test(TestFactories.java:61) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at java.lang. Thread .run( Thread .java:745) The log gives: TEST FAIL: useCharFilter= false text='1' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts= true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter= false text='2' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts= true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter= false text='\ufe0f\ufe06\ufe03\ufe04\ufe0c\ufe0f\ufe08\ufe06\ufe04 7' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts= true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter= true text='\u7e08\uda09\udd65\u0cc2= \u04cd\u3bc3\uc780lW 39' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts= true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter= true text='ell'emendamento n. 8' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts= true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter= true text='uaoqfm \ue274\ud98f\udce3\ud771\ufd82\ub336 8' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts= true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter= false text='\u06c1L 1' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts= true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter= true text='R\u06b0\u4410\u03a1\u03d4 205' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts= true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter= true text='etaan huomenna klo 1' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts= true -Dtests.file.encoding=UTF-8 NOTE: leaving temporary files on disk at: /tmp/lucene.analysis.core.TestFactories_18C3960FB72D4F07-001 NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=17439, maxDocsPerChunk=83, blockSize=212), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=17439, blockSize=212)), sim=ClassicSimilarity, locale=uk, timezone=Atlantic/Jan_Mayen NOTE: Linux 4.4.0-38- generic amd64/Oracle Corporation 1.8.0_91 (64-bit)/cpus=4,threads=1,free=34383216,total=359661568 NOTE: All tests run in this JVM: [TestFactories]
        Hide
        mikemccand Michael McCandless added a comment -

        Thanks Andy Hind.

        Do you have a Reproduce with: ... line showing the test failure? What's spooky about this is that one should never call incrementToken after it has already returned false, and token filters should not be required / expected to guard against such abuse (this patch), so I'd like to understand who is abusing JapaneseNumberFilter's incrementToken in this test failure.

        Show
        mikemccand Michael McCandless added a comment - Thanks Andy Hind . Do you have a Reproduce with: ... line showing the test failure? What's spooky about this is that one should never call incrementToken after it has already returned false, and token filters should not be required / expected to guard against such abuse (this patch), so I'd like to understand who is abusing JapaneseNumberFilter 's incrementToken in this test failure.
        Hide
        andyhind Andy Hind added a comment -

        Patch supplied - TestFactories ran with more then 700 seeds without error.

        Show
        andyhind Andy Hind added a comment - Patch supplied - TestFactories ran with more then 700 seeds without error.

          People

          • Assignee:
            Unassigned
            Reporter:
            andyhind Andy Hind
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development