Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Trivial
-
Resolution: Fixed
-
Affects Version/s: 6.2.1
-
Component/s: core/other
-
Labels:None
-
Lucene Fields:New
Description
Repeatedly running TestFactories show this test to fail ~10% of the time.
I believe the fix is trivial and related to loosing the state of the underlying input stream when testing some analyzer life cycle flows.
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Thanks Andy Hind.
Do you have a Reproduce with: ... line showing the test failure? What's spooky about this is that one should never call incrementToken after it has already returned false, and token filters should not be required / expected to guard against such abuse (this patch), so I'd like to understand who is abusing JapaneseNumberFilter's incrementToken in this test failure.
I spotted this running org.apache.lucene.analysis.core.TestFactories with @Repeat (iterations = 100) from eclipse
I just got 9 failures running this again. It is odd that I do not see them in the build failures.
I believe the 9 fails are all the same
java.lang.IllegalStateException: incrementToken() called while in wrong state: INCREMENT_FALSE at __randomizedtesting.SeedInfo.seed([18C3960FB72D4F07:2AB7AA6A139D55E3]:0) at org.apache.lucene.analysis.MockTokenizer.fail(MockTokenizer.java:125) at org.apache.lucene.analysis.MockTokenizer.incrementToken(MockTokenizer.java:136) at org.apache.lucene.analysis.ja.JapaneseNumberFilter.incrementToken(JapaneseNumberFilter.java:152) at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:716) at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:627) at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:525) at org.apache.lucene.analysis.core.TestFactories.doTestTokenFilter(TestFactories.java:108) at org.apache.lucene.analysis.core.TestFactories.test(TestFactories.java:61) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at java.lang.Thread.run(Thread.java:745)
The log gives:
TEST FAIL: useCharFilter=false text='1' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter=false text='2' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter=false text='\ufe0f\ufe06\ufe03\ufe04\ufe0c\ufe0f\ufe08\ufe06\ufe04 7' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter=true text='\u7e08\uda09\udd65\u0cc2= \u04cd\u3bc3\uc780lW 39' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter=true text='ell'emendamento n. 8' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter=true text='uaoqfm \ue274\ud98f\udce3\ud771\ufd82\ub336 8' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter=false text='\u06c1L 1' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter=true text='R\u06b0\u4410\u03a1\u03d4 205' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8 TEST FAIL: useCharFilter=true text='etaan huomenna klo 1' NOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=18C3960FB72D4F07 -Dtests.locale=uk -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8 NOTE: leaving temporary files on disk at: /tmp/lucene.analysis.core.TestFactories_18C3960FB72D4F07-001 NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=17439, maxDocsPerChunk=83, blockSize=212), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=17439, blockSize=212)), sim=ClassicSimilarity, locale=uk, timezone=Atlantic/Jan_Mayen NOTE: Linux 4.4.0-38-generic amd64/Oracle Corporation 1.8.0_91 (64-bit)/cpus=4,threads=1,free=34383216,total=359661568 NOTE: All tests run in this JVM: [TestFactories]
Running the tests 100 times via ant produces no issue. This seems to be an eclipse configuration issue.
ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.asserts=true -Dtests.file.encoding=UTF-8
OK thanks Andy Hind, I think this is in fact a real bug!
The reason it happens in Eclipse but not ant is because when TestFactories runs from Eclipse, it apparently includes the JapaneseNumberFilter in the CLASSPATH but ant does not, which is annoying. (There is a TODO about this in TestFactories).
I'm attaching a patch (based on 6.x) that copies TestFactories.java into the kuromoji module, and then I can reproduce this failure:
.TEST FAIL: useCharFilter=false text='alari/power.html 301' ENOTE: reproduce with: ant test -Dtestcase=TestFactories -Dtests.method=test -Dtests.seed=4139A5350B1C10BE -Dtests.slow=true -Dtests.linedocsfile=/lucenedata/hudson.enwiki.random.lines.txt.fixed -Dtests.locale=cs -Dtests.timezone=America/Paramaribo -Dtests.asserts=true -Dtests.file.encoding=UTF-8 NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=6, maxDocsPerChunk=1004, blockSize=452), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=6, blockSize=452)), sim=ClassicSimilarity, locale=cs, timezone=America/Paramaribo NOTE: Linux 4.4.0-38-generic amd64/Oracle Corporation 1.8.0_92 (64-bit)/cpus=8,threads=1,free=415218864,total=514850816 NOTE: All tests run in this JVM: [TestFactories] Time: 2.298 There was 1 failure: 1) test(org.apache.lucene.analysis.ja.TestFactories) java.lang.IllegalStateException: incrementToken() called while in wrong state: INCREMENT_FALSE at __randomizedtesting.SeedInfo.seed([4139A5350B1C10BE:C96D9AEFA5E07D46]:0) at org.apache.lucene.analysis.MockTokenizer.fail(MockTokenizer.java:125) at org.apache.lucene.analysis.MockTokenizer.incrementToken(MockTokenizer.java:136) at org.apache.lucene.analysis.ja.JapaneseNumberFilter.incrementToken(JapaneseNumberFilter.java:152) at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:716) at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:627) at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:525) at org.apache.lucene.analysis.ja.TestFactories.doTestTokenFilter(TestFactories.java:104) at org.apache.lucene.analysis.ja.TestFactories.test(TestFactories.java:57)
And your original patch indeed fixes the issue.
The problems is the incrementToken that JapaneseNumberFilter does lower down (in the while loop): in this case, today, the preceding tokenizer can return false while JapaneseNumberFilter returns true, causing a future call to incrementToken to illegally call incrementToken from the preceding tokenizer.
Here's your patch (w/ some small code style edits) combined with my patch (copying over TestFactories.java) ... I think it's ready.
Commit e585c84f244f4b8c8752152e9f232eba192041dc in lucene-solr's branch refs/heads/branch_6x from Mike McCandless
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e585c84 ]
LUCENE-7476: JapaneseNumberFilter should not invoke incrementToken on its input after it's exhausted
Commit 257ea3423f5b03495e08a8aa4263f9520deb662b in lucene-solr's branch refs/heads/master from Mike McCandless
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=257ea34 ]
LUCENE-7476: JapaneseNumberFilter should not invoke incrementToken on its input after it's exhausted
Thank you Andy Hind!
Closing after 6.3.0 release.
Patch supplied - TestFactories ran with more then 700 seeds without error.