Lucene - Core
  1. Lucene - Core
  2. LUCENE-4595

EnwikiContentSource thread safety problem (NPE) in 'forever' mode

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: modules/benchmark
    • Labels:
      None

      Description

      If close() is invoked around when an additional input stream reader is recreated for the 'forever' behavior, an uncaught NPE might occur.
      This bug was probably always there, just exposed now with the EnwikioContentSourceTest added in LUCENE-4588.

        Activity

        Hide
        Doron Cohen added a comment -

        Jenkin's reproduce params and error log:

        Build: http://jenkins.sd-datasolutions.de/job/Lucene-Solr-trunk-Linux/3093/
        Java: 32bit/jdk1.6.0_37 -server -XX:+UseSerialGC
        
        1 tests failed.
        FAILED:  org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSourceTest.testForever
        
        Error Message:
        Captured an uncaught exception in thread: Thread[id=140, name=Thread-2, state=RUNNABLE, group=TGRP-EnwikiContentSourceTest]
        
        Stack Trace:
        com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=140, name=Thread-2, state=RUNNABLE, group=TGRP-EnwikiContentSourceTest]
                at __randomizedtesting.SeedInfo.seed([EF7AF10441351C3B:AB004FFFCF2C6B8C]:0)
        Caused by: java.lang.NullPointerException
                at __randomizedtesting.SeedInfo.seed([EF7AF10441351C3B]:0)
                at java.io.Reader.<init>(Reader.java:61)
                at java.io.InputStreamReader.<init>(InputStreamReader.java:112)
                at org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSource$Parser.run(EnwikiContentSource.java:186)
                at java.lang.Thread.run(Thread.java:662)
        
        Build Log:
        [...truncated 5173 lines...]
        [junit4:junit4] Suite: org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSourceTest
        [junit4:junit4]   2> 7 Δεκ 2012 6:39:53 πμ com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
        [junit4:junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-2,5,TGRP-EnwikiContentSourceTest]
        [junit4:junit4]   2> java.lang.NullPointerException
        [junit4:junit4]   2>    at __randomizedtesting.SeedInfo.seed([EF7AF10441351C3B]:0)
        [junit4:junit4]   2>    at java.io.Reader.<init>(Reader.java:61)
        [junit4:junit4]   2>    at java.io.InputStreamReader.<init>(InputStreamReader.java:112)
        [junit4:junit4]   2>    at org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSource$Parser.run(EnwikiContentSource.java:186)
        [junit4:junit4]   2>    at java.lang.Thread.run(Thread.java:662)
        [junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=EnwikiContentSourceTest -Dtests.method=testForever -Dtests.seed=EF7AF10441351C3B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=el -Dtests.timezone=SST -Dtests.file.encoding=UTF-8
        [junit4:junit4] ERROR   0.07s J1 | EnwikiContentSourceTest.testForever <<<
        [junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=140, name=Thread-2, state=RUNNABLE, group=TGRP-EnwikiContentSourceTest]
        [junit4:junit4]    >    at __randomizedtesting.SeedInfo.seed([EF7AF10441351C3B:AB004FFFCF2C6B8C]:0)
        [junit4:junit4]    > Caused by: java.lang.NullPointerException
        [junit4:junit4]    >    at __randomizedtesting.SeedInfo.seed([EF7AF10441351C3B]:0)
        [junit4:junit4]    >    at java.io.Reader.<init>(Reader.java:61)
        [junit4:junit4]    >    at java.io.InputStreamReader.<init>(InputStreamReader.java:112)
        [junit4:junit4]    >    at org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSource$Parser.run(EnwikiContentSource.java:186)
        [junit4:junit4]    >    at java.lang.Thread.run(Thread.java:662)
        [junit4:junit4]   2> NOTE: test params are: codec=Lucene41: {}, sim=DefaultSimilarity, locale=el, timezone=SST
        [junit4:junit4]   2> NOTE: Linux 3.2.0-34-generic i386/Sun Microsystems Inc. 1.6.0_37 (32-bit)/cpus=8,threads=1,free=47084536,total=64946176
        [junit4:junit4]   2> NOTE: All tests run in this JVM: [TrecContentSourceTest, TestConfig, DocMakerTest, SearchWithSortTaskTest, StreamUtilsTest, WriteLineDocTaskTest, CreateIndexTaskTest, TestQualityRun, LineDocSourceTest, TestPerfTasksParse, AddIndexesTaskTest, PerfTaskTest, AltPackageTaskTest, EnwikiContentSourceTest]
        [junit4:junit4] Completed on J1 in 0.30s, 3 tests, 1 error <<< FAILURES!
        
        Show
        Doron Cohen added a comment - Jenkin's reproduce params and error log: Build: http://jenkins.sd-datasolutions.de/job/Lucene-Solr-trunk-Linux/3093/ Java: 32bit/jdk1.6.0_37 -server -XX:+UseSerialGC 1 tests failed. FAILED: org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSourceTest.testForever Error Message: Captured an uncaught exception in thread: Thread[id=140, name=Thread-2, state=RUNNABLE, group=TGRP-EnwikiContentSourceTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=140, name=Thread-2, state=RUNNABLE, group=TGRP-EnwikiContentSourceTest] at __randomizedtesting.SeedInfo.seed([EF7AF10441351C3B:AB004FFFCF2C6B8C]:0) Caused by: java.lang.NullPointerException at __randomizedtesting.SeedInfo.seed([EF7AF10441351C3B]:0) at java.io.Reader.<init>(Reader.java:61) at java.io.InputStreamReader.<init>(InputStreamReader.java:112) at org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSource$Parser.run(EnwikiContentSource.java:186) at java.lang.Thread.run(Thread.java:662) Build Log: [...truncated 5173 lines...] [junit4:junit4] Suite: org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSourceTest [junit4:junit4] 2> 7 Δεκ 2012 6:39:53 πμ com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4:junit4] 2> WARNING: Uncaught exception in thread: Thread[Thread-2,5,TGRP-EnwikiContentSourceTest] [junit4:junit4] 2> java.lang.NullPointerException [junit4:junit4] 2> at __randomizedtesting.SeedInfo.seed([EF7AF10441351C3B]:0) [junit4:junit4] 2> at java.io.Reader.<init>(Reader.java:61) [junit4:junit4] 2> at java.io.InputStreamReader.<init>(InputStreamReader.java:112) [junit4:junit4] 2> at org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSource$Parser.run(EnwikiContentSource.java:186) [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=EnwikiContentSourceTest -Dtests.method=testForever -Dtests.seed=EF7AF10441351C3B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=el -Dtests.timezone=SST -Dtests.file.encoding=UTF-8 [junit4:junit4] ERROR 0.07s J1 | EnwikiContentSourceTest.testForever <<< [junit4:junit4] > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=140, name=Thread-2, state=RUNNABLE, group=TGRP-EnwikiContentSourceTest] [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([EF7AF10441351C3B:AB004FFFCF2C6B8C]:0) [junit4:junit4] > Caused by: java.lang.NullPointerException [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([EF7AF10441351C3B]:0) [junit4:junit4] > at java.io.Reader.<init>(Reader.java:61) [junit4:junit4] > at java.io.InputStreamReader.<init>(InputStreamReader.java:112) [junit4:junit4] > at org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSource$Parser.run(EnwikiContentSource.java:186) [junit4:junit4] > at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2> NOTE: test params are: codec=Lucene41: {}, sim=DefaultSimilarity, locale=el, timezone=SST [junit4:junit4] 2> NOTE: Linux 3.2.0-34-generic i386/Sun Microsystems Inc. 1.6.0_37 (32-bit)/cpus=8,threads=1,free=47084536,total=64946176 [junit4:junit4] 2> NOTE: All tests run in this JVM: [TrecContentSourceTest, TestConfig, DocMakerTest, SearchWithSortTaskTest, StreamUtilsTest, WriteLineDocTaskTest, CreateIndexTaskTest, TestQualityRun, LineDocSourceTest, TestPerfTasksParse, AddIndexesTaskTest, PerfTaskTest, AltPackageTaskTest, EnwikiContentSourceTest] [junit4:junit4] Completed on J1 in 0.30s, 3 tests, 1 error <<< FAILURES!
        Hide
        Doron Cohen added a comment -

        Patch supposed to fix this.
        But I was not able to recreate the bug, so couldn't actually test it.

        Show
        Doron Cohen added a comment - Patch supposed to fix this. But I was not able to recreate the bug, so couldn't actually test it.
        Hide
        Robert Muir added a comment -

        This failed in my jenkins too.

        I can recreate it by running the test many times (adding -Dtests.dups=1000 or some large number to the end of the seed).

        For example:

        ant test  -Dtestcase=EnwikiContentSourceTest -Dtests.method=testForever -Dtests.seed=D4FC46984E855EE -Dtests.slow=true -Dtests.locale=es_NI -Dtests.timezone=America/Montevideo -Dtests.file.encoding=UTF-8 -Dtests.dups=1000
        
        Show
        Robert Muir added a comment - This failed in my jenkins too. I can recreate it by running the test many times (adding -Dtests.dups=1000 or some large number to the end of the seed). For example: ant test -Dtestcase=EnwikiContentSourceTest -Dtests.method=testForever -Dtests.seed=D4FC46984E855EE -Dtests.slow=true -Dtests.locale=es_NI -Dtests.timezone=America/Montevideo -Dtests.file.encoding=UTF-8 -Dtests.dups=1000
        Hide
        Robert Muir added a comment -

        +1 to commit this fix, I ran with patch:

        [junit4:junit4] JVM J0:     1.02 ..   362.37 =   361.35s
        [junit4:junit4] JVM J1:     1.02 ..   362.21 =   361.20s
        [junit4:junit4] JVM J2:     1.02 ..   362.41 =   361.39s
        [junit4:junit4] JVM J3:     1.02 ..   362.33 =   361.31s
        [junit4:junit4] Execution time total: 6 minutes 2 seconds
        [junit4:junit4] Tests summary: 10000 suites, 10000 tests
             [echo] 5 slowest tests:
        [junit4:tophints]   0.01s | org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSourceTest
        
        BUILD SUCCESSFUL
        Total time: 6 minutes 11 seconds
        

        Thanks Doron.

        Show
        Robert Muir added a comment - +1 to commit this fix, I ran with patch: [junit4:junit4] JVM J0: 1.02 .. 362.37 = 361.35s [junit4:junit4] JVM J1: 1.02 .. 362.21 = 361.20s [junit4:junit4] JVM J2: 1.02 .. 362.41 = 361.39s [junit4:junit4] JVM J3: 1.02 .. 362.33 = 361.31s [junit4:junit4] Execution time total: 6 minutes 2 seconds [junit4:junit4] Tests summary: 10000 suites, 10000 tests [echo] 5 slowest tests: [junit4:tophints] 0.01s | org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSourceTest BUILD SUCCESSFUL Total time: 6 minutes 11 seconds Thanks Doron.
        Hide
        Doron Cohen added a comment -

        Thanks for verifying Robert.
        Committed the fix, let's see if the build becomes stable again.
        Issue remains open for porting to 4x.

        Show
        Doron Cohen added a comment - Thanks for verifying Robert. Committed the fix, let's see if the build becomes stable again. Issue remains open for porting to 4x.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Doron Cohen
        http://svn.apache.org/viewvc?view=revision&revision=1418925

        LUCENE-4595: Merge from trunk: EnwikiContentSource had a thread safety problem (NPE) in 'forever' mode.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Doron Cohen http://svn.apache.org/viewvc?view=revision&revision=1418925 LUCENE-4595 : Merge from trunk: EnwikiContentSource had a thread safety problem (NPE) in 'forever' mode.
        Hide
        Doron Cohen added a comment -

        Fixed.

        Seems the tag bot missed the trunk commit for this one,
        so her they are both:

        Show
        Doron Cohen added a comment - Fixed. Seems the tag bot missed the trunk commit for this one, so her they are both: trunk: r1418281 4x: r1418925
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Doron Cohen
        http://svn.apache.org/viewvc?view=revision&revision=1418925

        LUCENE-4595: Merge from trunk: EnwikiContentSource had a thread safety problem (NPE) in 'forever' mode.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Doron Cohen http://svn.apache.org/viewvc?view=revision&revision=1418925 LUCENE-4595 : Merge from trunk: EnwikiContentSource had a thread safety problem (NPE) in 'forever' mode.

          People

          • Assignee:
            Doron Cohen
            Reporter:
            Doron Cohen
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development