Lucene - Core
  1. Lucene - Core
  2. LUCENE-5212

java 7u40 causes sigsegv and corrupt term vectors

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Lucene Fields:
      New
    1. crashFaster2.0.patch
      1 kB
      Robert Muir
    2. crashFaster.patch
      1 kB
      Robert Muir
    3. jenkins.txt
      124 kB
      Robert Muir
    4. hs_err_pid32714.log
      58 kB
      Robert Muir

      Issue Links

        Activity

        Robert Muir created issue -
        Hide
        Robert Muir added a comment -

        This has happened twice in jenkins since Uwe upgraded, so i tried to reproduce myself.

        With update 25, no issues.
        So i ugpraded to update 40: on the first try with the jenkins commandline:

        rmuir@beast:~/workspace/lucene-trunk/lucene/core$ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dtests.dynamicAssignmentRatio=0 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC"

           [junit4] #
           [junit4] # A fatal error has been detected by the Java Runtime Environment:
           [junit4] #
           [junit4] #  SIGSEGV (0xb) at pc=0x00007f163d2d34dd, pid=32714, tid=139732803393280
           [junit4] #
           [junit4] # JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43)
           [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 )
           [junit4] # Problematic frame:
           [junit4] # J  org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(I)Lorg/apache/lucene/index/Fields;
           [junit4] #
           [junit4] # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
           [junit4] #
           [junit4] # An error report file with more information is saved as:
           [junit4] # /home/rmuir/workspace/lucene-trunk/lucene/build/core/test/J0/hs_err_pid32714.log
        
        Show
        Robert Muir added a comment - This has happened twice in jenkins since Uwe upgraded, so i tried to reproduce myself. With update 25, no issues. So i ugpraded to update 40: on the first try with the jenkins commandline: rmuir@beast:~/workspace/lucene-trunk/lucene/core$ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dtests.dynamicAssignmentRatio=0 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC" [junit4] # [junit4] # A fatal error has been detected by the Java Runtime Environment: [junit4] # [junit4] # SIGSEGV (0xb) at pc=0x00007f163d2d34dd, pid=32714, tid=139732803393280 [junit4] # [junit4] # JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43) [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 ) [junit4] # Problematic frame: [junit4] # J org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(I)Lorg/apache/lucene/index/Fields; [junit4] # [junit4] # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again [junit4] # [junit4] # An error report file with more information is saved as: [junit4] # /home/rmuir/workspace/lucene-trunk/lucene/build/core/test/J0/hs_err_pid32714.log
        Hide
        Robert Muir added a comment -

        Attached is the hs_err

        Show
        Robert Muir added a comment - Attached is the hs_err
        Robert Muir made changes -
        Field Original Value New Value
        Attachment hs_err_pid32714.log [ 12603182 ]
        Hide
        Robert Muir added a comment -

        i ran this 3 times... first 2 times SIGSEGV, third time it passed. so its not totally consistent but seems to reproduce.

        I will now try to reproduce the other seed from this morning.

        Show
        Robert Muir added a comment - i ran this 3 times... first 2 times SIGSEGV, third time it passed. so its not totally consistent but seems to reproduce. I will now try to reproduce the other seed from this morning.
        Hide
        Robert Muir added a comment -

        The other seed i cant reproduce:

        (from lucene/classification: ant test -Dtests.seed=68F81CB00E47182B -Dtests.jvms=2 -Dtests.dynamicAssignmentRatio=0 -Dargs="-XX:-UseCompressedOops -XX:+UseG1GC")

        Note you have to rm -rf your .caches (sorry, dynamicAssignmentRatio=0 doesnt quite work), or the tests wont be executed in the same order as jenkins did.

        Still i cant reproduce that one, but the later seed above in core tests is easier anyway since it only launches 1 JVM and it SIGSEGVs.

        Show
        Robert Muir added a comment - The other seed i cant reproduce: (from lucene/classification: ant test -Dtests.seed=68F81CB00E47182B -Dtests.jvms=2 -Dtests.dynamicAssignmentRatio=0 -Dargs="-XX:-UseCompressedOops -XX:+UseG1GC") Note you have to rm -rf your .caches (sorry, dynamicAssignmentRatio=0 doesnt quite work), or the tests wont be executed in the same order as jenkins did. Still i cant reproduce that one, but the later seed above in core tests is easier anyway since it only launches 1 JVM and it SIGSEGVs.
        Hide
        Robert Muir added a comment -

        attached is original jenkins log (they get removed from the server pretty fast).

        in jenkins on os X, it didnt crash but had corrupt term vectors in several unrelated tests.

        Show
        Robert Muir added a comment - attached is original jenkins log (they get removed from the server pretty fast). in jenkins on os X, it didnt crash but had corrupt term vectors in several unrelated tests.
        Robert Muir made changes -
        Attachment jenkins.txt [ 12603183 ]
        Hide
        Dawid Weiss added a comment -

        > (sorry, dynamicAssignmentRatio=0 doesnt quite work),

        The lack of determinism may be due to work stealing – dynamicAssignmentRatio=0 preallocates tasks to JVMs, while work stealing may actually assign differently on every run.

        Perhaps we should run jenkins jobs with dynamicAssignmentRatio=0... but then all the benefits of work time balancing among cores would be gone and fixed task assignment only helps in very specific bug cases (inter-suite dependency or JVM bugs). Hard to tell.

        Show
        Dawid Weiss added a comment - > (sorry, dynamicAssignmentRatio=0 doesnt quite work), The lack of determinism may be due to work stealing – dynamicAssignmentRatio=0 preallocates tasks to JVMs, while work stealing may actually assign differently on every run. Perhaps we should run jenkins jobs with dynamicAssignmentRatio=0... but then all the benefits of work time balancing among cores would be gone and fixed task assignment only helps in very specific bug cases (inter-suite dependency or JVM bugs). Hard to tell.
        Hide
        Robert Muir added a comment -

        I'm not complaining, I'm happy we are able to reproduce the 'whole run' in any shape or form. one extra command does not bother me, we shouldnt slow all of our jenkins jobs down to make jvm bug-debugging easier

        Show
        Robert Muir added a comment - I'm not complaining, I'm happy we are able to reproduce the 'whole run' in any shape or form. one extra command does not bother me, we shouldnt slow all of our jenkins jobs down to make jvm bug-debugging easier
        Hide
        Michael McCandless added a comment -

        in jenkins on os X, it didnt crash but had corrupt term vectors in several unrelated tests.

        Very scary that this can also be a "silently introduces index corruption" JVM bug! I'd much prefer the sudden death SEGV.

        Show
        Michael McCandless added a comment - in jenkins on os X, it didnt crash but had corrupt term vectors in several unrelated tests. Very scary that this can also be a "silently introduces index corruption" JVM bug! I'd much prefer the sudden death SEGV.
        Show
        Robert Muir added a comment - https://bugs.openjdk.java.net/browse/JDK-8024830
        Hide
        Robert Muir added a comment -

        I crashed again but with a core file (set 'ulimit -c unlimited').

        zip file with core dump and hs_err is here: http://people.apache.org/~rmuir/crash.zip (its too large for JIRA, sorry)

        For some more context, it always happens fairly early in the test run:
        so when it doesnt crash at this exact point, you can ^C and run again until it does.

        Here was my commands with output below: (i tried to simplify the procedure to make it easy to reproduce, but its not easy, it took me quite a few tries)

        # note: we are pulling the exact revision that jenkins failed on, because things have changed in lucene codebase over the weekend!!!!
        svn co -r 1523179 https://svn.apache.org/repos/asf/lucene/dev/trunk
        # just go to core tests
        cd trunk/lucene/core
        #
        # now the following two commands: just run again and again until it crashes.
        #
        rm -rf ../../.caches/
        ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC"
        

        Here was the output:

        rmuir@beast:~/workspace/trunk/lucene/core$ rm -rf ../../.caches/
        rmuir@beast:~/workspace/trunk/lucene/core$ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC"
        Buildfile: /home/rmuir/workspace/trunk/lucene/core/build.xml
        
        -clover.disable:
        
        -clover.load:
        
        -clover.classpath:
        
        -clover.setup:
        
        clover:
        
        ivy-availability-check:
        
        ivy-fail:
        
        ivy-configure:
        [ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ ::
        [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml
        
        resolve:
        
        init:
        
        compile-core:
        
        compile-test-framework:
        
        ivy-availability-check:
        
        ivy-fail:
        
        ivy-configure:
        [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml
        
        resolve:
        
        init:
        
        compile-lucene-core:
        
        ivy-availability-check:
        
        ivy-fail:
        
        ivy-configure:
        [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml
        
        resolve:
        
        init:
        
        -clover.disable:
        
        -clover.load:
        
        -clover.classpath:
        
        -clover.setup:
        
        clover:
        
        compile-core:
        
        compile-codecs:
             [echo] Building codecs...
        
        ivy-availability-check:
             [echo] Building codecs...
        
        ivy-fail:
        
        ivy-configure:
        [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml
        
        resolve:
        
        common.init:
        
        compile-lucene-core:
        
        init:
        
        -clover.disable:
        
        -clover.load:
        
        -clover.classpath:
        
        -clover.setup:
        
        clover:
        
        compile-core:
        
        -clover.disable:
        
        -clover.load:
        
        -clover.classpath:
        
        -clover.setup:
        
        clover:
        
        common.compile-core:
        
        compile-core:
        
        compile-test:
        
        install-junit4-taskdef:
        
        validate:
        
        test:
        [junit4:pickseed] Seed property 'tests.seed' already defined: 43A1116E7F98BED4
            [mkdir] Created dir: /home/rmuir/workspace/trunk/.caches/test-stats/core
           [junit4] <JUnit4> says ciao! Master seed: 43A1116E7F98BED4
           [junit4] Executing 367 suites with 1 JVM.
           [junit4] 
           [junit4] Started J0 PID(26780@beast).
           [junit4] Suite: org.apache.lucene.store.TestHugeRamFile
           [junit4] Completed in 1.26s, 1 test
           [junit4] 
           [junit4] Suite: org.apache.lucene.search.TestTimeLimitingCollector
           [junit4] Completed in 3.26s, 6 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.index.TestLazyProxSkipping
           [junit4] Completed in 0.68s, 2 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.index.TestPostingsOffsets
           [junit4] Completed in 0.78s, 11 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.search.TestRegexpQuery
           [junit4] Completed in 0.11s, 7 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.index.TestTryDelete
           [junit4] Completed in 0.03s, 3 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.util.TestDoubleBarrelLRUCache
           [junit4] Completed in 1.02s, 2 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.analysis.TestGraphTokenizers
           [junit4] Completed in 3.01s, 21 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging
           [junit4] Completed in 10.21s, 6 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.search.TestSearchAfter
           [junit4] Completed in 2.05s, 1 test
           [junit4] 
           [junit4] Suite: org.apache.lucene.index.TestNoMergeScheduler
           [junit4] Completed in 0.02s, 3 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.TestSearchForDuplicates
           [junit4] Completed in 0.08s, 1 test
           [junit4] 
           [junit4] Suite: org.apache.lucene.util.TestBytesRef
           [junit4] Completed in 0.02s, 5 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.index.Test4GBStoredFields
           [junit4] IGNOR/A 0.02s | Test4GBStoredFields.test
           [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly)
           [junit4] Completed in 0.03s, 1 test, 1 skipped
           [junit4] 
           [junit4] Suite: org.apache.lucene.util.TestAttributeSource
           [junit4] Completed in 0.02s, 5 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.index.TestIndexWriterMergePolicy
           [junit4] Completed in 2.52s, 7 tests
           [junit4] 
           [junit4] JVM J0: stdout was not empty, see: /home/rmuir/workspace/trunk/lucene/build/core/test/temp/junit4-J0-20130916_220432_129.sysout
           [junit4] >>> JVM J0: stdout (verbatim) ----
           [junit4] #
           [junit4] # A fatal error has been detected by the Java Runtime Environment:
           [junit4] #
           [junit4] #  SIGSEGV (0xb) at pc=0x00007f72952cd74d, pid=26780, tid=140129037821696
           [junit4] #
           [junit4] # JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43)
           [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 )
           [junit4] # Problematic frame:
           [junit4] # J  org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(I)Lorg/apache/lucene/index/Fields;
           [junit4] #
           [junit4] # Core dump written. Default location: /home/rmuir/workspace/trunk/lucene/build/core/test/J0/core or core.26780
           [junit4] #
           [junit4] # An error report file with more information is saved as:
           [junit4] # /home/rmuir/workspace/trunk/lucene/build/core/test/J0/hs_err_pid26780.log
           [junit4] #
           [junit4] # If you would like to submit a bug report, please visit:
           [junit4] #   http://bugreport.sun.com/bugreport/crash.jsp
           [junit4] #
           [junit4] <<< JVM J0: EOF ----
           [junit4] Execution time total: 28 seconds
        
        Show
        Robert Muir added a comment - I crashed again but with a core file (set 'ulimit -c unlimited'). zip file with core dump and hs_err is here: http://people.apache.org/~rmuir/crash.zip (its too large for JIRA, sorry) For some more context, it always happens fairly early in the test run: so when it doesnt crash at this exact point, you can ^C and run again until it does. Here was my commands with output below: (i tried to simplify the procedure to make it easy to reproduce, but its not easy, it took me quite a few tries) # note: we are pulling the exact revision that jenkins failed on, because things have changed in lucene codebase over the weekend!!!! svn co -r 1523179 https://svn.apache.org/repos/asf/lucene/dev/trunk # just go to core tests cd trunk/lucene/core # # now the following two commands: just run again and again until it crashes. # rm -rf ../../.caches/ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC" Here was the output: rmuir@beast:~/workspace/trunk/lucene/core$ rm -rf ../../.caches/ rmuir@beast:~/workspace/trunk/lucene/core$ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC" Buildfile: /home/rmuir/workspace/trunk/lucene/core/build.xml -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: ivy-availability-check: ivy-fail: ivy-configure: [ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ :: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: init: compile-core: compile-test-framework: ivy-availability-check: ivy-fail: ivy-configure: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: init: compile-lucene-core: ivy-availability-check: ivy-fail: ivy-configure: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: init: -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: compile-core: compile-codecs: [echo] Building codecs... ivy-availability-check: [echo] Building codecs... ivy-fail: ivy-configure: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: common.init: compile-lucene-core: init: -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: compile-core: -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: common.compile-core: compile-core: compile-test: install-junit4-taskdef: validate: test: [junit4:pickseed] Seed property 'tests.seed' already defined: 43A1116E7F98BED4 [mkdir] Created dir: /home/rmuir/workspace/trunk/.caches/test-stats/core [junit4] <JUnit4> says ciao! Master seed: 43A1116E7F98BED4 [junit4] Executing 367 suites with 1 JVM. [junit4] [junit4] Started J0 PID(26780@beast). [junit4] Suite: org.apache.lucene.store.TestHugeRamFile [junit4] Completed in 1.26s, 1 test [junit4] [junit4] Suite: org.apache.lucene.search.TestTimeLimitingCollector [junit4] Completed in 3.26s, 6 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestLazyProxSkipping [junit4] Completed in 0.68s, 2 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestPostingsOffsets [junit4] Completed in 0.78s, 11 tests [junit4] [junit4] Suite: org.apache.lucene.search.TestRegexpQuery [junit4] Completed in 0.11s, 7 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestTryDelete [junit4] Completed in 0.03s, 3 tests [junit4] [junit4] Suite: org.apache.lucene.util.TestDoubleBarrelLRUCache [junit4] Completed in 1.02s, 2 tests [junit4] [junit4] Suite: org.apache.lucene.analysis.TestGraphTokenizers [junit4] Completed in 3.01s, 21 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging [junit4] Completed in 10.21s, 6 tests [junit4] [junit4] Suite: org.apache.lucene.search.TestSearchAfter [junit4] Completed in 2.05s, 1 test [junit4] [junit4] Suite: org.apache.lucene.index.TestNoMergeScheduler [junit4] Completed in 0.02s, 3 tests [junit4] [junit4] Suite: org.apache.lucene.TestSearchForDuplicates [junit4] Completed in 0.08s, 1 test [junit4] [junit4] Suite: org.apache.lucene.util.TestBytesRef [junit4] Completed in 0.02s, 5 tests [junit4] [junit4] Suite: org.apache.lucene.index.Test4GBStoredFields [junit4] IGNOR/A 0.02s | Test4GBStoredFields.test [junit4] > Assumption #1: 'nightly' test group is disabled (@Nightly) [junit4] Completed in 0.03s, 1 test, 1 skipped [junit4] [junit4] Suite: org.apache.lucene.util.TestAttributeSource [junit4] Completed in 0.02s, 5 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestIndexWriterMergePolicy [junit4] Completed in 2.52s, 7 tests [junit4] [junit4] JVM J0: stdout was not empty, see: /home/rmuir/workspace/trunk/lucene/build/core/test/temp/junit4-J0-20130916_220432_129.sysout [junit4] >>> JVM J0: stdout (verbatim) ---- [junit4] # [junit4] # A fatal error has been detected by the Java Runtime Environment: [junit4] # [junit4] # SIGSEGV (0xb) at pc=0x00007f72952cd74d, pid=26780, tid=140129037821696 [junit4] # [junit4] # JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43) [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 ) [junit4] # Problematic frame: [junit4] # J org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(I)Lorg/apache/lucene/index/Fields; [junit4] # [junit4] # Core dump written. Default location: /home/rmuir/workspace/trunk/lucene/build/core/test/J0/core or core.26780 [junit4] # [junit4] # An error report file with more information is saved as: [junit4] # /home/rmuir/workspace/trunk/lucene/build/core/test/J0/hs_err_pid26780.log [junit4] # [junit4] # If you would like to submit a bug report, please visit: [junit4] # http://bugreport.sun.com/bugreport/crash.jsp [junit4] # [junit4] <<< JVM J0: EOF ---- [junit4] Execution time total: 28 seconds
        Hide
        Robert Muir added a comment -

        I ran again and here is the output where it does not crash, but instead corrupts:

        rmuir@beast:~/workspace/trunk/lucene/core$ rm -rf ../../.caches/
        rmuir@beast:~/workspace/trunk/lucene/core$ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC"
        Buildfile: /home/rmuir/workspace/trunk/lucene/core/build.xml
        
        -clover.disable:
        
        -clover.load:
        
        -clover.classpath:
        
        -clover.setup:
        
        clover:
        
        ivy-availability-check:
        
        ivy-fail:
        
        ivy-configure:
        [ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ ::
        [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml
        
        resolve:
        
        init:
        
        compile-core:
        
        compile-test-framework:
        
        ivy-availability-check:
        
        ivy-fail:
        
        ivy-configure:
        [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml
        
        resolve:
        
        init:
        
        compile-lucene-core:
        
        ivy-availability-check:
        
        ivy-fail:
        
        ivy-configure:
        [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml
        
        resolve:
        
        init:
        
        -clover.disable:
        
        -clover.load:
        
        -clover.classpath:
        
        -clover.setup:
        
        clover:
        
        compile-core:
        
        compile-codecs:
             [echo] Building codecs...
        
        ivy-availability-check:
             [echo] Building codecs...
        
        ivy-fail:
        
        ivy-configure:
        [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml
        
        resolve:
        
        common.init:
        
        compile-lucene-core:
        
        init:
        
        -clover.disable:
        
        -clover.load:
        
        -clover.classpath:
        
        -clover.setup:
        
        clover:
        
        compile-core:
        
        -clover.disable:
        
        -clover.load:
        
        -clover.classpath:
        
        -clover.setup:
        
        clover:
        
        common.compile-core:
        
        compile-core:
        
        compile-test:
        
        install-junit4-taskdef:
        
        validate:
        
        test:
        [junit4:pickseed] Seed property 'tests.seed' already defined: 43A1116E7F98BED4
            [mkdir] Created dir: /home/rmuir/workspace/trunk/.caches/test-stats/core
           [junit4] <JUnit4> says ciao! Master seed: 43A1116E7F98BED4
           [junit4] Executing 367 suites with 1 JVM.
           [junit4] 
           [junit4] Started J0 PID(27313@beast).
           [junit4] Suite: org.apache.lucene.store.TestHugeRamFile
           [junit4] Completed in 1.55s, 1 test
           [junit4] 
           [junit4] Suite: org.apache.lucene.search.TestTimeLimitingCollector
           [junit4] Completed in 3.23s, 6 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.index.TestLazyProxSkipping
           [junit4] Completed in 0.57s, 2 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.index.TestPostingsOffsets
           [junit4] Completed in 0.78s, 11 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.search.TestRegexpQuery
           [junit4] Completed in 0.11s, 7 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.index.TestTryDelete
           [junit4] Completed in 0.04s, 3 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.util.TestDoubleBarrelLRUCache
           [junit4] Completed in 1.02s, 2 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.analysis.TestGraphTokenizers
           [junit4] Completed in 3.05s, 21 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging
           [junit4] Completed in 10.27s, 6 tests
           [junit4] 
           [junit4] Suite: org.apache.lucene.search.TestSearchAfter
           [junit4]   1> CheckIndex failed
           [junit4]   1> Segments file=segments_2 numSegments=2 version=5.0 format=
           [junit4]   1>   1 of 2: name=_0 docCount=156
           [junit4]   1>     codec=Lucene45
           [junit4]   1>     compound=false
           [junit4]   1>     numFiles=30
           [junit4]   1>     size (MB)=0.157
           [junit4]   1>     diagnostics = {timestamp=1379384646861, os=Linux, os.version=3.5.0-27-generic, source=flush, lucene.version=5.0-SNAPSHOT, os.arch=amd64, java.version=1.7.0_40, java.vendor=Oracle Corporation}
           [junit4]   1>     no deletions
           [junit4]   1>     test: open reader.........OK
           [junit4]   1>     test: fields..............OK [15 fields]
           [junit4]   1>     test: field norms.........OK [2 fields]
           [junit4]   1>     test: terms, freq, prox...OK [8628 terms; 10659 terms/docs pairs; 423 tokens]
           [junit4]   1>     test: stored fields.......OK [1 total field count; avg 0.006 fields per doc]
           [junit4]   1>     test: term vectors........OK [3 total vector count; avg 1 term/freq vector fields per doc]
           [junit4]   1>     test: docvalues...........OK [5 docvalues fields; 1 BINARY; 2 NUMERIC; 2 SORTED; 0 SORTED_SET]
           [junit4]   1> 
           [junit4]   1>   2 of 2: name=_1 docCount=57
           [junit4]   1>     codec=Lucene45
           [junit4]   1>     compound=false
           [junit4]   1>     numFiles=30
           [junit4]   1>     size (MB)=0.06
           [junit4]   1>     diagnostics = {timestamp=1379384646900, os=Linux, os.version=3.5.0-27-generic, source=flush, lucene.version=5.0-SNAPSHOT, os.arch=amd64, java.version=1.7.0_40, java.vendor=Oracle Corporation}
           [junit4]   1>     no deletions
           [junit4]   1>     test: open reader.........OK
           [junit4]   1>     test: fields..............OK [15 fields]
           [junit4]   1>     test: field norms.........OK [2 fields]
           [junit4]   1>     test: terms, freq, prox...OK [3296 terms; 3969 terms/docs pairs; 227 tokens]
           [junit4]   1>     test: stored fields.......OK [2 total field count; avg 0.035 fields per doc]
           [junit4]   1>     test: term vectors........ERROR [null]
           [junit4]   1> java.lang.NegativeArraySizeException
           [junit4]   1> 	at org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(CompressingTermVectorsReader.java:433)
           [junit4]   1> 	at org.apache.lucene.index.SegmentReader.getTermVectors(SegmentReader.java:169)
           [junit4]   1> 	at org.apache.lucene.index.CheckIndex.testTermVectors(CheckIndex.java:1524)
           [junit4]   1> 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:611)
           [junit4]   1> 	at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:223)
           [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:641)
           [junit4]   1> 	at org.apache.lucene.search.TestSearchAfter.tearDown(TestSearchAfter.java:86)
           [junit4]   1> 	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
           [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:795)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
           [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
           [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
           [junit4]   1> 	at java.lang.Thread.run(Thread.java:724)
           [junit4]   1>     test: docvalues...........OK [5 docvalues fields; 1 BINARY; 2 NUMERIC; 2 SORTED; 0 SORTED_SET]
           [junit4]   1> FAILED
           [junit4]   1>     WARNING: fixIndex() would remove reference to this segment; full exception:
           [junit4]   1> java.lang.RuntimeException: Term Vector test failed
           [junit4]   1> 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:624)
           [junit4]   1> 	at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:223)
           [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:641)
           [junit4]   1> 	at org.apache.lucene.search.TestSearchAfter.tearDown(TestSearchAfter.java:86)
           [junit4]   1> 	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
           [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:795)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
           [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
           [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
           [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
           [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
           [junit4]   1> 	at java.lang.Thread.run(Thread.java:724)
           [junit4]   1> 
           [junit4]   1> WARNING: 1 broken segments (containing 57 documents) detected
           [junit4]   1> 
           [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSearchAfter -Dtests.method=testQueries -Dtests.seed=43A1116E7F98BED4 -Dtests.slow=true -Dtests.locale=th_TH_TH_#u-nu-thai -Dtests.timezone=JST -Dtests.file.encoding=ISO-8859-1
           [junit4] ERROR   2.09s | TestSearchAfter.testQueries <<<
           [junit4]    > Throwable #1: java.lang.RuntimeException: CheckIndex failed
           [junit4]    > 	at __randomizedtesting.SeedInfo.seed([43A1116E7F98BED4:1F2FDDB565F10B7A]:0)
           [junit4]    > 	at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:227)
           [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:641)
           [junit4]    > 	at org.apache.lucene.search.TestSearchAfter.tearDown(TestSearchAfter.java:86)
           [junit4]    > 	at java.lang.Thread.run(Thread.java:724)
           [junit4]   2> NOTE: test params are: codec=Lucene45: {short=PostingsFormat(name=FSTPulsing41), oddeven=PostingsFormat(name=Lucene41WithOrds), bytes=PostingsFormat(name=TestBloomFilteredLucene41Postings), byte=PostingsFormat(name=FSTPulsing41), int=PostingsFormat(name=TestBloomFilteredLucene41Postings), bytesval=PostingsFormat(name=FSTPulsing41), long=PostingsFormat(name=FSTPulsing41), double=PostingsFormat(name=Lucene41WithOrds), float=PostingsFormat(name=FSTPulsing41), english=PostingsFormat(name=Lucene41VarGapFixedInterval)}, docValues:{straightbytesdocvalues=DocValuesFormat(name=SimpleText), intdocvalues=DocValuesFormat(name=Lucene45), floatdocvalues=DocValuesFormat(name=Disk), sortedbytesdocvalues=DocValuesFormat(name=Disk), sortedbytesdocvaluesval=DocValuesFormat(name=SimpleText)}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {oddeven=DFR GBZ(0.3), english=DFR I(F)BZ(0.3)}, locale=th_TH_TH_#u-nu-thai, timezone=JST
           [junit4]   2> NOTE: Linux 3.5.0-27-generic amd64/Oracle Corporation 1.7.0_40 (64-bit)/cpus=8,threads=1,free=167266488,total=400556032
           [junit4]   2> NOTE: All tests run in this JVM: [TestHugeRamFile, TestTimeLimitingCollector, TestLazyProxSkipping, TestPostingsOffsets, TestRegexpQuery, TestTryDelete, TestDoubleBarrelLRUCache, TestGraphTokenizers, TestIndexWriterMerging, TestSearchAfter]
           [junit4] Completed in 2.11s, 1 test, 1 error <<< FAILURES!
        
        Show
        Robert Muir added a comment - I ran again and here is the output where it does not crash, but instead corrupts: rmuir@beast:~/workspace/trunk/lucene/core$ rm -rf ../../.caches/ rmuir@beast:~/workspace/trunk/lucene/core$ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC" Buildfile: /home/rmuir/workspace/trunk/lucene/core/build.xml -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: ivy-availability-check: ivy-fail: ivy-configure: [ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ :: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: init: compile-core: compile-test-framework: ivy-availability-check: ivy-fail: ivy-configure: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: init: compile-lucene-core: ivy-availability-check: ivy-fail: ivy-configure: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: init: -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: compile-core: compile-codecs: [echo] Building codecs... ivy-availability-check: [echo] Building codecs... ivy-fail: ivy-configure: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: common.init: compile-lucene-core: init: -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: compile-core: -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: common.compile-core: compile-core: compile-test: install-junit4-taskdef: validate: test: [junit4:pickseed] Seed property 'tests.seed' already defined: 43A1116E7F98BED4 [mkdir] Created dir: /home/rmuir/workspace/trunk/.caches/test-stats/core [junit4] <JUnit4> says ciao! Master seed: 43A1116E7F98BED4 [junit4] Executing 367 suites with 1 JVM. [junit4] [junit4] Started J0 PID(27313@beast). [junit4] Suite: org.apache.lucene.store.TestHugeRamFile [junit4] Completed in 1.55s, 1 test [junit4] [junit4] Suite: org.apache.lucene.search.TestTimeLimitingCollector [junit4] Completed in 3.23s, 6 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestLazyProxSkipping [junit4] Completed in 0.57s, 2 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestPostingsOffsets [junit4] Completed in 0.78s, 11 tests [junit4] [junit4] Suite: org.apache.lucene.search.TestRegexpQuery [junit4] Completed in 0.11s, 7 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestTryDelete [junit4] Completed in 0.04s, 3 tests [junit4] [junit4] Suite: org.apache.lucene.util.TestDoubleBarrelLRUCache [junit4] Completed in 1.02s, 2 tests [junit4] [junit4] Suite: org.apache.lucene.analysis.TestGraphTokenizers [junit4] Completed in 3.05s, 21 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging [junit4] Completed in 10.27s, 6 tests [junit4] [junit4] Suite: org.apache.lucene.search.TestSearchAfter [junit4] 1> CheckIndex failed [junit4] 1> Segments file=segments_2 numSegments=2 version=5.0 format= [junit4] 1> 1 of 2: name=_0 docCount=156 [junit4] 1> codec=Lucene45 [junit4] 1> compound=false [junit4] 1> numFiles=30 [junit4] 1> size (MB)=0.157 [junit4] 1> diagnostics = {timestamp=1379384646861, os=Linux, os.version=3.5.0-27-generic, source=flush, lucene.version=5.0-SNAPSHOT, os.arch=amd64, java.version=1.7.0_40, java.vendor=Oracle Corporation} [junit4] 1> no deletions [junit4] 1> test: open reader.........OK [junit4] 1> test: fields..............OK [15 fields] [junit4] 1> test: field norms.........OK [2 fields] [junit4] 1> test: terms, freq, prox...OK [8628 terms; 10659 terms/docs pairs; 423 tokens] [junit4] 1> test: stored fields.......OK [1 total field count; avg 0.006 fields per doc] [junit4] 1> test: term vectors........OK [3 total vector count; avg 1 term/freq vector fields per doc] [junit4] 1> test: docvalues...........OK [5 docvalues fields; 1 BINARY; 2 NUMERIC; 2 SORTED; 0 SORTED_SET] [junit4] 1> [junit4] 1> 2 of 2: name=_1 docCount=57 [junit4] 1> codec=Lucene45 [junit4] 1> compound=false [junit4] 1> numFiles=30 [junit4] 1> size (MB)=0.06 [junit4] 1> diagnostics = {timestamp=1379384646900, os=Linux, os.version=3.5.0-27-generic, source=flush, lucene.version=5.0-SNAPSHOT, os.arch=amd64, java.version=1.7.0_40, java.vendor=Oracle Corporation} [junit4] 1> no deletions [junit4] 1> test: open reader.........OK [junit4] 1> test: fields..............OK [15 fields] [junit4] 1> test: field norms.........OK [2 fields] [junit4] 1> test: terms, freq, prox...OK [3296 terms; 3969 terms/docs pairs; 227 tokens] [junit4] 1> test: stored fields.......OK [2 total field count; avg 0.035 fields per doc] [junit4] 1> test: term vectors........ERROR [null] [junit4] 1> java.lang.NegativeArraySizeException [junit4] 1> at org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(CompressingTermVectorsReader.java:433) [junit4] 1> at org.apache.lucene.index.SegmentReader.getTermVectors(SegmentReader.java:169) [junit4] 1> at org.apache.lucene.index.CheckIndex.testTermVectors(CheckIndex.java:1524) [junit4] 1> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:611) [junit4] 1> at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:223) [junit4] 1> at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:641) [junit4] 1> at org.apache.lucene.search.TestSearchAfter.tearDown(TestSearchAfter.java:86) [junit4] 1> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) [junit4] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit4] 1> at java.lang.reflect.Method.invoke(Method.java:606) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:795) [junit4] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) [junit4] 1> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) [junit4] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) [junit4] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) [junit4] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) [junit4] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) [junit4] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4] 1> at java.lang.Thread.run(Thread.java:724) [junit4] 1> test: docvalues...........OK [5 docvalues fields; 1 BINARY; 2 NUMERIC; 2 SORTED; 0 SORTED_SET] [junit4] 1> FAILED [junit4] 1> WARNING: fixIndex() would remove reference to this segment; full exception: [junit4] 1> java.lang.RuntimeException: Term Vector test failed [junit4] 1> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:624) [junit4] 1> at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:223) [junit4] 1> at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:641) [junit4] 1> at org.apache.lucene.search.TestSearchAfter.tearDown(TestSearchAfter.java:86) [junit4] 1> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) [junit4] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit4] 1> at java.lang.reflect.Method.invoke(Method.java:606) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:795) [junit4] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) [junit4] 1> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) [junit4] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) [junit4] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) [junit4] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) [junit4] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) [junit4] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4] 1> at java.lang.Thread.run(Thread.java:724) [junit4] 1> [junit4] 1> WARNING: 1 broken segments (containing 57 documents) detected [junit4] 1> [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSearchAfter -Dtests.method=testQueries -Dtests.seed=43A1116E7F98BED4 -Dtests.slow=true -Dtests.locale=th_TH_TH_#u-nu-thai -Dtests.timezone=JST -Dtests.file.encoding=ISO-8859-1 [junit4] ERROR 2.09s | TestSearchAfter.testQueries <<< [junit4] > Throwable #1: java.lang.RuntimeException: CheckIndex failed [junit4] > at __randomizedtesting.SeedInfo.seed([43A1116E7F98BED4:1F2FDDB565F10B7A]:0) [junit4] > at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:227) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:641) [junit4] > at org.apache.lucene.search.TestSearchAfter.tearDown(TestSearchAfter.java:86) [junit4] > at java.lang.Thread.run(Thread.java:724) [junit4] 2> NOTE: test params are: codec=Lucene45: {short=PostingsFormat(name=FSTPulsing41), oddeven=PostingsFormat(name=Lucene41WithOrds), bytes=PostingsFormat(name=TestBloomFilteredLucene41Postings), byte=PostingsFormat(name=FSTPulsing41), int=PostingsFormat(name=TestBloomFilteredLucene41Postings), bytesval=PostingsFormat(name=FSTPulsing41), long=PostingsFormat(name=FSTPulsing41), double=PostingsFormat(name=Lucene41WithOrds), float=PostingsFormat(name=FSTPulsing41), english=PostingsFormat(name=Lucene41VarGapFixedInterval)}, docValues:{straightbytesdocvalues=DocValuesFormat(name=SimpleText), intdocvalues=DocValuesFormat(name=Lucene45), floatdocvalues=DocValuesFormat(name=Disk), sortedbytesdocvalues=DocValuesFormat(name=Disk), sortedbytesdocvaluesval=DocValuesFormat(name=SimpleText)}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {oddeven=DFR GBZ(0.3), english=DFR I(F)BZ(0.3)}, locale=th_TH_TH_#u-nu-thai, timezone=JST [junit4] 2> NOTE: Linux 3.5.0-27-generic amd64/Oracle Corporation 1.7.0_40 (64-bit)/cpus=8,threads=1,free=167266488,total=400556032 [junit4] 2> NOTE: All tests run in this JVM: [TestHugeRamFile, TestTimeLimitingCollector, TestLazyProxSkipping, TestPostingsOffsets, TestRegexpQuery, TestTryDelete, TestDoubleBarrelLRUCache, TestGraphTokenizers, TestIndexWriterMerging, TestSearchAfter] [junit4] Completed in 2.11s, 1 test, 1 error <<< FAILURES!
        Hide
        Robert Muir added a comment -

        FYI: I ran this procedure about 10 times with the suggested workaround from https://bugs.openjdk.java.net/browse/JDK-8024830 and tests always pass:

        -XX:-UseLoopPredicate

        Show
        Robert Muir added a comment - FYI: I ran this procedure about 10 times with the suggested workaround from https://bugs.openjdk.java.net/browse/JDK-8024830 and tests always pass: -XX:-UseLoopPredicate
        Hide
        Dawid Weiss added a comment -

        Is there any way to subscribe to that openjdk JIRA to watch issues? I don't see any way to do it

        Show
        Dawid Weiss added a comment - Is there any way to subscribe to that openjdk JIRA to watch issues? I don't see any way to do it
        Hide
        Robert Muir added a comment -

        I cant figure out how to comment or anything. I wanted to just include this information to try to help the openjdk developers reproduce the bug and so on.

        But for now I know I can do it here...

        Show
        Robert Muir added a comment - I cant figure out how to comment or anything. I wanted to just include this information to try to help the openjdk developers reproduce the bug and so on. But for now I know I can do it here...
        Hide
        Dawid Weiss added a comment -

        I don't think you can comment or do anything until you're logged in. And you can't log in because it seems like a registration-closed system...

        Show
        Dawid Weiss added a comment - I don't think you can comment or do anything until you're logged in. And you can't log in because it seems like a registration-closed system...
        Hide
        Dawid Weiss added a comment -

        For the record, this is a different bug than LUCENE-5168 (or a different manifestation of the same underlying bug?) because -XX:-UseLoopPredicate doesn't affect LUCENE-5168 (it still miscompiles).

        Show
        Dawid Weiss added a comment - For the record, this is a different bug than LUCENE-5168 (or a different manifestation of the same underlying bug?) because -XX:-UseLoopPredicate doesn't affect LUCENE-5168 (it still miscompiles).
        Hide
        Robert Muir added a comment -

        Yes thats a separate, older bug with the only workaround of -XX:-DoEscapeAnalysis correct?

        If somehow these two bugs decide to drink beers together in the same jenkins run it will fry my brain.

        Show
        Robert Muir added a comment - Yes thats a separate, older bug with the only workaround of -XX:-DoEscapeAnalysis correct? If somehow these two bugs decide to drink beers together in the same jenkins run it will fry my brain.
        Hide
        Dawid Weiss added a comment -

        -XX:-DoEscapeAnalysis

        Yes, the bug is somewhere in the escape analysis/ ideal graph optimization chain so disabling it does the job. The first JVM revision that had this bug is jdk7u04, versions before that one are not affected.

        The good news is that this problem seems to affect only 32-bit JVMs, I couldn't reproduce it on a 64-bit JVM. It may be due to a larger number of CPU registers (so spill stack variables are not needed and thus there are no missed updates...). If this is the case then the problem may still occur somewhere else, depending on the method/ inlines/ switches/, you name it...

        The current status of LUCENE-5168 is that one of hotspot core developers (Vladimir Kozlov) has collected the information needed for debugging, we know after which commit to hotpot the problem starts appearing and we know it's still present both in jdk7 and in jdk8. I'd have to devote myself full time to getting up to speed with internal hotspot code to figure out anything more – even the logs get really hairy (and huge) at the detail level we're looking at the problem. Hopefully Vladimir will be able to look into it at some point.

        Show
        Dawid Weiss added a comment - -XX:-DoEscapeAnalysis Yes, the bug is somewhere in the escape analysis/ ideal graph optimization chain so disabling it does the job. The first JVM revision that had this bug is jdk7u04, versions before that one are not affected. The good news is that this problem seems to affect only 32-bit JVMs, I couldn't reproduce it on a 64-bit JVM. It may be due to a larger number of CPU registers (so spill stack variables are not needed and thus there are no missed updates...). If this is the case then the problem may still occur somewhere else, depending on the method/ inlines/ switches/, you name it... The current status of LUCENE-5168 is that one of hotspot core developers (Vladimir Kozlov) has collected the information needed for debugging, we know after which commit to hotpot the problem starts appearing and we know it's still present both in jdk7 and in jdk8. I'd have to devote myself full time to getting up to speed with internal hotspot code to figure out anything more – even the logs get really hairy (and huge) at the detail level we're looking at the problem. Hopefully Vladimir will be able to look into it at some point.
        Hide
        Robert Muir added a comment -

        Attached 'crashFaster.patch', which you can apply to the checkout.

        It disables any unnecessary tests to reproduce the bug and speed up bisection.

        Show
        Robert Muir added a comment - Attached 'crashFaster.patch', which you can apply to the checkout. It disables any unnecessary tests to reproduce the bug and speed up bisection.
        Robert Muir made changes -
        Attachment crashFaster.patch [ 12603875 ]
        Hide
        Robert Muir added a comment -

        I noticed on my system: if the index goes corrupt it tends to happen in TestSearchAfter, but when it SIGSEGVs sometimes that happens in TestLucene45DocValuesFormat (sometimes it corrupts here, too)

        So I updated the patch to include that test too, to try to make it easier to reproduce... (it still takes several tries for me)

        Show
        Robert Muir added a comment - I noticed on my system: if the index goes corrupt it tends to happen in TestSearchAfter, but when it SIGSEGVs sometimes that happens in TestLucene45DocValuesFormat (sometimes it corrupts here, too) So I updated the patch to include that test too, to try to make it easier to reproduce... (it still takes several tries for me)
        Robert Muir made changes -
        Attachment crashFaster2.0.patch [ 12603884 ]
        Dawid Weiss made changes -
        Link This issue relates to LUCENE-5241 [ LUCENE-5241 ]
        Hide
        Dawid Weiss added a comment -

        Don't know for sure if they're related or not but linking nonetheless.

        Show
        Dawid Weiss added a comment - Don't know for sure if they're related or not but linking nonetheless.
        Hide
        Littlestar added a comment - - edited

        java 7u40? mybe also corrupt docvalue.
        ================
        LUCENE-5218 background merge hit exception && Caused by: java.lang.ArrayIndexOutOfBoundsException

        Show
        Littlestar added a comment - - edited java 7u40? mybe also corrupt docvalue. ================ LUCENE-5218 background merge hit exception && Caused by: java.lang.ArrayIndexOutOfBoundsException
        Hide
        Bill Bell added a comment -

        It appears this happens on 7u40 64-bit too. See https://bugs.openjdk.java.net/browse/JDK-8024830

        Am I reading this wrong?

        Start failing around hs24-b21:

        [junit4] # SIGSEGV (0xb) at pc=0xfffffd7ff91d9f7d, pid=23810, tid=343
        [junit4] #
        [junit4] # JRE version: Java(TM) SE Runtime Environment (8.0-b54)
        [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b21 mixed mode solaris-amd64 )
        [junit4] # Problematic frame:
        [junit4] # J org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(I)Lorg/apache/lucene/index/Fields;
        [junit4] #

        Note, first 7u40 build b01 has hs24-b24.

        Next, I will try to find changeset.

        Show
        Bill Bell added a comment - It appears this happens on 7u40 64-bit too. See https://bugs.openjdk.java.net/browse/JDK-8024830 Am I reading this wrong? Start failing around hs24-b21: [junit4] # SIGSEGV (0xb) at pc=0xfffffd7ff91d9f7d, pid=23810, tid=343 [junit4] # [junit4] # JRE version: Java(TM) SE Runtime Environment (8.0-b54) [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b21 mixed mode solaris-amd64 ) [junit4] # Problematic frame: [junit4] # J org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(I)Lorg/apache/lucene/index/Fields; [junit4] # Note, first 7u40 build b01 has hs24-b24. Next, I will try to find changeset.
        Hide
        Littlestar added a comment -

        Is there a small corrupt index can reproduce the problem by CheckIndex, Thanks.

        Show
        Littlestar added a comment - Is there a small corrupt index can reproduce the problem by CheckIndex, Thanks.
        Hide
        mjuarez added a comment -

        It seems like Oracle might punt on this until jdk8 comes around. Which would preclude any safe use of Lucene for jdk7u40 and higher.

        From the last comment on the JDK issue:

        https://bugs.openjdk.java.net/browse/JDK-8024830

        Azeem Jiva added a comment - 2013-10-17 10:06 - edited

        8-compiler-defer: It requires a very specific set of circumstances to have this occur. Request to defer and fix when the issue is better understood. Fix for JDK8 update.

        Show
        mjuarez added a comment - It seems like Oracle might punt on this until jdk8 comes around. Which would preclude any safe use of Lucene for jdk7u40 and higher. From the last comment on the JDK issue: https://bugs.openjdk.java.net/browse/JDK-8024830 Azeem Jiva added a comment - 2013-10-17 10:06 - edited 8-compiler-defer: It requires a very specific set of circumstances to have this occur. Request to defer and fix when the issue is better understood. Fix for JDK8 update.
        Hide
        Dawid Weiss added a comment -

        This is really disappointing. It's like saying we'll fix your car but for now, in very specific circumstances, it can blow right into your face...

        Show
        Dawid Weiss added a comment - This is really disappointing. It's like saying we'll fix your car but for now, in very specific circumstances, it can blow right into your face...
        Hide
        Uwe Schindler added a comment -

        Hi,

        the bug did not appear anymore recently on Jenkins. Michael McCandless refactored the horrible big method around CompressingTermVectorsReader, so the bug seems to be fixed.

        In general we should not create such big methods Eclipse "Extract method" is wonderful!

        Show
        Uwe Schindler added a comment - Hi, the bug did not appear anymore recently on Jenkins. Michael McCandless refactored the horrible big method around CompressingTermVectorsReader, so the bug seems to be fixed. In general we should not create such big methods Eclipse "Extract method" is wonderful!
        Hide
        Dawid Weiss added a comment -

        I'm talking about the JVM bug, not the particular trigger that caused it in Lucene code. We can dodge those errors but sooner or later somebody will trip over them anyway.

        Show
        Dawid Weiss added a comment - I'm talking about the JVM bug, not the particular trigger that caused it in Lucene code. We can dodge those errors but sooner or later somebody will trip over them anyway.
        Hide
        Uwe Schindler added a comment -

        Of course

        Show
        Uwe Schindler added a comment - Of course
        Hide
        Dawid Weiss added a comment -

        Vladimir just pinged me that this issue is (hopefully) fixed, the webrev is here:
        http://cr.openjdk.java.net/~kvn/8024830/webrev/

        I'll test this patch (with jdk7u40) and let you know if I can (or can no longer) reproduce the issue.

        JDK bug ref: https://bugs.openjdk.java.net/browse/JDK-8024830

        Show
        Dawid Weiss added a comment - Vladimir just pinged me that this issue is (hopefully) fixed, the webrev is here: http://cr.openjdk.java.net/~kvn/8024830/webrev/ I'll test this patch (with jdk7u40) and let you know if I can (or can no longer) reproduce the issue. JDK bug ref: https://bugs.openjdk.java.net/browse/JDK-8024830
        Hide
        Dawid Weiss added a comment -

        I confirm that this patch fixes the problem. I've tested svn rev. 1523179 (trunk) against jdk8-b114 with and without Vladimir's patch. Without the patch the test sequence ends about 50% of the time in a sigsegv. With the patch all executions ended without any errors.

        Note that the problem only affects CPUs with the AVX extension. A workaround for affected VMs is to disable vectorization with -XX:-UseSuperWord.

        Show
        Dawid Weiss added a comment - I confirm that this patch fixes the problem. I've tested svn rev. 1523179 (trunk) against jdk8-b114 with and without Vladimir's patch. Without the patch the test sequence ends about 50% of the time in a sigsegv. With the patch all executions ended without any errors. Note that the problem only affects CPUs with the AVX extension. A workaround for affected VMs is to disable vectorization with -XX:-UseSuperWord.
        Hide
        Robert Muir added a comment -

        The fix is committed to openjdk trunk.

        Show
        Robert Muir added a comment - The fix is committed to openjdk trunk.
        Robert Muir made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Done [ 11 ]
        Hide
        Uwe Schindler added a comment -

        Hi,
        I figured out: Policeman Jenkins has no vector extensions! So the bug cannot affect Policeman Jenkins (has Lynfield CPU, no Sanybridge). So the bug causing the sigsegv here is not the same bug as the one we hit on Jenkins (that corrumpted term vectors).
        Of course it could be that the vector extensions bug also affected other CPUs, but I have no idea of that code.

        Show
        Uwe Schindler added a comment - Hi, I figured out: Policeman Jenkins has no vector extensions! So the bug cannot affect Policeman Jenkins (has Lynfield CPU, no Sanybridge). So the bug causing the sigsegv here is not the same bug as the one we hit on Jenkins (that corrumpted term vectors). Of course it could be that the vector extensions bug also affected other CPUs, but I have no idea of that code.
        Hide
        Robert Muir added a comment -

        Uwe, i just read the issue slightly, but maybe it would be good to ask Vladimir if the bug really only impacts AVX instructions.

        See the patch, perhaps it impacts 128bit SSE as well and is just easier to trip with the AVX ones since they have 256bit extensions?

        Show
        Robert Muir added a comment - Uwe, i just read the issue slightly, but maybe it would be good to ask Vladimir if the bug really only impacts AVX instructions. See the patch, perhaps it impacts 128bit SSE as well and is just easier to trip with the AVX ones since they have 256bit extensions?
        Hide
        Dawid Weiss added a comment -

        Vladimir said to me I needed an AVX CPU to test the patch. Odd. I may take a look at the openjdk code later on to see when this optimization is enabled.

        Show
        Dawid Weiss added a comment - Vladimir said to me I needed an AVX CPU to test the patch. Odd. I may take a look at the openjdk code later on to see when this optimization is enabled.
        Hide
        Dawid Weiss added a comment -

        We inspected the code with Robert Muir and came to the conclusion that this bug may also affect SSE machines. Since hotspot code is very complex I also contacted Vladimir and he kindly replied that yes, this is the case.

        And you are correct the problem also affects SSE for which we can generate 16 bytes vectors which are larger then 8 bytes stack frame alignment we have. Actually in the bug there were only 16 bytes vectors max.
        I think it may be related to register masks we use on AVX machine, they are larger to be able map 32 bytes vectors. But again with SSE and 16 bytes vectors you can hit the same 8024830 problem but may be much less frequent.

        Show
        Dawid Weiss added a comment - We inspected the code with Robert Muir and came to the conclusion that this bug may also affect SSE machines. Since hotspot code is very complex I also contacted Vladimir and he kindly replied that yes, this is the case. And you are correct the problem also affects SSE for which we can generate 16 bytes vectors which are larger then 8 bytes stack frame alignment we have. Actually in the bug there were only 16 bytes vectors max. I think it may be related to register masks we use on AVX machine, they are larger to be able map 32 bytes vectors. But again with SSE and 16 bytes vectors you can hit the same 8024830 problem but may be much less frequent.
        Uwe Schindler made changes -
        Labels Java7 Java8
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        58d 20h 38m 1 Robert Muir 12/Nov/13 03:29

          People

          • Assignee:
            Unassigned
            Reporter:
            Robert Muir
          • Votes:
            2 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development