Lucene - Core
  1. Lucene - Core
  2. LUCENE-5212

java 7u40 causes sigsegv and corrupt term vectors

        Details

        • Type: Bug Bug
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Done
        • Affects Version/s: None
        • Fix Version/s: None
        • Component/s: None
        • Labels:
        • Lucene Fields:
          New
        1. Text File
          crashFaster.patch
          1 kB
          Robert Muir
        2. Text File
          crashFaster2.0.patch
          1 kB
          Robert Muir
        3. Text File
          hs_err_pid32714.log
          58 kB
          Robert Muir
        4. Text File
          jenkins.txt
          124 kB
          Robert Muir

          Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. LUCENE-5241 Hotspot crash, SIGSEGV with Java 1.6u45

          • Major - Major loss of function.
          • Resolved

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            Robert Muir added a comment -

            This has happened twice in jenkins since Uwe upgraded, so i tried to reproduce myself.

            With update 25, no issues.
            So i ugpraded to update 40: on the first try with the jenkins commandline:

            rmuir@beast:~/workspace/lucene-trunk/lucene/core$ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dtests.dynamicAssignmentRatio=0 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC"

               [junit4] #
   [junit4] # A fatal error has been detected by the Java Runtime Environment:
   [junit4] #
   [junit4] #  SIGSEGV (0xb) at pc=0x00007f163d2d34dd, pid=32714, tid=139732803393280
   [junit4] #
   [junit4] # JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43)
   [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 )
   [junit4] # Problematic frame:
   [junit4] # J  org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(I)Lorg/apache/lucene/index/Fields;
   [junit4] #
   [junit4] # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
   [junit4] #
   [junit4] # An error report file with more information is saved as:
   [junit4] # /home/rmuir/workspace/lucene-trunk/lucene/build/core/test/J0/hs_err_pid32714.log
            Show
            Robert Muir added a comment - This has happened twice in jenkins since Uwe upgraded, so i tried to reproduce myself. With update 25, no issues. So i ugpraded to update 40: on the first try with the jenkins commandline: rmuir@beast:~/workspace/lucene-trunk/lucene/core$ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dtests.dynamicAssignmentRatio=0 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC" [junit4] # [junit4] # A fatal error has been detected by the Java Runtime Environment: [junit4] # [junit4] # SIGSEGV (0xb) at pc=0x00007f163d2d34dd, pid=32714, tid=139732803393280 [junit4] # [junit4] # JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43) [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 ) [junit4] # Problematic frame: [junit4] # J org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(I)Lorg/apache/lucene/index/Fields; [junit4] # [junit4] # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again [junit4] # [junit4] # An error report file with more information is saved as: [junit4] # /home/rmuir/workspace/lucene-trunk/lucene/build/core/test/J0/hs_err_pid32714.log
            Hide
            Permalink
            Robert Muir added a comment -

            Attached is the hs_err

            Show
            Robert Muir added a comment - Attached is the hs_err
            Hide
            Permalink
            Robert Muir added a comment -

            i ran this 3 times... first 2 times SIGSEGV, third time it passed. so its not totally consistent but seems to reproduce.

            I will now try to reproduce the other seed from this morning.

            Show
            Robert Muir added a comment - i ran this 3 times... first 2 times SIGSEGV, third time it passed. so its not totally consistent but seems to reproduce. I will now try to reproduce the other seed from this morning.
            Hide
            Permalink
            Robert Muir added a comment -

            The other seed i cant reproduce:

            (from lucene/classification: ant test -Dtests.seed=68F81CB00E47182B -Dtests.jvms=2 -Dtests.dynamicAssignmentRatio=0 -Dargs="-XX:-UseCompressedOops -XX:+UseG1GC")

            Note you have to rm -rf your .caches (sorry, dynamicAssignmentRatio=0 doesnt quite work), or the tests wont be executed in the same order as jenkins did.

            Still i cant reproduce that one, but the later seed above in core tests is easier anyway since it only launches 1 JVM and it SIGSEGVs.

            Show
            Robert Muir added a comment - The other seed i cant reproduce: (from lucene/classification: ant test -Dtests.seed=68F81CB00E47182B -Dtests.jvms=2 -Dtests.dynamicAssignmentRatio=0 -Dargs="-XX:-UseCompressedOops -XX:+UseG1GC") Note you have to rm -rf your .caches (sorry, dynamicAssignmentRatio=0 doesnt quite work), or the tests wont be executed in the same order as jenkins did. Still i cant reproduce that one, but the later seed above in core tests is easier anyway since it only launches 1 JVM and it SIGSEGVs.
            Hide
            Permalink
            Robert Muir added a comment -

            attached is original jenkins log (they get removed from the server pretty fast).

            in jenkins on os X, it didnt crash but had corrupt term vectors in several unrelated tests.

            Show
            Robert Muir added a comment - attached is original jenkins log (they get removed from the server pretty fast). in jenkins on os X, it didnt crash but had corrupt term vectors in several unrelated tests.
            Hide
            Permalink
            Dawid Weiss added a comment -

            > (sorry, dynamicAssignmentRatio=0 doesnt quite work),

            The lack of determinism may be due to work stealing – dynamicAssignmentRatio=0 preallocates tasks to JVMs, while work stealing may actually assign differently on every run.

            Perhaps we should run jenkins jobs with dynamicAssignmentRatio=0... but then all the benefits of work time balancing among cores would be gone and fixed task assignment only helps in very specific bug cases (inter-suite dependency or JVM bugs). Hard to tell.

            Show
            Dawid Weiss added a comment - > (sorry, dynamicAssignmentRatio=0 doesnt quite work), The lack of determinism may be due to work stealing – dynamicAssignmentRatio=0 preallocates tasks to JVMs, while work stealing may actually assign differently on every run. Perhaps we should run jenkins jobs with dynamicAssignmentRatio=0... but then all the benefits of work time balancing among cores would be gone and fixed task assignment only helps in very specific bug cases (inter-suite dependency or JVM bugs). Hard to tell.
            Hide
            Permalink
            Robert Muir added a comment -

            I'm not complaining, I'm happy we are able to reproduce the 'whole run' in any shape or form. one extra command does not bother me, we shouldnt slow all of our jenkins jobs down to make jvm bug-debugging easier

            Show
            Robert Muir added a comment - I'm not complaining, I'm happy we are able to reproduce the 'whole run' in any shape or form. one extra command does not bother me, we shouldnt slow all of our jenkins jobs down to make jvm bug-debugging easier
            Hide
            Permalink
            Michael McCandless added a comment -

            in jenkins on os X, it didnt crash but had corrupt term vectors in several unrelated tests.

            Very scary that this can also be a "silently introduces index corruption" JVM bug! I'd much prefer the sudden death SEGV.

            Show
            Michael McCandless added a comment - in jenkins on os X, it didnt crash but had corrupt term vectors in several unrelated tests. Very scary that this can also be a "silently introduces index corruption" JVM bug! I'd much prefer the sudden death SEGV.
            Hide
            Permalink
            Robert Muir added a comment -

            https://bugs.openjdk.java.net/browse/JDK-8024830

            Show
            Robert Muir added a comment - https://bugs.openjdk.java.net/browse/JDK-8024830
            Hide
            Permalink
            Robert Muir added a comment -

            I crashed again but with a core file (set 'ulimit -c unlimited').

            zip file with core dump and hs_err is here: http://people.apache.org/~rmuir/crash.zip (its too large for JIRA, sorry)

            For some more context, it always happens fairly early in the test run:
            so when it doesnt crash at this exact point, you can ^C and run again until it does.

            Here was my commands with output below: (i tried to simplify the procedure to make it easy to reproduce, but its not easy, it took me quite a few tries)

            # note: we are pulling the exact revision that jenkins failed on, because things have changed in lucene codebase over the weekend!!!!
svn co -r 1523179 https://svn.apache.org/repos/asf/lucene/dev/trunk
# just go to core tests
cd trunk/lucene/core
#
# now the following two commands: just run again and again until it crashes.
#
rm -rf ../../.caches/
ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC"

            Here was the output:

            rmuir@beast:~/workspace/trunk/lucene/core$ rm -rf ../../.caches/
rmuir@beast:~/workspace/trunk/lucene/core$ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC"
Buildfile: /home/rmuir/workspace/trunk/lucene/core/build.xml

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

ivy-availability-check:

ivy-fail:

ivy-configure:
[ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml

resolve:

init:

compile-core:

compile-test-framework:

ivy-availability-check:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml

resolve:

init:

compile-lucene-core:

ivy-availability-check:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml

resolve:

init:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

compile-core:

compile-codecs:
     [echo] Building codecs...

ivy-availability-check:
     [echo] Building codecs...

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml

resolve:

common.init:

compile-lucene-core:

init:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

compile-core:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

common.compile-core:

compile-core:

compile-test:

install-junit4-taskdef:

validate:

test:
[junit4:pickseed] Seed property 'tests.seed' already defined: 43A1116E7F98BED4
    [mkdir] Created dir: /home/rmuir/workspace/trunk/.caches/test-stats/core
   [junit4] <JUnit4> says ciao! Master seed: 43A1116E7F98BED4
   [junit4] Executing 367 suites with 1 JVM.
   [junit4] 
   [junit4] Started J0 PID(26780@beast).
   [junit4] Suite: org.apache.lucene.store.TestHugeRamFile
   [junit4] Completed in 1.26s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTimeLimitingCollector
   [junit4] Completed in 3.26s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestLazyProxSkipping
   [junit4] Completed in 0.68s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPostingsOffsets
   [junit4] Completed in 0.78s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestRegexpQuery
   [junit4] Completed in 0.11s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTryDelete
   [junit4] Completed in 0.03s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestDoubleBarrelLRUCache
   [junit4] Completed in 1.02s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestGraphTokenizers
   [junit4] Completed in 3.01s, 21 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging
   [junit4] Completed in 10.21s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSearchAfter
   [junit4] Completed in 2.05s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNoMergeScheduler
   [junit4] Completed in 0.02s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestSearchForDuplicates
   [junit4] Completed in 0.08s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestBytesRef
   [junit4] Completed in 0.02s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.Test4GBStoredFields
   [junit4] IGNOR/A 0.02s | Test4GBStoredFields.test
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly)
   [junit4] Completed in 0.03s, 1 test, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestAttributeSource
   [junit4] Completed in 0.02s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterMergePolicy
   [junit4] Completed in 2.52s, 7 tests
   [junit4] 
   [junit4] JVM J0: stdout was not empty, see: /home/rmuir/workspace/trunk/lucene/build/core/test/temp/junit4-J0-20130916_220432_129.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] #
   [junit4] # A fatal error has been detected by the Java Runtime Environment:
   [junit4] #
   [junit4] #  SIGSEGV (0xb) at pc=0x00007f72952cd74d, pid=26780, tid=140129037821696
   [junit4] #
   [junit4] # JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43)
   [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 )
   [junit4] # Problematic frame:
   [junit4] # J  org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(I)Lorg/apache/lucene/index/Fields;
   [junit4] #
   [junit4] # Core dump written. Default location: /home/rmuir/workspace/trunk/lucene/build/core/test/J0/core or core.26780
   [junit4] #
   [junit4] # An error report file with more information is saved as:
   [junit4] # /home/rmuir/workspace/trunk/lucene/build/core/test/J0/hs_err_pid26780.log
   [junit4] #
   [junit4] # If you would like to submit a bug report, please visit:
   [junit4] #   http://bugreport.sun.com/bugreport/crash.jsp
   [junit4] #
   [junit4] <<< JVM J0: EOF ----
   [junit4] Execution time total: 28 seconds
            Show
            Robert Muir added a comment - I crashed again but with a core file (set 'ulimit -c unlimited'). zip file with core dump and hs_err is here: http://people.apache.org/~rmuir/crash.zip (its too large for JIRA, sorry) For some more context, it always happens fairly early in the test run: so when it doesnt crash at this exact point, you can ^C and run again until it does. Here was my commands with output below: (i tried to simplify the procedure to make it easy to reproduce, but its not easy, it took me quite a few tries) # note: we are pulling the exact revision that jenkins failed on, because things have changed in lucene codebase over the weekend!!!! svn co -r 1523179 https://svn.apache.org/repos/asf/lucene/dev/trunk # just go to core tests cd trunk/lucene/core # # now the following two commands: just run again and again until it crashes. # rm -rf ../../.caches/ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC" Here was the output: rmuir@beast:~/workspace/trunk/lucene/core$ rm -rf ../../.caches/ rmuir@beast:~/workspace/trunk/lucene/core$ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC" Buildfile: /home/rmuir/workspace/trunk/lucene/core/build.xml -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: ivy-availability-check: ivy-fail: ivy-configure: [ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ :: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: init: compile-core: compile-test-framework: ivy-availability-check: ivy-fail: ivy-configure: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: init: compile-lucene-core: ivy-availability-check: ivy-fail: ivy-configure: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: init: -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: compile-core: compile-codecs: [echo] Building codecs... ivy-availability-check: [echo] Building codecs... ivy-fail: ivy-configure: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: common.init: compile-lucene-core: init: -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: compile-core: -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: common.compile-core: compile-core: compile-test: install-junit4-taskdef: validate: test: [junit4:pickseed] Seed property 'tests.seed' already defined: 43A1116E7F98BED4 [mkdir] Created dir: /home/rmuir/workspace/trunk/.caches/test-stats/core [junit4] <JUnit4> says ciao! Master seed: 43A1116E7F98BED4 [junit4] Executing 367 suites with 1 JVM. [junit4] [junit4] Started J0 PID(26780@beast). [junit4] Suite: org.apache.lucene.store.TestHugeRamFile [junit4] Completed in 1.26s, 1 test [junit4] [junit4] Suite: org.apache.lucene.search.TestTimeLimitingCollector [junit4] Completed in 3.26s, 6 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestLazyProxSkipping [junit4] Completed in 0.68s, 2 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestPostingsOffsets [junit4] Completed in 0.78s, 11 tests [junit4] [junit4] Suite: org.apache.lucene.search.TestRegexpQuery [junit4] Completed in 0.11s, 7 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestTryDelete [junit4] Completed in 0.03s, 3 tests [junit4] [junit4] Suite: org.apache.lucene.util.TestDoubleBarrelLRUCache [junit4] Completed in 1.02s, 2 tests [junit4] [junit4] Suite: org.apache.lucene.analysis.TestGraphTokenizers [junit4] Completed in 3.01s, 21 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging [junit4] Completed in 10.21s, 6 tests [junit4] [junit4] Suite: org.apache.lucene.search.TestSearchAfter [junit4] Completed in 2.05s, 1 test [junit4] [junit4] Suite: org.apache.lucene.index.TestNoMergeScheduler [junit4] Completed in 0.02s, 3 tests [junit4] [junit4] Suite: org.apache.lucene.TestSearchForDuplicates [junit4] Completed in 0.08s, 1 test [junit4] [junit4] Suite: org.apache.lucene.util.TestBytesRef [junit4] Completed in 0.02s, 5 tests [junit4] [junit4] Suite: org.apache.lucene.index.Test4GBStoredFields [junit4] IGNOR/A 0.02s | Test4GBStoredFields.test [junit4] > Assumption #1: 'nightly' test group is disabled (@Nightly) [junit4] Completed in 0.03s, 1 test, 1 skipped [junit4] [junit4] Suite: org.apache.lucene.util.TestAttributeSource [junit4] Completed in 0.02s, 5 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestIndexWriterMergePolicy [junit4] Completed in 2.52s, 7 tests [junit4] [junit4] JVM J0: stdout was not empty, see: /home/rmuir/workspace/trunk/lucene/build/core/test/temp/junit4-J0-20130916_220432_129.sysout [junit4] >>> JVM J0: stdout (verbatim) ---- [junit4] # [junit4] # A fatal error has been detected by the Java Runtime Environment: [junit4] # [junit4] # SIGSEGV (0xb) at pc=0x00007f72952cd74d, pid=26780, tid=140129037821696 [junit4] # [junit4] # JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43) [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 ) [junit4] # Problematic frame: [junit4] # J org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(I)Lorg/apache/lucene/index/Fields; [junit4] # [junit4] # Core dump written. Default location: /home/rmuir/workspace/trunk/lucene/build/core/test/J0/core or core.26780 [junit4] # [junit4] # An error report file with more information is saved as: [junit4] # /home/rmuir/workspace/trunk/lucene/build/core/test/J0/hs_err_pid26780.log [junit4] # [junit4] # If you would like to submit a bug report, please visit: [junit4] # http://bugreport.sun.com/bugreport/crash.jsp [junit4] # [junit4] <<< JVM J0: EOF ---- [junit4] Execution time total: 28 seconds
            Hide
            Permalink
            Robert Muir added a comment -

            I ran again and here is the output where it does not crash, but instead corrupts:

            rmuir@beast:~/workspace/trunk/lucene/core$ rm -rf ../../.caches/
rmuir@beast:~/workspace/trunk/lucene/core$ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC"
Buildfile: /home/rmuir/workspace/trunk/lucene/core/build.xml

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

ivy-availability-check:

ivy-fail:

ivy-configure:
[ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml

resolve:

init:

compile-core:

compile-test-framework:

ivy-availability-check:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml

resolve:

init:

compile-lucene-core:

ivy-availability-check:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml

resolve:

init:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

compile-core:

compile-codecs:
     [echo] Building codecs...

ivy-availability-check:
     [echo] Building codecs...

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml

resolve:

common.init:

compile-lucene-core:

init:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

compile-core:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

common.compile-core:

compile-core:

compile-test:

install-junit4-taskdef:

validate:

test:
[junit4:pickseed] Seed property 'tests.seed' already defined: 43A1116E7F98BED4
    [mkdir] Created dir: /home/rmuir/workspace/trunk/.caches/test-stats/core
   [junit4] <JUnit4> says ciao! Master seed: 43A1116E7F98BED4
   [junit4] Executing 367 suites with 1 JVM.
   [junit4] 
   [junit4] Started J0 PID(27313@beast).
   [junit4] Suite: org.apache.lucene.store.TestHugeRamFile
   [junit4] Completed in 1.55s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTimeLimitingCollector
   [junit4] Completed in 3.23s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestLazyProxSkipping
   [junit4] Completed in 0.57s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPostingsOffsets
   [junit4] Completed in 0.78s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestRegexpQuery
   [junit4] Completed in 0.11s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTryDelete
   [junit4] Completed in 0.04s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestDoubleBarrelLRUCache
   [junit4] Completed in 1.02s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestGraphTokenizers
   [junit4] Completed in 3.05s, 21 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging
   [junit4] Completed in 10.27s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSearchAfter
   [junit4]   1> CheckIndex failed
   [junit4]   1> Segments file=segments_2 numSegments=2 version=5.0 format=
   [junit4]   1>   1 of 2: name=_0 docCount=156
   [junit4]   1>     codec=Lucene45
   [junit4]   1>     compound=false
   [junit4]   1>     numFiles=30
   [junit4]   1>     size (MB)=0.157
   [junit4]   1>     diagnostics = {timestamp=1379384646861, os=Linux, os.version=3.5.0-27-generic, source=flush, lucene.version=5.0-SNAPSHOT, os.arch=amd64, java.version=1.7.0_40, java.vendor=Oracle Corporation}
   [junit4]   1>     no deletions
   [junit4]   1>     test: open reader.........OK
   [junit4]   1>     test: fields..............OK [15 fields]
   [junit4]   1>     test: field norms.........OK [2 fields]
   [junit4]   1>     test: terms, freq, prox...OK [8628 terms; 10659 terms/docs pairs; 423 tokens]
   [junit4]   1>     test: stored fields.......OK [1 total field count; avg 0.006 fields per doc]
   [junit4]   1>     test: term vectors........OK [3 total vector count; avg 1 term/freq vector fields per doc]
   [junit4]   1>     test: docvalues...........OK [5 docvalues fields; 1 BINARY; 2 NUMERIC; 2 SORTED; 0 SORTED_SET]
   [junit4]   1> 
   [junit4]   1>   2 of 2: name=_1 docCount=57
   [junit4]   1>     codec=Lucene45
   [junit4]   1>     compound=false
   [junit4]   1>     numFiles=30
   [junit4]   1>     size (MB)=0.06
   [junit4]   1>     diagnostics = {timestamp=1379384646900, os=Linux, os.version=3.5.0-27-generic, source=flush, lucene.version=5.0-SNAPSHOT, os.arch=amd64, java.version=1.7.0_40, java.vendor=Oracle Corporation}
   [junit4]   1>     no deletions
   [junit4]   1>     test: open reader.........OK
   [junit4]   1>     test: fields..............OK [15 fields]
   [junit4]   1>     test: field norms.........OK [2 fields]
   [junit4]   1>     test: terms, freq, prox...OK [3296 terms; 3969 terms/docs pairs; 227 tokens]
   [junit4]   1>     test: stored fields.......OK [2 total field count; avg 0.035 fields per doc]
   [junit4]   1>     test: term vectors........ERROR [null]
   [junit4]   1> java.lang.NegativeArraySizeException
   [junit4]   1> 	at org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(CompressingTermVectorsReader.java:433)
   [junit4]   1> 	at org.apache.lucene.index.SegmentReader.getTermVectors(SegmentReader.java:169)
   [junit4]   1> 	at org.apache.lucene.index.CheckIndex.testTermVectors(CheckIndex.java:1524)
   [junit4]   1> 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:611)
   [junit4]   1> 	at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:223)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:641)
   [junit4]   1> 	at org.apache.lucene.search.TestSearchAfter.tearDown(TestSearchAfter.java:86)
   [junit4]   1> 	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:795)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   1>     test: docvalues...........OK [5 docvalues fields; 1 BINARY; 2 NUMERIC; 2 SORTED; 0 SORTED_SET]
   [junit4]   1> FAILED
   [junit4]   1>     WARNING: fixIndex() would remove reference to this segment; full exception:
   [junit4]   1> java.lang.RuntimeException: Term Vector test failed
   [junit4]   1> 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:624)
   [junit4]   1> 	at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:223)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:641)
   [junit4]   1> 	at org.apache.lucene.search.TestSearchAfter.tearDown(TestSearchAfter.java:86)
   [junit4]   1> 	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:795)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   1> 
   [junit4]   1> WARNING: 1 broken segments (containing 57 documents) detected
   [junit4]   1> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSearchAfter -Dtests.method=testQueries -Dtests.seed=43A1116E7F98BED4 -Dtests.slow=true -Dtests.locale=th_TH_TH_#u-nu-thai -Dtests.timezone=JST -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   2.09s | TestSearchAfter.testQueries <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: CheckIndex failed
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([43A1116E7F98BED4:1F2FDDB565F10B7A]:0)
   [junit4]    > 	at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:227)
   [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:641)
   [junit4]    > 	at org.apache.lucene.search.TestSearchAfter.tearDown(TestSearchAfter.java:86)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> NOTE: test params are: codec=Lucene45: {short=PostingsFormat(name=FSTPulsing41), oddeven=PostingsFormat(name=Lucene41WithOrds), bytes=PostingsFormat(name=TestBloomFilteredLucene41Postings), byte=PostingsFormat(name=FSTPulsing41), int=PostingsFormat(name=TestBloomFilteredLucene41Postings), bytesval=PostingsFormat(name=FSTPulsing41), long=PostingsFormat(name=FSTPulsing41), double=PostingsFormat(name=Lucene41WithOrds), float=PostingsFormat(name=FSTPulsing41), english=PostingsFormat(name=Lucene41VarGapFixedInterval)}, docValues:{straightbytesdocvalues=DocValuesFormat(name=SimpleText), intdocvalues=DocValuesFormat(name=Lucene45), floatdocvalues=DocValuesFormat(name=Disk), sortedbytesdocvalues=DocValuesFormat(name=Disk), sortedbytesdocvaluesval=DocValuesFormat(name=SimpleText)}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {oddeven=DFR GBZ(0.3), english=DFR I(F)BZ(0.3)}, locale=th_TH_TH_#u-nu-thai, timezone=JST
   [junit4]   2> NOTE: Linux 3.5.0-27-generic amd64/Oracle Corporation 1.7.0_40 (64-bit)/cpus=8,threads=1,free=167266488,total=400556032
   [junit4]   2> NOTE: All tests run in this JVM: [TestHugeRamFile, TestTimeLimitingCollector, TestLazyProxSkipping, TestPostingsOffsets, TestRegexpQuery, TestTryDelete, TestDoubleBarrelLRUCache, TestGraphTokenizers, TestIndexWriterMerging, TestSearchAfter]
   [junit4] Completed in 2.11s, 1 test, 1 error <<< FAILURES!
            Show
            Robert Muir added a comment - I ran again and here is the output where it does not crash, but instead corrupts: rmuir@beast:~/workspace/trunk/lucene/core$ rm -rf ../../.caches/ rmuir@beast:~/workspace/trunk/lucene/core$ ant test -Dtests.seed=43A1116E7F98BED4 -Dtests.jvms=1 -Dargs="-XX:-UseCompressedOops -XX:+UseParallelGC" Buildfile: /home/rmuir/workspace/trunk/lucene/core/build.xml -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: ivy-availability-check: ivy-fail: ivy-configure: [ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ :: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: init: compile-core: compile-test-framework: ivy-availability-check: ivy-fail: ivy-configure: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: init: compile-lucene-core: ivy-availability-check: ivy-fail: ivy-configure: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: init: -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: compile-core: compile-codecs: [echo] Building codecs... ivy-availability-check: [echo] Building codecs... ivy-fail: ivy-configure: [ivy:configure] :: loading settings :: file = /home/rmuir/workspace/trunk/lucene/ivy-settings.xml resolve: common.init: compile-lucene-core: init: -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: compile-core: -clover.disable: -clover.load: -clover.classpath: -clover.setup: clover: common.compile-core: compile-core: compile-test: install-junit4-taskdef: validate: test: [junit4:pickseed] Seed property 'tests.seed' already defined: 43A1116E7F98BED4 [mkdir] Created dir: /home/rmuir/workspace/trunk/.caches/test-stats/core [junit4] <JUnit4> says ciao! Master seed: 43A1116E7F98BED4 [junit4] Executing 367 suites with 1 JVM. [junit4] [junit4] Started J0 PID(27313@beast). [junit4] Suite: org.apache.lucene.store.TestHugeRamFile [junit4] Completed in 1.55s, 1 test [junit4] [junit4] Suite: org.apache.lucene.search.TestTimeLimitingCollector [junit4] Completed in 3.23s, 6 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestLazyProxSkipping [junit4] Completed in 0.57s, 2 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestPostingsOffsets [junit4] Completed in 0.78s, 11 tests [junit4] [junit4] Suite: org.apache.lucene.search.TestRegexpQuery [junit4] Completed in 0.11s, 7 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestTryDelete [junit4] Completed in 0.04s, 3 tests [junit4] [junit4] Suite: org.apache.lucene.util.TestDoubleBarrelLRUCache [junit4] Completed in 1.02s, 2 tests [junit4] [junit4] Suite: org.apache.lucene.analysis.TestGraphTokenizers [junit4] Completed in 3.05s, 21 tests [junit4] [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging [junit4] Completed in 10.27s, 6 tests [junit4] [junit4] Suite: org.apache.lucene.search.TestSearchAfter [junit4] 1> CheckIndex failed [junit4] 1> Segments file=segments_2 numSegments=2 version=5.0 format= [junit4] 1> 1 of 2: name=_0 docCount=156 [junit4] 1> codec=Lucene45 [junit4] 1> compound=false [junit4] 1> numFiles=30 [junit4] 1> size (MB)=0.157 [junit4] 1> diagnostics = {timestamp=1379384646861, os=Linux, os.version=3.5.0-27-generic, source=flush, lucene.version=5.0-SNAPSHOT, os.arch=amd64, java.version=1.7.0_40, java.vendor=Oracle Corporation} [junit4] 1> no deletions [junit4] 1> test: open reader.........OK [junit4] 1> test: fields..............OK [15 fields] [junit4] 1> test: field norms.........OK [2 fields] [junit4] 1> test: terms, freq, prox...OK [8628 terms; 10659 terms/docs pairs; 423 tokens] [junit4] 1> test: stored fields.......OK [1 total field count; avg 0.006 fields per doc] [junit4] 1> test: term vectors........OK [3 total vector count; avg 1 term/freq vector fields per doc] [junit4] 1> test: docvalues...........OK [5 docvalues fields; 1 BINARY; 2 NUMERIC; 2 SORTED; 0 SORTED_SET] [junit4] 1> [junit4] 1> 2 of 2: name=_1 docCount=57 [junit4] 1> codec=Lucene45 [junit4] 1> compound=false [junit4] 1> numFiles=30 [junit4] 1> size (MB)=0.06 [junit4] 1> diagnostics = {timestamp=1379384646900, os=Linux, os.version=3.5.0-27-generic, source=flush, lucene.version=5.0-SNAPSHOT, os.arch=amd64, java.version=1.7.0_40, java.vendor=Oracle Corporation} [junit4] 1> no deletions [junit4] 1> test: open reader.........OK [junit4] 1> test: fields..............OK [15 fields] [junit4] 1> test: field norms.........OK [2 fields] [junit4] 1> test: terms, freq, prox...OK [3296 terms; 3969 terms/docs pairs; 227 tokens] [junit4] 1> test: stored fields.......OK [2 total field count; avg 0.035 fields per doc] [junit4] 1> test: term vectors........ERROR [null] [junit4] 1> java.lang.NegativeArraySizeException [junit4] 1> at org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(CompressingTermVectorsReader.java:433) [junit4] 1> at org.apache.lucene.index.SegmentReader.getTermVectors(SegmentReader.java:169) [junit4] 1> at org.apache.lucene.index.CheckIndex.testTermVectors(CheckIndex.java:1524) [junit4] 1> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:611) [junit4] 1> at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:223) [junit4] 1> at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:641) [junit4] 1> at org.apache.lucene.search.TestSearchAfter.tearDown(TestSearchAfter.java:86) [junit4] 1> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) [junit4] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit4] 1> at java.lang.reflect.Method.invoke(Method.java:606) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:795) [junit4] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) [junit4] 1> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) [junit4] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) [junit4] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) [junit4] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) [junit4] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) [junit4] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4] 1> at java.lang.Thread.run(Thread.java:724) [junit4] 1> test: docvalues...........OK [5 docvalues fields; 1 BINARY; 2 NUMERIC; 2 SORTED; 0 SORTED_SET] [junit4] 1> FAILED [junit4] 1> WARNING: fixIndex() would remove reference to this segment; full exception: [junit4] 1> java.lang.RuntimeException: Term Vector test failed [junit4] 1> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:624) [junit4] 1> at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:223) [junit4] 1> at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:641) [junit4] 1> at org.apache.lucene.search.TestSearchAfter.tearDown(TestSearchAfter.java:86) [junit4] 1> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) [junit4] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit4] 1> at java.lang.reflect.Method.invoke(Method.java:606) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:795) [junit4] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) [junit4] 1> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) [junit4] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) [junit4] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) [junit4] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) [junit4] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) [junit4] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) [junit4] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4] 1> at java.lang.Thread.run(Thread.java:724) [junit4] 1> [junit4] 1> WARNING: 1 broken segments (containing 57 documents) detected [junit4] 1> [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSearchAfter -Dtests.method=testQueries -Dtests.seed=43A1116E7F98BED4 -Dtests.slow=true -Dtests.locale=th_TH_TH_#u-nu-thai -Dtests.timezone=JST -Dtests.file.encoding=ISO-8859-1 [junit4] ERROR 2.09s | TestSearchAfter.testQueries <<< [junit4] > Throwable #1: java.lang.RuntimeException: CheckIndex failed [junit4] > at __randomizedtesting.SeedInfo.seed([43A1116E7F98BED4:1F2FDDB565F10B7A]:0) [junit4] > at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:227) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:641) [junit4] > at org.apache.lucene.search.TestSearchAfter.tearDown(TestSearchAfter.java:86) [junit4] > at java.lang.Thread.run(Thread.java:724) [junit4] 2> NOTE: test params are: codec=Lucene45: {short=PostingsFormat(name=FSTPulsing41), oddeven=PostingsFormat(name=Lucene41WithOrds), bytes=PostingsFormat(name=TestBloomFilteredLucene41Postings), byte=PostingsFormat(name=FSTPulsing41), int=PostingsFormat(name=TestBloomFilteredLucene41Postings), bytesval=PostingsFormat(name=FSTPulsing41), long=PostingsFormat(name=FSTPulsing41), double=PostingsFormat(name=Lucene41WithOrds), float=PostingsFormat(name=FSTPulsing41), english=PostingsFormat(name=Lucene41VarGapFixedInterval)}, docValues:{straightbytesdocvalues=DocValuesFormat(name=SimpleText), intdocvalues=DocValuesFormat(name=Lucene45), floatdocvalues=DocValuesFormat(name=Disk), sortedbytesdocvalues=DocValuesFormat(name=Disk), sortedbytesdocvaluesval=DocValuesFormat(name=SimpleText)}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {oddeven=DFR GBZ(0.3), english=DFR I(F)BZ(0.3)}, locale=th_TH_TH_#u-nu-thai, timezone=JST [junit4] 2> NOTE: Linux 3.5.0-27-generic amd64/Oracle Corporation 1.7.0_40 (64-bit)/cpus=8,threads=1,free=167266488,total=400556032 [junit4] 2> NOTE: All tests run in this JVM: [TestHugeRamFile, TestTimeLimitingCollector, TestLazyProxSkipping, TestPostingsOffsets, TestRegexpQuery, TestTryDelete, TestDoubleBarrelLRUCache, TestGraphTokenizers, TestIndexWriterMerging, TestSearchAfter] [junit4] Completed in 2.11s, 1 test, 1 error <<< FAILURES!
            Hide
            Permalink
            Robert Muir added a comment -

            FYI: I ran this procedure about 10 times with the suggested workaround from https://bugs.openjdk.java.net/browse/JDK-8024830 and tests always pass:

            -XX:-UseLoopPredicate

            Show
            Robert Muir added a comment - FYI: I ran this procedure about 10 times with the suggested workaround from https://bugs.openjdk.java.net/browse/JDK-8024830 and tests always pass: -XX:-UseLoopPredicate
            Hide
            Permalink
            Dawid Weiss added a comment -

            Is there any way to subscribe to that openjdk JIRA to watch issues? I don't see any way to do it

            Show
            Dawid Weiss added a comment - Is there any way to subscribe to that openjdk JIRA to watch issues? I don't see any way to do it
            Hide
            Permalink
            Robert Muir added a comment -

            I cant figure out how to comment or anything. I wanted to just include this information to try to help the openjdk developers reproduce the bug and so on.

            But for now I know I can do it here...

            Show
            Robert Muir added a comment - I cant figure out how to comment or anything. I wanted to just include this information to try to help the openjdk developers reproduce the bug and so on. But for now I know I can do it here...
            Hide
            Permalink
            Dawid Weiss added a comment -

            I don't think you can comment or do anything until you're logged in. And you can't log in because it seems like a registration-closed system...

            Show
            Dawid Weiss added a comment - I don't think you can comment or do anything until you're logged in. And you can't log in because it seems like a registration-closed system...
            Hide
            Permalink
            Dawid Weiss added a comment -

            For the record, this is a different bug than LUCENE-5168 (or a different manifestation of the same underlying bug?) because -XX:-UseLoopPredicate doesn't affect LUCENE-5168 (it still miscompiles).

            Show
            Dawid Weiss added a comment - For the record, this is a different bug than LUCENE-5168 (or a different manifestation of the same underlying bug?) because -XX:-UseLoopPredicate doesn't affect LUCENE-5168 (it still miscompiles).
            Hide
            Permalink
            Robert Muir added a comment -

            Yes thats a separate, older bug with the only workaround of -XX:-DoEscapeAnalysis correct?

            If somehow these two bugs decide to drink beers together in the same jenkins run it will fry my brain.

            Show
            Robert Muir added a comment - Yes thats a separate, older bug with the only workaround of -XX:-DoEscapeAnalysis correct? If somehow these two bugs decide to drink beers together in the same jenkins run it will fry my brain.
            Hide
            Permalink
            Dawid Weiss added a comment -

            -XX:-DoEscapeAnalysis

            Yes, the bug is somewhere in the escape analysis/ ideal graph optimization chain so disabling it does the job. The first JVM revision that had this bug is jdk7u04, versions before that one are not affected.

            The good news is that this problem seems to affect only 32-bit JVMs, I couldn't reproduce it on a 64-bit JVM. It may be due to a larger number of CPU registers (so spill stack variables are not needed and thus there are no missed updates...). If this is the case then the problem may still occur somewhere else, depending on the method/ inlines/ switches/, you name it...

            The current status of LUCENE-5168 is that one of hotspot core developers (Vladimir Kozlov) has collected the information needed for debugging, we know after which commit to hotpot the problem starts appearing and we know it's still present both in jdk7 and in jdk8. I'd have to devote myself full time to getting up to speed with internal hotspot code to figure out anything more – even the logs get really hairy (and huge) at the detail level we're looking at the problem. Hopefully Vladimir will be able to look into it at some point.

            Show
            Dawid Weiss added a comment - -XX:-DoEscapeAnalysis Yes, the bug is somewhere in the escape analysis/ ideal graph optimization chain so disabling it does the job. The first JVM revision that had this bug is jdk7u04, versions before that one are not affected. The good news is that this problem seems to affect only 32-bit JVMs, I couldn't reproduce it on a 64-bit JVM. It may be due to a larger number of CPU registers (so spill stack variables are not needed and thus there are no missed updates...). If this is the case then the problem may still occur somewhere else, depending on the method/ inlines/ switches/, you name it... The current status of LUCENE-5168 is that one of hotspot core developers (Vladimir Kozlov) has collected the information needed for debugging, we know after which commit to hotpot the problem starts appearing and we know it's still present both in jdk7 and in jdk8. I'd have to devote myself full time to getting up to speed with internal hotspot code to figure out anything more – even the logs get really hairy (and huge) at the detail level we're looking at the problem. Hopefully Vladimir will be able to look into it at some point.
            Hide
            Permalink
            Robert Muir added a comment -

            Attached 'crashFaster.patch', which you can apply to the checkout.

            It disables any unnecessary tests to reproduce the bug and speed up bisection.

            Show
            Robert Muir added a comment - Attached 'crashFaster.patch', which you can apply to the checkout. It disables any unnecessary tests to reproduce the bug and speed up bisection.
            Hide
            Permalink
            Robert Muir added a comment -

            I noticed on my system: if the index goes corrupt it tends to happen in TestSearchAfter, but when it SIGSEGVs sometimes that happens in TestLucene45DocValuesFormat (sometimes it corrupts here, too)

            So I updated the patch to include that test too, to try to make it easier to reproduce... (it still takes several tries for me)

            Show
            Robert Muir added a comment - I noticed on my system: if the index goes corrupt it tends to happen in TestSearchAfter, but when it SIGSEGVs sometimes that happens in TestLucene45DocValuesFormat (sometimes it corrupts here, too) So I updated the patch to include that test too, to try to make it easier to reproduce... (it still takes several tries for me)
            Hide
            Permalink
            Dawid Weiss added a comment -

            Don't know for sure if they're related or not but linking nonetheless.

            Show
            Dawid Weiss added a comment - Don't know for sure if they're related or not but linking nonetheless.
            Hide
            Permalink
            Littlestar added a comment - - edited

            java 7u40? mybe also corrupt docvalue.
            ================
            LUCENE-5218 background merge hit exception && Caused by: java.lang.ArrayIndexOutOfBoundsException

            Show
            Littlestar added a comment - - edited java 7u40? mybe also corrupt docvalue. ================ LUCENE-5218 background merge hit exception && Caused by: java.lang.ArrayIndexOutOfBoundsException
            Hide
            Permalink
            Bill Bell added a comment -

            It appears this happens on 7u40 64-bit too. See https://bugs.openjdk.java.net/browse/JDK-8024830

            Am I reading this wrong?

            Start failing around hs24-b21:

            [junit4] # SIGSEGV (0xb) at pc=0xfffffd7ff91d9f7d, pid=23810, tid=343
            [junit4] #
            [junit4] # JRE version: Java(TM) SE Runtime Environment (8.0-b54)
            [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b21 mixed mode solaris-amd64 )
            [junit4] # Problematic frame:
            [junit4] # J org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(I)Lorg/apache/lucene/index/Fields;
            [junit4] #

            Note, first 7u40 build b01 has hs24-b24.

            Next, I will try to find changeset.

            Show
            Bill Bell added a comment - It appears this happens on 7u40 64-bit too. See https://bugs.openjdk.java.net/browse/JDK-8024830 Am I reading this wrong? Start failing around hs24-b21: [junit4] # SIGSEGV (0xb) at pc=0xfffffd7ff91d9f7d, pid=23810, tid=343 [junit4] # [junit4] # JRE version: Java(TM) SE Runtime Environment (8.0-b54) [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b21 mixed mode solaris-amd64 ) [junit4] # Problematic frame: [junit4] # J org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(I)Lorg/apache/lucene/index/Fields; [junit4] # Note, first 7u40 build b01 has hs24-b24. Next, I will try to find changeset.
            Hide
            Permalink
            Littlestar added a comment -

            Is there a small corrupt index can reproduce the problem by CheckIndex, Thanks.

            Show
            Littlestar added a comment - Is there a small corrupt index can reproduce the problem by CheckIndex, Thanks.
            Hide
            Permalink
            mjuarez added a comment -

            It seems like Oracle might punt on this until jdk8 comes around. Which would preclude any safe use of Lucene for jdk7u40 and higher.

            From the last comment on the JDK issue:

            https://bugs.openjdk.java.net/browse/JDK-8024830

            Azeem Jiva added a comment - 2013-10-17 10:06 - edited

            8-compiler-defer: It requires a very specific set of circumstances to have this occur. Request to defer and fix when the issue is better understood. Fix for JDK8 update.

            Show
            mjuarez added a comment - It seems like Oracle might punt on this until jdk8 comes around. Which would preclude any safe use of Lucene for jdk7u40 and higher. From the last comment on the JDK issue: https://bugs.openjdk.java.net/browse/JDK-8024830 Azeem Jiva added a comment - 2013-10-17 10:06 - edited 8-compiler-defer: It requires a very specific set of circumstances to have this occur. Request to defer and fix when the issue is better understood. Fix for JDK8 update.
            Hide
            Permalink
            Dawid Weiss added a comment -

            This is really disappointing. It's like saying we'll fix your car but for now, in very specific circumstances, it can blow right into your face...

            Show
            Dawid Weiss added a comment - This is really disappointing. It's like saying we'll fix your car but for now, in very specific circumstances, it can blow right into your face...
            Hide
            Permalink
            Uwe Schindler added a comment -

            Hi,

            the bug did not appear anymore recently on Jenkins. Michael McCandless refactored the horrible big method around CompressingTermVectorsReader, so the bug seems to be fixed.

            In general we should not create such big methods Eclipse "Extract method" is wonderful!

            Show
            Uwe Schindler added a comment - Hi, the bug did not appear anymore recently on Jenkins. Michael McCandless refactored the horrible big method around CompressingTermVectorsReader, so the bug seems to be fixed. In general we should not create such big methods Eclipse "Extract method" is wonderful!
            Hide
            Permalink
            Dawid Weiss added a comment -

            I'm talking about the JVM bug, not the particular trigger that caused it in Lucene code. We can dodge those errors but sooner or later somebody will trip over them anyway.

            Show
            Dawid Weiss added a comment - I'm talking about the JVM bug, not the particular trigger that caused it in Lucene code. We can dodge those errors but sooner or later somebody will trip over them anyway.
            Hide
            Permalink
            Uwe Schindler added a comment -

            Of course

            Show
            Uwe Schindler added a comment - Of course
            Hide
            Permalink
            Dawid Weiss added a comment -

            Vladimir just pinged me that this issue is (hopefully) fixed, the webrev is here:
            http://cr.openjdk.java.net/~kvn/8024830/webrev/

            I'll test this patch (with jdk7u40) and let you know if I can (or can no longer) reproduce the issue.

            JDK bug ref: https://bugs.openjdk.java.net/browse/JDK-8024830

            Show
            Dawid Weiss added a comment - Vladimir just pinged me that this issue is (hopefully) fixed, the webrev is here: http://cr.openjdk.java.net/~kvn/8024830/webrev/ I'll test this patch (with jdk7u40) and let you know if I can (or can no longer) reproduce the issue. JDK bug ref: https://bugs.openjdk.java.net/browse/JDK-8024830
            Hide
            Permalink
            Dawid Weiss added a comment -

            I confirm that this patch fixes the problem. I've tested svn rev. 1523179 (trunk) against jdk8-b114 with and without Vladimir's patch. Without the patch the test sequence ends about 50% of the time in a sigsegv. With the patch all executions ended without any errors.

            Note that the problem only affects CPUs with the AVX extension. A workaround for affected VMs is to disable vectorization with -XX:-UseSuperWord.

            Show
            Dawid Weiss added a comment - I confirm that this patch fixes the problem. I've tested svn rev. 1523179 (trunk) against jdk8-b114 with and without Vladimir's patch. Without the patch the test sequence ends about 50% of the time in a sigsegv. With the patch all executions ended without any errors. Note that the problem only affects CPUs with the AVX extension. A workaround for affected VMs is to disable vectorization with -XX:-UseSuperWord.
            Hide
            Permalink
            Robert Muir added a comment -

            The fix is committed to openjdk trunk.

            Show
            Robert Muir added a comment - The fix is committed to openjdk trunk.
            Hide
            Permalink
            Uwe Schindler added a comment -

            Hi,
            I figured out: Policeman Jenkins has no vector extensions! So the bug cannot affect Policeman Jenkins (has Lynfield CPU, no Sanybridge). So the bug causing the sigsegv here is not the same bug as the one we hit on Jenkins (that corrumpted term vectors).
            Of course it could be that the vector extensions bug also affected other CPUs, but I have no idea of that code.

            Show
            Uwe Schindler added a comment - Hi, I figured out: Policeman Jenkins has no vector extensions! So the bug cannot affect Policeman Jenkins (has Lynfield CPU, no Sanybridge). So the bug causing the sigsegv here is not the same bug as the one we hit on Jenkins (that corrumpted term vectors). Of course it could be that the vector extensions bug also affected other CPUs, but I have no idea of that code.
            Hide
            Permalink
            Robert Muir added a comment -

            Uwe, i just read the issue slightly, but maybe it would be good to ask Vladimir if the bug really only impacts AVX instructions.

            See the patch, perhaps it impacts 128bit SSE as well and is just easier to trip with the AVX ones since they have 256bit extensions?

            Show
            Robert Muir added a comment - Uwe, i just read the issue slightly, but maybe it would be good to ask Vladimir if the bug really only impacts AVX instructions. See the patch, perhaps it impacts 128bit SSE as well and is just easier to trip with the AVX ones since they have 256bit extensions?
            Hide
            Permalink
            Dawid Weiss added a comment -

            Vladimir said to me I needed an AVX CPU to test the patch. Odd. I may take a look at the openjdk code later on to see when this optimization is enabled.

            Show
            Dawid Weiss added a comment - Vladimir said to me I needed an AVX CPU to test the patch. Odd. I may take a look at the openjdk code later on to see when this optimization is enabled.
            Hide
            Permalink
            Dawid Weiss added a comment -

            We inspected the code with Robert Muir and came to the conclusion that this bug may also affect SSE machines. Since hotspot code is very complex I also contacted Vladimir and he kindly replied that yes, this is the case.

            And you are correct the problem also affects SSE for which we can generate 16 bytes vectors which are larger then 8 bytes stack frame alignment we have. Actually in the bug there were only 16 bytes vectors max.
            I think it may be related to register masks we use on AVX machine, they are larger to be able map 32 bytes vectors. But again with SSE and 16 bytes vectors you can hit the same 8024830 problem but may be much less frequent.

            Show
            Dawid Weiss added a comment - We inspected the code with Robert Muir and came to the conclusion that this bug may also affect SSE machines. Since hotspot code is very complex I also contacted Vladimir and he kindly replied that yes, this is the case. And you are correct the problem also affects SSE for which we can generate 16 bytes vectors which are larger then 8 bytes stack frame alignment we have. Actually in the bug there were only 16 bytes vectors max. I think it may be related to register masks we use on AVX machine, they are larger to be able map 32 bytes vectors. But again with SSE and 16 bytes vectors you can hit the same 8024830 problem but may be much less frequent.

              People

              • Assignee:
                Unassigned
                Reporter:
                Robert Muir
              • Votes:
                2 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development