Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-11200

CompactionExecutor thread error brings down JVM in 3.0.3

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Urgent
    • Resolution: Cannot Reproduce
    • Fix Version/s: None
    • Component/s: Local/Compaction
    • Labels:
      None
    • Environment:

      debian jesse latest release, updated Feb. 20th

    • Severity:
      Critical

      Description

      When launching Cassandra 3.0.3, with java version "1.8.0_74", Cassandra writes the following to the debug file before a segmentation fault occurs bringing down the JVM - the problem is repeatable.

      DEBUG [CompactionExecutor:1] 2016-02-20 18:26:16,892 CompactionTask.java:146 - Compacting (56f677c0-d829-11e5-b23a-25dbd4d727f6) [/var/lib/cassandra/data/sensordb/periodicReading/ma-367-big-Data.db:level=0, /var/lib/cassandra/data/sensordb/periodicReading/ma-368-big-Data.db:level=0, /var/lib/cassandra/data/sensordb/periodicReading/ma-371-big-Data.db:level=0, /var/lib/cassandra/data/sensordb/periodicReading/ma-370-big-Data.db:level=0, /var/lib/cassandra/data/sensordb/periodicReading/ma-369-big-Data.db:level=0, ]

      The JVM error that occurs is the following:

      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # SIGBUS (0x7) at pc=0x00007fa8a1052150, pid=12179, tid=140361951868672
      #
      # JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build 1.8.0_74-b02)
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode linux-amd64 compressed oops)
      # Problematic frame:
      # v ~StubRoutines::jbyte_disjoint_arraycopy
      #
      # Core dump written. Default location: /tmp/core or core.12179
      #
      # If you would like to submit a bug report, please visit:
      # http://bugreport.java.com/bugreport/crash.jsp
      #

      --------------- T H R E A D ---------------

      Current thread (0x00007fa89c56ac20): JavaThread "CompactionExecutor:1" daemon [_thread_in_Java, id=12323, stack(0x00007fa89043f000,0x00007fa890480000)]

      siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr: 0x00007fa838988002

      Even if all of the files associated with "ma-[NNN]" are removed, the JVM dies with the same error after the next group of "ma-[NNN]" are eventually written out and compacted.

      Though this may be strictly a JVM problem, I have seen the issue in Oracle JVM 8.0_65 and 8.0_74 and I raise it in case this problem is due to JNI usage of an external compression library or some direct memory usage.

      I have a core dump if that is helpful to anyone.

      Bug CASSANDRA-11201 may also be related although when the exception referenced in the bug occurs, the JVM remains alive.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                blerer Benjamin Lerer
                Reporter:
                longtimer Jason Kania
                Authors:
                Benjamin Lerer
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: