[CASSANDRA-13757] Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate

Environment

Operation System: Debian Jessie
Java: Oracle JDK 1.8.0_131
Cassandra: 3.5.0

• Severity: Normal
• Assignee: Unassigned
• Reporter: Serhat Rıfat Demircan
• Created: 11/Aug/17 12:40
• Updated: 16/Apr/19 09:30
• Resolved: 19/Nov/18 01:39

Description

We got following error while repair job running on our cluster. One of the nodes stop due to segmantation fault in JVM and repair job fails.

We could not reproduce this problem on our test and staging enviroment (main difference is data size).

#
#  SIGSEGV (0xb) at pc=0x00007fd80a399e70, pid=1305, tid=0x00007fd7ee7c4700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [liblz4-java3580121503903465201.so+0x5e70]  LZ4_decompress_fast+0xd0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x00007fce32dad1b0):  JavaThread "CompactionExecutor:9798" daemon [_thread_in_native, id=16879, stack(0x00007fd7ee784000,0x00007fd7ee7c5000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007fd450c4d000

Registers:
RAX=0x00007fcde6560d32, RBX=0x00007fd450c4cff9, RCX=0x00007fcde6560c7a, RDX=0x00007fcde6560d3e
RSP=0x00007fd7ee7c3160, RBP=0x00007fd450c44ae6, RSI=0x00007fcde6562ff8, RDI=0x00000000000000c2
R8 =0x00007fcde6562ff4, R9 =0x00007fcde6563000, R10=0x0000000000000000, R11=0x0000000000000000
R12=0x000000000000000c, R13=0x00007fd4501cd000, R14=0x00007fcde6562ff7, R15=0x00007fcde6562ffb
RIP=0x00007fd80a399e70, EFLAGS=0x0000000000010283, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fd7ee7c3160)
0x00007fd7ee7c3160:   0000000000000008 00007fd81e21c3d0
0x00007fd7ee7c3170:   0000000000000004 0000000000000001
0x00007fd7ee7c3180:   0000000000000002 0000000000000001
0x00007fd7ee7c3190:   0000000000000004 0000000000000004
0x00007fd7ee7c31a0:   0000000000000004 0000000000000004
0x00007fd7ee7c31b0:   0000000000000000 0000000000000000
0x00007fd7ee7c31c0:   0000000000000000 ffffffffffffffff
0x00007fd7ee7c31d0:   0000000000000000 0000000000000001
0x00007fd7ee7c31e0:   0000000000000002 0000000000000003
0x00007fd7ee7c31f0:   00007fd7ee7c32b8 00007fce32dad3a8
0x00007fd7ee7c3200:   0000000000000000 0000000000000000
0x00007fd7ee7c3210:   00007fd4501cd000 00007fcde6553000
0x00007fd7ee7c3220:   0000000000a77ae6 00007fd80a39659d
0x00007fd7ee7c3230:   0000000000000000 00000000dcb8fc9b
0x00007fd7ee7c3240:   00007fd7ee7c32d0 0000000000000000
0x00007fd7ee7c3250:   00000006e5c7e4d8 00007fd7ee7c32b8
0x00007fd7ee7c3260:   00007fce32dad1b0 00007fd81df2099d
0x00007fd7ee7c3270:   00007fd7ee7c32a8 0000000000000000
0x00007fd7ee7c3280:   0000000000010000 0000000000000000
0x00007fd7ee7c3290:   00000006e5c7e528 00007fd81d74df10
0x00007fd7ee7c32a0:   0000000000000000 00000006e5c7e4d8
0x00007fd7ee7c32b0:   00000006f6c7fbf8 00000006f6e957f0
0x00007fd7ee7c32c0:   00000006e5c7e350 00007fd87fffffff
0x00007fd7ee7c32d0:   00000006e5c7e528 00007fd81fa867e0
0x00007fd7ee7c32e0:   00a77ae200010000 0000000000a77ae2
0x00007fd7ee7c32f0:   00000006e5c7e488 000000000112d5f1
0x00007fd7ee7c3300:   dcb8fc9b000099ce 0001000000a77ae6
0x00007fd7ee7c3310:   00a814b000a814b4 00000006e5c7e4d8
0x00007fd7ee7c3320:   00000006e5c7e4d8 00000006f6a4df38
0x00007fd7ee7c3330:   0000000600010000 000000067fffffff
0x00007fd7ee7c3340:   0089715800002c8a 0006189d87852057
0x00007fd7ee7c3350:   0000000000000000 00000000e5244e71
Instructions: (pc=0x00007fd80a399e70)
0x00007fd80a399e50:   e4 0f 49 83 fc 0f 0f 84 94 00 00 00 4a 8d 14 20
0x00007fd80a399e60:   48 39 f2 0f 87 c0 00 00 00 0f 1f 80 00 00 00 00
0x00007fd80a399e70:   48 8b 0b 48 83 c3 08 48 89 08 48 83 c0 08 48 39
0x00007fd80a399e80:   c2 77 ed 48 29 d0 48 89 d1 48 29 c3 0f b7 03 48

Register to memory mapping:

RAX=0x00007fcde6560d32 is an unknown value
RBX=0x00007fd450c4cff9 is an unknown value
RCX=0x00007fcde6560c7a is an unknown value
RDX=0x00007fcde6560d3e is an unknown value
RSP=0x00007fd7ee7c3160 is pointing into the stack for thread: 0x00007fce32dad1b0
RBP=0x00007fd450c44ae6 is an unknown value
RSI=0x00007fcde6562ff8 is an unknown value
RDI=0x00000000000000c2 is an unknown value
R8 =0x00007fcde6562ff4 is an unknown value
R9 =0x00007fcde6563000 is an unknown value
R10=0x0000000000000000 is an unknown value
R11=0x0000000000000000 is an unknown value
R12=0x000000000000000c is an unknown value
R13=0x00007fd4501cd000 is an unknown value
R14=0x00007fcde6562ff7 is an unknown value
R15=0x00007fcde6562ffb is an unknown value


Stack: [0x00007fd7ee784000,0x00007fd7ee7c5000],  sp=0x00007fd7ee7c3160,  free space=252k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [liblz4-java3580121503903465201.so+0x5e70]  LZ4_decompress_fast+0xd0

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 5084  net.jpountz.lz4.LZ4JNI.LZ4_decompress_fast([BLjava/nio/ByteBuffer;I[BLjava/nio/ByteBuffer;II)I (0 bytes) @ 0x00007fd81df20923 [0x00007fd81df20880+0xa3]
J 20311 C2 org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap()V (354 bytes) @ 0x00007fd81fa867e0 [0x00007fd81fa86160+0x680]
J 10706 C2 org.apache.cassandra.io.util.RandomAccessReader.reBuffer()V (78 bytes) @ 0x00007fd81ea59c58 [0x00007fd81ea59b20+0x138]
J 19996 C2 org.apache.cassandra.io.util.SegmentedFile.createReader(J)Lorg/apache/cassandra/io/util/FileDataInput; (12 bytes) @ 0x00007fd81fa32934 [0x00007fd81fa32540+0x3f4]
J 20645 C2 org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(Lorg/apache/cassandra/io/sstable/format/SSTableReader;Lorg/apache/cassandra/io/util/FileDataInput;Lorg/apache/cassandra/db/DecoratedKey;Lorg/apache/cassandra/db/RowIndexEntry;Lorg/apache/cassandra/db/Slices;Lorg/apache/cassandra/db/filter/ColumnFilter;Z)V (423 bytes) @ 0x00007fd81fbf8cec [0x00007fd81fbf8a20+0x2cc]
J 20644 C2 org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext()Lorg/apache/cassandra/db/rows/Unfiltered; (120 bytes) @ 0x00007fd81e61b5d0 [0x00007fd81e61b140+0x490]
J 9597 C2 org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext()Ljava/lang/Object; (5 bytes) @ 0x00007fd81da8dad4 [0x00007fd81da8daa0+0x34]
J 3432 C2 org.apache.cassandra.utils.AbstractIterator.hasNext()Z (80 bytes) @ 0x00007fd81da28f20 [0x00007fd81da28e80+0xa0]
J 19755 C2 org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext()Ljava/lang/Object; (9 bytes) @ 0x00007fd81f81d00c [0x00007fd81f81cb60+0x4ac]
J 21810 C2 org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext()Lorg/apache/cassandra/db/rows/Unfiltered; (42 bytes) @ 0x00007fd81ef929fc [0x00007fd81ef927a0+0x25c]
J 15842 C2 org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(Ljava/util/List;Lorg/apache/cassandra/metrics/TableMetrics;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; (71 bytes) @ 0x00007fd81ece33b4 [0x00007fd81ece3200+0x1b4]
J 21162 C2 org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; (635 bytes) @ 0x00007fd81fdb2ff0 [0x00007fd81fdb2100+0xef0]
J 27194 C2 org.apache.cassandra.db.Keyspace.indexPartition(Lorg/apache/cassandra/db/DecoratedKey;Lorg/apache/cassandra/db/ColumnFamilyStore;Ljava/util/Set;)V (274 bytes) @ 0x00007fd820c8b044 [0x00007fd820c8aae0+0x564]
J 27422 C2 org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build()V (73 bytes) @ 0x00007fd81f0b5a00 [0x00007fd81f0b5940+0xc0]
J 27619 C1 org.apache.cassandra.db.compaction.CompactionManager$12.run()V (56 bytes) @ 0x00007fd82051313c [0x00007fd820512a00+0x73c]
J 24483 C2 java.util.concurrent.FutureTask.run()V (126 bytes) @ 0x00007fd81d50faf4 [0x00007fd81d50fa20+0xd4]
J 23783 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007fd81fca71a8 [0x00007fd81fca70a0+0x108]
J 21390 C2 java.lang.Thread.run()V (17 bytes) @ 0x00007fd81e6d3cb8 [0x00007fd81e6d3c60+0x58]
v  ~StubRoutines::call_stub

Issue Links

• duplicates: CASSANDRA-14284 Chunk checksum test needs to occur before uncompress to avoid JVM crash
• is duplicated by: CASSANDRA-14284 Chunk checksum test needs to occur before uncompress to avoid JVM crash