[CASSANDRA-14711] Apache Cassandra 3.2 crashing with exception org.apache.cassandra.db.marshal.TimestampType.compareCustom

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Legacy/Core
• Severity: Normal
• Assignee: Unassigned
• Reporter: Saurabh
• Created: 10/Sep/18 07:40
• Updated: 16/Apr/19 09:29

Description

Hi Team,

I am using Apache Cassandra 3.2 with Java 1.8.0_161-b12..

Issue:

Cassandra is continuously crashing with generating an HEAP dump log. There are no errors reported in system.log OR Debug.log.

Exception in hs_err_PID.log:

1. Problematic frame:
2. J 8283 C2 org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I (6 bytes) @ 0x00002b7d3d417fb4 [0x00002b7d3d417c80+0x334]

Java Threads: ( => current thread )
0x00002b7da57924a0 JavaThread "MemtableReclaimMemory:52" daemon [_thread_blocked, id=117880, stack(0x00002b7d917ff000,0x00002b7d91840000)]
0x00002b7d39f6a9e0 JavaThread "PerDiskMemtableFlushWriter_0:52" daemon [_thread_blocked, id=117879, stack(0x00002b7e4ea94000,0x00002b7e4ead5000)]
0x00002b7d39d0f520 JavaThread "MemtablePostFlush:53" daemon [_thread_blocked, id=117878, stack(0x00002b7e407dd000,0x00002b7e4081e000)]
0x00002b7df31a9150 JavaThread "MemtableFlushWriter:52" daemon [_thread_blocked, id=117877, stack(0x00002b7e406d9000,0x00002b7e4071a000)]
0x00002b7e53e60110 JavaThread "RMI TCP Connection(1795)-127.0.0.1" daemon
:
:
lot of threads in BLOCKED status

Other Threads:
0x00002b7d38de5ea0 VMThread [stack: 0x00002b7d8208d000,0x00002b7d8218d000] [id=32098]
0x00002b7d38fa9de0 WatcherThread [stack: 0x00002b7d88ee9000,0x00002b7d88fe9000] [id=32108]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap:
garbage-first heap total 8388608K, used 6791168K [0x00000003c0000000, 0x00000003c0404000, 0x00000007c0000000)
region size 4096K, 785 young (3215360K), 55 survivors (225280K)
Metaspace used 40915K, capacity 42044K, committed 42368K, reserved 1087488K
class space used 4429K, capacity 4646K, committed 4736K, reserved 1048576K

Heap Regions: (Y=young(eden), SU=young(survivor), HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, TS=gc time stamp, PTAMS=previous top-at-mark-start, NTAMS=next top-at-mark-start)
AC 0 O TS 0 PTAMS 0x00000003c0400000 NTAMS 0x00000003c0400000 space 4096K, 100% used [0x00000003c0000000, 0x00000003c0400000)
AC 0 O TS 0 PTAMS 0x00000003c0800000 NTAMS 0x00000003c0800000 space 4096K, 100% used [0x00000003c0400000, 0x00000003c0800000)
AC 0 O TS 9 PTAMS 0x00000003c0800000 NTAMS 0x00000003c0800000 space 4096K, 100% used [0x00000003c0800000, 0x00000003c0c00000)
AC 0 O TS 11 PTAMS 0x00000003c0c00000 NTAMS 0x00000003c0c00000 space 4096K, 100% used [0x00000003c0c00000, 0x00000003c1000000)
AC 0 O TS 11 PTAMS 0x00000003c1000000 NTAMS 0x00000003c1000000 space 4096K, 100% used [0x00000003c1000000, 0x00000003c1400000)
AC 0 O TS 11 PTAMS 0x00000003c1400000 NTAMS 0x00000003c1400000 space 4096K, 100% used [0x00000003c1400000, 0x00000003c1800000)
:
:
lot of such messages