Details
-
Bug
-
Status: Open
-
Normal
-
Resolution: Unresolved
-
Cassandra 2.1.13, Ubuntu 14.04.5 LTS, Docker version 1.9.1, run as a container, 4 core server with 16GB memory.
-
Normal
Description
During a nodetool repair -par on one of our keyspaces, Cassandra crashed due to what seems like memory exhaustion within the JVM. The machine itself had plenty of available memory at the time and did not appear to be under any significant load.
In the system log, before the crash, the following was logged:
... INFO [AntiEntropySessions:55] 2017-05-10 18:18:20,627 RepairJob.java:163 - [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] requesting merkle trees for stats_day_aggregates (to [/54.162.66.114, /54.236.226.76, /52.221.228.170, /54.154.35.144, /54.154.96.213, /52.221.217.27]) INFO [ValidationExecutor:54] 2017-05-10 18:18:20,628 ColumnFamilyStore.java:905 - Enqueuing flush of stats_day_aggregates: 7018 (0%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:13608] 2017-05-10 18:18:20,628 Memtable.java:347 - Writing Memtable-stats_day_aggregates@3469792(2.734KiB serialized bytes, 14 ops, 0%/0% of on/off-heap limit) INFO [MemtableFlushWriter:13608] 2017-05-10 18:18:20,629 Memtable.java:382 - Completed flushing /var/lib/cassandra/data/ably_production_0_stats/stats_day_aggregates-b6e29201e3d111e5bbf3091830ac5256/ably_production_0_stats-stats_day_aggregates-tmp-ka-43026-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1491420635101, position=24224955) INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,220 StreamResultFuture.java:180 - [Stream #0008cac1-35ad-11e7-b7e4-091830ac5256] Session with /52.203.21.193 is complete INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,220 StreamResultFuture.java:212 - [Stream #0008cac1-35ad-11e7-b7e4-091830ac5256] All sessions completed INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,221 StreamingRepairTask.java:96 - [repair #fe7e3320-35ac-11e7-b7e4-091830ac5256] streaming task succeed, returning response to /52.221.217.27 INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,230 CqlSlowLogWriter.java:151 - Recording statements with duration of 4844 in slow log INFO [Service Thread] 2017-05-10 18:18:26,233 GCInspector.java:258 - G1 Old Generation GC in 4781ms. G1 Eden Space: 1310720000 -> 0; G1 Old Gen: 2774539816 -> 1830851216; G1 Survivor Space: 37748736 -> 0; INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,237 RepairSession.java:171 - [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for stats_day_aggregates from /54.162.66.114 INFO [StreamConnectionEstablisher:1] 2017-05-10 18:18:26,239 StreamCoordinator.java:209 - [Stream #0293bb60-35ad-11e7-b7e4-091830ac5256, ID#0] Beginning stream session with /54.154.226.20 INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,294 RepairSession.java:171 - [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for stats_day_aggregates from /54.236.226.76 INFO [Service Thread] 2017-05-10 18:18:26,298 StatusLogger.java:51 - Pool Name Active Pending Completed Blocked All Time Blocked INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 - [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for stats_day_aggregates from /54.154.35.144 INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,344 CqlSlowLogWriter.java:151 - Recording statements with duration of 5035 in slow log WARN [GossipTasks:1] 2017-05-10 18:18:26,344 FailureDetector.java:258 - Not marking nodes down due to local pause of 5109502584 > 5000000000 INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 - [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for stats_day_aggregates from /54.154.96.213 INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 - [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for stats_day_aggregates from /52.221.228.170 INFO [Service Thread] 2017-05-10 18:18:26,345 StatusLogger.java:66 - MutationStage 0 0 1199223432 0 0 INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,345 CqlSlowLogWriter.java:151 - Recording statements with duration of 5095 in slow log ERROR [StreamReceiveTask:640] 2017-05-10 18:18:26,346 StreamReceiveTask.java:183 - Error applying streamed data: org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:399) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:365) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:174) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:463) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:447) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:442) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:141) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.8.0_101] at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:391) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] ... 11 common frames omitted Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_101] at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_101] ... 12 common frames omitted ERROR [StreamReceiveTask:640] 2017-05-10 18:18:26,349 JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting forcefully due to: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_101] at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_101] at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:391) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:365) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:174) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:463) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:447) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:442) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:141) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
The JVM then exited completely and logged the following:
# # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # In 32 bit mode, the process size limit was hit # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Use 64 bit Java on a 64 bit OS # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (os_linux.cpp:2627), pid=79, tid=0x00007f06b6e56700 # # JRE version: Java(TM) SE Runtime Environment (8.0_101-b13) (build 1.8.0_101-b13) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.101-b13 mixed mode linux-amd64 compressed oops) # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # --------------- T H R E A D --------------- Current thread (0x00007f03d5054000): JavaThread "Thread-1" [_thread_new, id=23656, stack(0x00007f06b6e16000,0x00007f06b6e57000)] Stack: [0x00007f06b6e16000,0x00007f06b6e57000], sp=0x00007f06b6e55720, free space=253k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0xac3f0a] VMError::report_and_die()+0x2ba V [libjvm.so+0x4fbf9b] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b V [libjvm.so+0x9222d3] os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x103 V [libjvm.so+0x92239c] os::pd_commit_memory(char*, unsigned long, bool)+0xc V [libjvm.so+0x91be4a] os::commit_memory(char*, unsigned long, bool)+0x2a V [libjvm.so+0x92056f] os::pd_create_stack_guard_pages(char*, unsigned long)+0x7f V [libjvm.so+0xa69ace] JavaThread::create_stack_guard_pages()+0x5e V [libjvm.so+0xa73484] JavaThread::run()+0x34 V [libjvm.so+0x924698] java_start(Thread*)+0x108 C [libpthread.so.0+0x8184] start_thread+0xc4 --------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0x00007f03d578c000 JavaThread "Daemon shutdown" [_thread_new, id=23658, stack(0x00007f074a62a000,0x00007f074a66b000)] 0x00007f03d5278000 JavaThread "Thread-0" [_thread_new, id=23657, stack(0x00007f074422f000,0x00007f0744270000)] =>0x00007f03d5054000 JavaThread "Thread-1" [_thread_new, id=23656, stack(0x00007f06b6e16000,0x00007f06b6e57000)] 0x00007f03d5187800 JavaThread "STREAM-OUT-/54.154.226.20" daemon [_thread_blocked, id=23644, stack(0x00007f0749704000,0x00007f0749745000)] 0x00007f03d465d800 JavaThread "STREAM-OUT-/54.169.103.14" daemon [_thread_blocked, id=23627, stack(0x00007f07493f7000,0x00007f0749438000)] 0x00007f05ac99d000 JavaThread "STREAM-OUT-/54.154.226.20" daemon [_thread_blocked, id=23625, stack(0x00007f074ab13000,0x00007f074ab54000)] 0x00007f03d4620800 JavaThread "STREAM-OUT-/54.154.226.20" daemon [_thread_blocked, id=23591, stack(0x00007f07253cb000,0x00007f072540c000)] 0x00007f03d5086000 JavaThread "STREAM-IN-/54.154.226.20" daemon [_thread_in_native, id=23511, stack(0x00007f074986b000,0x00007f07498ac000)] 0x00007f0449a77000 JavaThread "STREAM-OUT-/54.154.226.20" daemon [_thread_blocked, id=23508, stack(0x00007f0747758000,0x00007f0747799000)] 0x00007f03d5185000 JavaThread "STREAM-IN-/54.154.226.20" daemon [_thread_in_native, id=23488, stack(0x00007f07491bc000,0x00007f07491fd000)] 0x00007f03d5104000 JavaThread "STREAM-IN-/54.154.226.20" daemon [_thread_in_native, id=23473, stack(0x00007f0743e40000,0x00007f0743e81000)] 0x00007f0448b64000 JavaThread "STREAM-IN-/54.154.226.20" daemon [_thread_in_native, id=23393, stack(0x00007f074917b000,0x00007f07491bc000)] 0x00007f03d4f7d800 JavaThread "STREAM-OUT-/52.203.21.193" daemon [_thread_blocked, id=23369, stack(0x00007f0749966000,0x00007f07499a7000)] 0x00007f03d4fed800 JavaThread "STREAM-OUT-/54.154.226.20" daemon [_thread_blocked, id=23367, stack(0x00007f074416c000,0x00007f07441ad000)] 0x00007f03d5302000 JavaThread "STREAM-OUT-/54.154.226.20" daemon [_thread_blocked, id=23303, stack(0x00007f0722044000,0x00007f0722085000)] 0x00007f044a921000 JavaThread "RepairJobTask:15" daemon [_thread_blocked, id=23131, stack(0x00007f074ae05000,0x00007f074ae46000)] 0x00007f044b9ae000 JavaThread "RepairJobTask:14" daemon [_thread_blocked, id=23121, stack(0x00007f074982a000,0x00007f074986b000)] 0x00007f044a920000 JavaThread "RepairJobTask:13" daemon [_thread_blocked, id=23116, stack(0x00007f074366e000,0x00007f07436af000)] 0x00007f03d45eb000 JavaThread "RepairJobTask:12" daemon [_thread_blocked, id=23115, stack(0x00007f0717d50000,0x00007f0717d91000)] 0x00007f03d45ea800 JavaThread "RepairJobTask:11" daemon [_thread_blocked, id=23103, stack(0x00007f0743f64000,0x00007f0743fa5000)] 0x00007f03d4fce000 JavaThread "RepairJobTask:10" daemon [_thread_blocked, id=23102, stack(0x00007f074ae6f000,0x00007f074aeb0000)] 0x00007f03d4e78800 JavaThread "RepairJobTask:9" daemon [_thread_blocked, id=23101, stack(0x00007f0720fa2000,0x00007f0720fe3000)] 0x00007f03d50bc000 JavaThread "RepairJobTask:8" daemon [_thread_blocked, id=23100, stack(0x00007f07210a6000,0x00007f07210e7000)] 0x00007f03d506f000 JavaThread "RepairJobTask:7" daemon [_thread_blocked, id=23099, stack(0x00007f074ad2d000,0x00007f074ad6e000)] 0x00007f03d5146000 JavaThread "RepairJobTask:6" daemon [_thread_blocked, id=23098, stack(0x00007f0747c0b000,0x00007f0747c4c000)] 0x00007f03d4f6d800 JavaThread "RepairJobTask:5" daemon [_thread_blocked, id=23097, stack(0x00007f0750043000,0x00007f0750084000)] 0x00007f03d5079800 JavaThread "RepairJobTask:4" daemon [_thread_blocked, id=23096, stack(0x00007f0742dde000,0x00007f0742e1f000)] 0x00007f03d4f94000 JavaThread "RepairJobTask:3" daemon [_thread_blocked, id=23095, stack(0x00007f074a5c4000,0x00007f074a605000)] 0x00007f03d52a6000 JavaThread "RepairJobTask:2" daemon [_thread_blocked, id=23094, stack(0x00007f072122c000,0x00007f072126d000)] 0x00007f03d52ee800 JavaThread "RepairJobTask:1" daemon [_thread_blocked, id=23093, stack(0x00007f07436f0000,0x00007f0743731000)] 0x00007f057f042000 JavaThread "RepairJobTask:15" daemon [_thread_blocked, id=23078, stack(0x00007f074a799000,0x00007f074a7da000)] 0x00007f03d57a4800 JavaThread "RepairJobTask:14" daemon [_thread_blocked, id=23077, stack(0x00007f0683bd4000,0x00007f0683c15000)] 0x00007f044ba0f800 JavaThread "RepairJobTask:13" daemon [_thread_blocked, id=23035, stack(0x00007f074aa41000,0x00007f074aa82000)] 0x00007f03d50b9000 JavaThread "RepairJobTask:12" daemon [_thread_blocked, id=23010, stack(0x00007f074384e000,0x00007f074388f000)] 0x00007f03d50b7800 JavaThread "RepairJobTask:11" daemon [_thread_blocked, id=23009, stack(0x00007f0749380000,0x00007f07493c1000)] 0x00007f03d50b6000 JavaThread "RepairJobTask:10" daemon [_thread_blocked, id=23008, stack(0x00007f07438d0000,0x00007f0743911000)] 0x00007f03d50b5800 JavaThread "RepairJobTask:9" daemon [_thread_blocked, id=23007, stack(0x00007f0744027000,0x00007f0744068000)] 0x00007f03d50af000 JavaThread "RepairJobTask:8" daemon [_thread_blocked, id=23006, stack(0x00007f074388f000,0x00007f07438d0000)] 0x00007f03d5097800 JavaThread "RepairJobTask:7" daemon [_thread_blocked, id=22989, stack(0x00007f074ba63000,0x00007f074baa4000)] 0x00007f03d5096800 JavaThread "RepairJobTask:6" daemon [_thread_blocked, id=22988, stack(0x00007f0742b98000,0x00007f0742bd9000)] 0x00007f03d5095000 JavaThread "RepairJobTask:5" daemon [_thread_blocked, id=22987, stack(0x00007f0721fc2000,0x00007f0722003000)] 0x00007f03d5093800 JavaThread "RepairJobTask:4" daemon [_thread_blocked, id=22986, stack(0x00007f0743f23000,0x00007f0743f64000)] 0x00007f03d5092800 JavaThread "RepairJobTask:3" daemon [_thread_blocked, id=22985, stack(0x00007f074a99e000,0x00007f074a9df000)] 0x00007f03d5091000 JavaThread "RepairJobTask:2" daemon [_thread_blocked, id=22984, stack(0x00007f074a6ac000,0x00007f074a6ed000)] 0x00007f03d5090800 JavaThread "RepairJobTask:1" daemon [_thread_blocked, id=22983, stack(0x00007f0749745000,0x00007f0749786000)] 0x00007f03d4f7a000 JavaThread "ReadRepairStage:73775" daemon [_thread_blocked, id=22227, stack(0x00007f074aabc000,0x00007f074aafd000)] 0x00007f044a94f800 JavaThread "ReadRepairStage:73774" daemon [_thread_blocked, id=22226, stack(0x00007f0744068000,0x00007f07440a9000)] 0x00007f03d437d000 JavaThread "ReadRepairStage:73773" daemon [_thread_blocked, id=21837, stack(0x00007f0743772000,0x00007f07437b3000)] 0x00007f03d4471000 JavaThread "ReadRepairStage:73770" daemon [_thread_blocked, id=18261, stack(0x00007f07491fd000,0x00007f074923e000)] 0x00007f060f438800 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=15398, stack(0x00007f0747011000,0x00007f0747052000)] 0x00007f060fa59000 JavaThread "RMI TCP Connection(5238)-172.17.0.5" daemon [_thread_blocked, id=10756, stack(0x00007f0743fa5000,0x00007f0743fe6000)] 0x00007f044b9aa000 JavaThread "ValidationExecutor:54" daemon [_thread_in_Java, id=9233, stack(0x00007f0748b0a000,0x00007f0748b4b000)] 0x00007f04494ef800 JavaThread "SharedPool-Worker-407" daemon [_thread_blocked, id=3652, stack(0x00007f074907e000,0x00007f07490bf000)] 0x00007f057e393000 JavaThread "SharedPool-Worker-406" daemon [_thread_blocked, id=3651, stack(0x00007f0744270000,0x00007f07442b1000)] 0x00007f044a847000 JavaThread "ValidationExecutor:51" daemon [_thread_blocked, id=442, stack(0x00007f07498ed000,0x00007f074992e000)] 0x00007f04b147b000 JavaThread "AntiEntropySessions:56" daemon [_thread_blocked, id=1481, stack(0x00007f0742b16000,0x00007f0742b57000)] 0x00007f0693cd4800 JavaThread "StreamReceiveTask:640" daemon [_thread_blocked, id=1467, stack(0x00007f074acec000,0x00007f074ad2d000)] 0x00007f05d9c8d800 JavaThread "StreamReceiveTask:639" daemon [_thread_blocked, id=1458, stack(0x00007f0749635000,0x00007f0749676000)] 0x00007f04b3e37000 JavaThread "StreamReceiveTask:638" daemon [_thread_blocked, id=1434, stack(0x00007f074a583000,0x00007f074a5c4000)] 0x00007f06924f7800 JavaThread "AntiEntropySessions:55" daemon [_thread_blocked, id=1412, stack(0x00007f074781b000,0x00007f074785c000)] 0x00007f074ceaf000 JavaThread "StreamReceiveTask:637" daemon [_thread_blocked, id=1401, stack(0x00007f0722085000,0x00007f07220c6000)] 0x00007f069398e800 JavaThread "MemtableFlushWriter:13609" daemon [_thread_blocked, id=1380, stack(0x00007f0748ac9000,0x00007f0748b0a000)] 0x00007f069374e800 JavaThread "MemtableReclaimMemory:9889" daemon [_thread_blocked, id=1379, stack(0x00007f07440ea000,0x00007f074412b000)] 0x00007f0787c63800 JavaThread "MemtableFlushWriter:13608" daemon [_thread_blocked, id=1378, stack(0x00007f074412b000,0x00007f074416c000)] 0x00007f0693cc6000 JavaThread "AntiEntropySessions:54" daemon [_thread_blocked, id=1376, stack(0x00007f07436af000,0x00007f07436f0000)] 0x00007f06936ff000 JavaThread "MemtablePostFlush:36174" daemon [_thread_blocked, id=1374, stack(0x00007f07442b1000,0x00007f07442f2000)] 0x00007f0787c19000 JavaThread "Thread-280182" daemon [_thread_blocked, id=1371, stack(0x00007f07498ac000,0x00007f07498ed000)] 0x00007f074f9da800 JavaThread "JMX server connection timeout 768070" daemon [_thread_blocked, id=1369, stack(0x00007f0743ee2000,0x00007f0743f23000)] 0x0000000001fed800 JavaThread "CompactionExecutor:16697" daemon [_thread_in_Java, id=1365, stack(0x00007f074b8ec000,0x00007f074b92d000)] 0x0000000001367800 JavaThread "CompactionExecutor:16696" daemon [_thread_in_Java, id=1364, stack(0x00007f0747695000,0x00007f07476d6000)] 0x00007f0690975800 JavaThread "MessagingService-Incoming-/54.174.252.207" [_thread_in_native, id=30389, stack(0x00007f07499e8000,0x00007f0749a29000)] 0x00007f074cd88800 JavaThread "MessagingService-Incoming-/54.174.252.207" [_thread_in_native, id=30381, stack(0x00007f0743911000,0x00007f0743952000)] 0x00007f0690a6a000 JavaThread "AntiEntropySessions:50" daemon [_thread_blocked, id=24703, stack(0x00007f07441ee000,0x00007f074422f000)] 0x00007f06924e9000 JavaThread "Thread-169959" daemon [_thread_blocked, id=24700, stack(0x00007f07476d6000,0x00007f0747717000)] 0x00007f074e24a000 JavaThread "JMX server connection timeout 174925" daemon [_thread_blocked, id=15333, stack(0x00007f04b808f000,0x00007f04b80d0000)] 0x00007f060cc21000 JavaThread "RMI TCP Connection(364)-172.17.0.4" daemon [_thread_blocked, id=15332, stack(0x00007f0683d57000,0x00007f0683d98000)] 0x00007f074f3e5800 JavaThread "StreamConnectionEstablisher:4" daemon [_thread_in_native, id=32671, stack(0x00007f074a6fb000,0x00007f074a73c000)] 0x00007f069044a800 JavaThread "StreamingTransferTaskTimeouts:1" daemon [_thread_blocked, id=32667, stack(0x00007f0747799000,0x00007f07477da000)] 0x00007f074d19d800 JavaThread "StreamConnectionEstablisher:3" daemon [_thread_in_native, id=32663, stack(0x00007f0747bca000,0x00007f0747c0b000)] 0x00007f078770d800 JavaThread "StreamConnectionEstablisher:2" daemon [_thread_blocked, id=32660, stack(0x00007f074a53e000,0x00007f074a57f000)] 0x00007f04b070d000 JavaThread "StreamConnectionEstablisher:1" daemon [_thread_blocked, id=32655, stack(0x00007f074ac70000,0x00007f074acb1000)] 0x00007f0693fd0800 JavaThread "MessagingService-Incoming-/172.17.0.2" [_thread_in_native, id=32641, stack(0x00007f0747c4c000,0x00007f0747c8d000)] 0x00007f06931c5800 JavaThread "MessagingService-Outgoing-9b57701d1728/172.17.0.2" daemon [_thread_blocked, id=32639, stack(0x00007f07495ef000,0x00007f0749630000)] 0x00007f04b028d000 JavaThread "MessagingService-Outgoing-9b57701d1728/172.17.0.2" daemon [_thread_blocked, id=32638, stack(0x00007f07497cd000,0x00007f074980e000)] 0x00007f07876a6800 JavaThread "MessagingService-Incoming-/54.162.66.114" [_thread_in_native, id=13194, stack(0x00007f074a66b000,0x00007f074a6ac000)] 0x00007f0693644800 JavaThread "MessagingService-Incoming-/54.236.226.76" [_thread_in_native, id=13192, stack(0x00007f0747b1b000,0x00007f0747b5c000)] 0x00007f04b02cf800 JavaThread "MessagingService-Incoming-/54.236.226.76" [_thread_in_native, id=13191, stack(0x00007f074a7da000,0x00007f074a81b000)] 0x00007f0693f5d800 JavaThread "MessagingService-Incoming-/54.154.35.144" [_thread_in_native, id=13189, stack(0x00007f074a9fd000,0x00007f074aa3e000)] 0x00007f0690909800 JavaThread "MessagingService-Incoming-/54.171.109.9" [_thread_in_native, id=13184, stack(0x00007f07499a7000,0x00007f07499e8000)] 0x00007f04b0728000 JavaThread "MessagingService-Incoming-/54.162.66.114" [_thread_in_native, id=13182, stack(0x00007f0683c15000,0x00007f0683c56000)] 0x00007f0690e07000 JavaThread "MessagingService-Incoming-/54.154.96.213" [_thread_in_native, id=13180, stack(0x00007f0707ec2000,0x00007f0707f03000)] 0x00007f05aca67800 JavaThread "MessagingService-Incoming-/54.154.226.20" [_thread_blocked, id=13177, stack(0x00007f0747052000,0x00007f0747093000)] 0x00007f04b10c6000 JavaThread "MessagingService-Incoming-/54.154.35.144" [_thread_in_native, id=13174, stack(0x00007f0747b5c000,0x00007f0747b9d000)] 0x00007f04b0e58800 JavaThread "MessagingService-Incoming-/54.154.96.213" [_thread_in_native, id=13173, stack(0x00007f0749bf9000,0x00007f0749c3a000)] 0x00007f04b07a1800 JavaThread "MessagingService-Incoming-/54.171.109.9" [_thread_in_native, id=13172, stack(0x00007f0750170000,0x00007f07501b1000)] 0x00007f0690e1a800 JavaThread "MessagingService-Incoming-/54.154.226.20" [_thread_in_native, id=13170, stack(0x00007f0707f85000,0x00007f0707fc6000)] 0x00007f04b02d0800 JavaThread "MessagingService-Incoming-/52.203.21.193" [_thread_in_native, id=13159, stack(0x00007f0717e13000,0x00007f0717e54000)] 0x00007f04b1bd2000 JavaThread "MessagingService-Incoming-/52.203.21.193" [_thread_in_native, id=13155, stack(0x00007f0747188000,0x00007f07471c9000)] 0x00007f04b0090000 JavaThread "SharedPool-Worker-398" daemon [_thread_blocked, id=2502, stack(0x00007f0683d98000,0x00007f0683dd9000)] 0x00007f0786dc5000 JavaThread "SharedPool-Worker-397" daemon [_thread_blocked, id=2501, stack(0x00007f0683dd9000,0x00007f0683e1a000)] 0x00007f06938cc000 JavaThread "SharedPool-Worker-399" daemon [_thread_blocked, id=2500, stack(0x00007f0683e1a000,0x00007f0683e5b000)] 0x00007f06937eb800 JavaThread "SharedPool-Worker-400" daemon [_thread_blocked, id=2499, stack(0x00007f0683e5b000,0x00007f0683e9c000)] 0x00007f0693686800 JavaThread "SharedPool-Worker-401" daemon [_thread_blocked, id=2498, stack(0x00007f0683e9c000,0x00007f0683edd000)] 0x00007f07877f8800 JavaThread "SharedPool-Worker-402" daemon [_thread_blocked, id=2497, stack(0x00007f0683edd000,0x00007f0683f1e000)] 0x00007f074c101800 JavaThread "SharedPool-Worker-403" daemon [_thread_blocked, id=2496, stack(0x00007f0683f1e000,0x00007f0683f5f000)] 0x00007f069397f000 JavaThread "SharedPool-Worker-404" daemon [_thread_blocked, id=2495, stack(0x00007f068566d000,0x00007f06856ae000)] 0x00007f0693572000 JavaThread "SharedPool-Worker-392" daemon [_thread_blocked, id=2494, stack(0x00007f06856ae000,0x00007f06856ef000)] 0x00007f0786f0f000 JavaThread "SharedPool-Worker-405" daemon [_thread_blocked, id=2493, stack(0x00007f0685995000,0x00007f06859d6000)] 0x00007f0786a7b000 JavaThread "SharedPool-Worker-376" daemon [_thread_blocked, id=2492, stack(0x00007f06859d6000,0x00007f0685a17000)] 0x00007f0693ef4800 JavaThread "SharedPool-Worker-379" daemon [_thread_blocked, id=2491, stack(0x00007f0685a17000,0x00007f0685a58000)] 0x00007f04b0281000 JavaThread "SharedPool-Worker-378" daemon [_thread_blocked, id=2490, stack(0x00007f0685a58000,0x00007f0685a99000)] 0x00007f04b0280000 JavaThread "SharedPool-Worker-381" daemon [_thread_blocked, id=2489, stack(0x00007f0685a99000,0x00007f0685ada000)] 0x00007f069046b800 JavaThread "SharedPool-Worker-382" daemon [_thread_blocked, id=2488, stack(0x00007f0685ada000,0x00007f0685b1b000)] 0x00007f069046b000 JavaThread "SharedPool-Worker-383" daemon [_thread_blocked, id=2487, stack(0x00007f0685b1b000,0x00007f0685b5c000)] 0x00007f0693a5f000 JavaThread "SharedPool-Worker-385" daemon [_thread_blocked, id=2486, stack(0x00007f0685b5c000,0x00007f0685b9d000)] 0x00007f0693a5e800 JavaThread "SharedPool-Worker-370" daemon [_thread_blocked, id=2485, stack(0x00007f0685b9d000,0x00007f0685bde000)] 0x00007f04b0649000 JavaThread "SharedPool-Worker-373" daemon [_thread_blocked, id=2484, stack(0x00007f0685bde000,0x00007f0685c1f000)] 0x00007f04b0648800 JavaThread "SharedPool-Worker-386" daemon [_thread_blocked, id=2483, stack(0x00007f0685c1f000,0x00007f0685c60000)] 0x00007f07876a5000 JavaThread "SharedPool-Worker-387" daemon [_thread_blocked, id=2482, stack(0x00007f0685c60000,0x00007f0685ca1000)] 0x00007f07876a4800 JavaThread "SharedPool-Worker-388" daemon [_thread_blocked, id=2481, stack(0x00007f0685ca1000,0x00007f0685ce2000)] 0x00007f04b0c91000 JavaThread "SharedPool-Worker-391" daemon [_thread_blocked, id=2480, stack(0x00007f0685ce2000,0x00007f0685d23000)] 0x00007f04b0c90000 JavaThread "SharedPool-Worker-394" daemon [_thread_blocked, id=2479, stack(0x00007f069404a000,0x00007f069408b000)] 0x00007f04b0752800 JavaThread "SharedPool-Worker-393" daemon [_thread_blocked, id=2478, stack(0x00007f069408b000,0x00007f06940cc000)] 0x00007f04b0751800 JavaThread "SharedPool-Worker-396" daemon [_thread_blocked, id=2477, stack(0x00007f06940cc000,0x00007f069410d000)] 0x00007f074caf4800 JavaThread "SharedPool-Worker-362" daemon [_thread_blocked, id=2476, stack(0x00007f069410d000,0x00007f069414e000)] 0x00007f074caf3800 JavaThread "SharedPool-Worker-395" daemon [_thread_blocked, id=2475, stack(0x00007f069414e000,0x00007f069418f000)] 0x00007f069393a800 JavaThread "SharedPool-Worker-364" daemon [_thread_blocked, id=2474, stack(0x00007f069418f000,0x00007f06941d0000)] 0x00007f069393a000 JavaThread "SharedPool-Worker-365" daemon [_thread_blocked, id=2473, stack(0x00007f06941d0000,0x00007f0694211000)] 0x00007f078765a000 JavaThread "SharedPool-Worker-390" daemon [_thread_blocked, id=2472, stack(0x00007f0694211000,0x00007f0694252000)] 0x00007f0787659800 JavaThread "SharedPool-Worker-389" daemon [_thread_blocked, id=2471, stack(0x00007f0694252000,0x00007f0694293000)] 0x00007f06938b2000 JavaThread "SharedPool-Worker-366" daemon [_thread_blocked, id=2470, stack(0x00007f0694293000,0x00007f06942d4000)] 0x00007f06938b1000 JavaThread "SharedPool-Worker-368" daemon [_thread_blocked, id=2469, stack(0x00007f06942d4000,0x00007f0694315000)] 0x00007f04b090e800 JavaThread "SharedPool-Worker-369" daemon [_thread_blocked, id=2468, stack(0x00007f0694315000,0x00007f0694356000)] 0x00007f04b090e000 JavaThread "SharedPool-Worker-384" daemon [_thread_blocked, id=2467, stack(0x00007f0694356000,0x00007f0694397000)] 0x00007f04b0866000 JavaThread "SharedPool-Worker-374" daemon [_thread_blocked, id=2466, stack(0x00007f06946ce000,0x00007f069470f000)] 0x00007f04b0865000 JavaThread "SharedPool-Worker-380" daemon [_thread_blocked, id=2465, stack(0x00007f069470f000,0x00007f0694750000)] 0x00007f074cde4800 JavaThread "SharedPool-Worker-377" daemon [_thread_blocked, id=2464, stack(0x00007f0694750000,0x00007f0694791000)] 0x00007f074cde3800 JavaThread "SharedPool-Worker-375" daemon [_thread_blocked, id=2463, stack(0x00007f0695812000,0x00007f0695853000)] 0x00007f074ce69000 JavaThread "SharedPool-Worker-359" daemon [_thread_blocked, id=2462, stack(0x00007f06b60da000,0x00007f06b611b000)] 0x00007f074ce68800 JavaThread "SharedPool-Worker-372" daemon [_thread_blocked, id=2461, stack(0x00007f06b611b000,0x00007f06b615c000)] 0x00007f07871f8800 JavaThread "SharedPool-Worker-371" daemon [_thread_blocked, id=2460, stack(0x00007f06b7cc5000,0x00007f06b7d06000)] 0x00007f07871f8000 JavaThread "SharedPool-Worker-361" daemon [_thread_blocked, id=2459, stack(0x00007f06c081e000,0x00007f06c085f000)] 0x00007f04b0c11000 JavaThread "SharedPool-Worker-367" daemon [_thread_blocked, id=2458, stack(0x00007f06c085f000,0x00007f06c08a0000)] 0x00007f04b0c10800 JavaThread "SharedPool-Worker-355" daemon [_thread_blocked, id=2457, stack(0x00007f06c0ae6000,0x00007f06c0b27000)] 0x00007f074cb38000 JavaThread "SharedPool-Worker-363" daemon [_thread_blocked, id=2456, stack(0x00007f06c0b27000,0x00007f06c0b68000)] 0x00007f074cb37800 JavaThread "SharedPool-Worker-353" daemon [_thread_blocked, id=2455, stack(0x00007f06c0b68000,0x00007f06c0ba9000)] 0x00007f0787de6800 JavaThread "SharedPool-Worker-360" daemon [_thread_blocked, id=2454, stack(0x00007f06c0ba9000,0x00007f06c0bea000)] 0x00007f0787de6000 JavaThread "SharedPool-Worker-356" daemon [_thread_blocked, id=2453, stack(0x00007f06c0bea000,0x00007f06c0c2b000)] 0x00007f04b0146000 JavaThread "SharedPool-Worker-358" daemon [_thread_blocked, id=2452, stack(0x00007f06c0c2b000,0x00007f06c0c6c000)] 0x00007f04b0145000 JavaThread "SharedPool-Worker-348" daemon [_thread_blocked, id=2451, stack(0x00007f06d5e0c000,0x00007f06d5e4d000)] 0x00007f0787ea3000 JavaThread "SharedPool-Worker-357" daemon [_thread_blocked, id=2450, stack(0x00007f06d5e4d000,0x00007f06d5e8e000)] 0x00007f0787ea2000 JavaThread "SharedPool-Worker-349" daemon [_thread_blocked, id=2449, stack(0x00007f06d5e8e000,0x00007f06d5ecf000)] 0x00007f04b0f5e000 JavaThread "SharedPool-Worker-354" daemon [_thread_blocked, id=2448, stack(0x00007f06d5ecf000,0x00007f06d5f10000)] 0x00007f04b0f5d000 JavaThread "SharedPool-Worker-350" daemon [_thread_blocked, id=2447, stack(0x00007f06d68fc000,0x00007f06d693d000)] 0x00007f074e06f800 JavaThread "SharedPool-Worker-352" daemon [_thread_blocked, id=2446, stack(0x00007f06d693d000,0x00007f06d697e000)] 0x00007f074e06f000 JavaThread "SharedPool-Worker-346" daemon [_thread_blocked, id=2445, stack(0x00007f06d697e000,0x00007f06d69bf000)] 0x00007f0693f77000 JavaThread "SharedPool-Worker-351" daemon [_thread_blocked, id=2444, stack(0x00007f06d69bf000,0x00007f06d6a00000)] 0x00007f0693f76000 JavaThread "SharedPool-Worker-347" daemon [_thread_blocked, id=2443, stack(0x00007f06d6a00000,0x00007f06d6a41000)] 0x00007f0787e23000 JavaThread "SharedPool-Worker-344" daemon [_thread_blocked, id=2442, stack(0x00007f06d6a41000,0x00007f06d6a82000)] 0x00007f0787e22800 JavaThread "SharedPool-Worker-345" daemon [_thread_blocked, id=2441, stack(0x00007f06d6a82000,0x00007f06d6ac3000)] 0x00007f06909f2000 JavaThread "SharedPool-Worker-343" daemon [_thread_blocked, id=2440, stack(0x00007f06df3f4000,0x00007f06df435000)] 0x00007f04b0ebb800 JavaThread "SharedPool-Worker-342" daemon [_thread_blocked, id=2439, stack(0x00007f0701182000,0x00007f07011c3000)] 0x00007f07877d1800 JavaThread "SharedPool-Worker-341" daemon [_thread_blocked, id=2438, stack(0x00007f07011c3000,0x00007f0701204000)] 0x00007f04b0233800 JavaThread "SharedPool-Worker-340" daemon [_thread_blocked, id=2437, stack(0x00007f0701204000,0x00007f0701245000)] 0x00007f074c302000 JavaThread "SharedPool-Worker-339" daemon [_thread_blocked, id=2436, stack(0x00007f0701245000,0x00007f0701286000)] 0x00007f04b08eb800 JavaThread "SharedPool-Worker-336" daemon [_thread_blocked, id=2435, stack(0x00007f0701286000,0x00007f07012c7000)] 0x00007f074d101000 JavaThread "SharedPool-Worker-337" daemon [_thread_blocked, id=2434, stack(0x00007f07012c7000,0x00007f0701308000)] 0x00007f04b08ed000 JavaThread "SharedPool-Worker-338" daemon [_thread_blocked, id=2433, stack(0x00007f0701308000,0x00007f0701349000)] 0x00007f0787e4e000 JavaThread "SharedPool-Worker-334" daemon [_thread_blocked, id=2432, stack(0x00007f0701349000,0x00007f070138a000)] 0x00007f04b0231800 JavaThread "SharedPool-Worker-324" daemon [_thread_blocked, id=2431, stack(0x00007f070138a000,0x00007f07013cb000)] 0x00007f06935f7800 JavaThread "SharedPool-Worker-321" daemon [_thread_blocked, id=2430, stack(0x00007f07013cb000,0x00007f070140c000)] 0x00007f04b00b0800 JavaThread "SharedPool-Worker-322" daemon [_thread_blocked, id=2429, stack(0x00007f070140c000,0x00007f070144d000)] 0x00007f04b0a37800 JavaThread "SharedPool-Worker-320" daemon [_thread_blocked, id=2428, stack(0x00007f070144d000,0x00007f070148e000)] 0x00007f04b0e91000 JavaThread "SharedPool-Worker-323" daemon [_thread_blocked, id=2427, stack(0x00007f0707a34000,0x00007f0707a75000)] 0x00007f0693474800 JavaThread "SharedPool-Worker-326" daemon [_thread_blocked, id=2426, stack(0x00007f0707d0c000,0x00007f0707d4d000)] 0x00007f04b0a29000 JavaThread "SharedPool-Worker-327" daemon [_thread_blocked, id=2425, stack(0x00007f0707d4d000,0x00007f0707d8e000)] 0x00007f0691394800 JavaThread "SharedPool-Worker-328" daemon [_thread_blocked, id=2424, stack(0x00007f0711b2c000,0x00007f0711b6d000)] 0x00007f0693f9d800 JavaThread "SharedPool-Worker-329" daemon [_thread_blocked, id=2423, stack(0x00007f0711b6d000,0x00007f0711bae000)] 0x00007f06935dd800 JavaThread "SharedPool-Worker-331" daemon [_thread_blocked, id=2422, stack(0x00007f0711bae000,0x00007f0711bef000)] 0x00007f0786de3000 JavaThread "SharedPool-Worker-332" daemon [_thread_blocked, id=2421, stack(0x00007f0711bef000,0x00007f0711c30000)] 0x00007f07877e2800 JavaThread "SharedPool-Worker-313" daemon [_thread_blocked, id=2420, stack(0x00007f0711c30000,0x00007f0711c71000)] 0x00007f04b00c6000 JavaThread "SharedPool-Worker-333" daemon [_thread_blocked, id=2419, stack(0x00007f0711eda000,0x00007f0711f1b000)] 0x00007f04b0928800 JavaThread "SharedPool-Worker-335" daemon [_thread_blocked, id=2418, stack(0x00007f0711f1b000,0x00007f0711f5c000)] 0x00007f0691507800 JavaThread "SharedPool-Worker-312" daemon [_thread_blocked, id=2417, stack(0x00007f0711f5c000,0x00007f0711f9d000)] 0x00007f074da87800 JavaThread "SharedPool-Worker-315" daemon [_thread_blocked, id=2416, stack(0x00007f0711f9d000,0x00007f0711fde000)] 0x00007f0787ea4800 JavaThread "SharedPool-Worker-330" daemon [_thread_blocked, id=2415, stack(0x00007f0711fde000,0x00007f071201f000)] 0x00007f074cb01000 JavaThread "SharedPool-Worker-317" daemon [_thread_blocked, id=2414, stack(0x00007f071201f000,0x00007f0712060000)] 0x00007f04b07ec800 JavaThread "SharedPool-Worker-325" daemon [_thread_blocked, id=2413, stack(0x00007f0712060000,0x00007f07120a1000)] 0x00007f04b035b800 JavaThread "SharedPool-Worker-318" daemon [_thread_blocked, id=2412, stack(0x00007f07120a1000,0x00007f07120e2000)] 0x00007f074d0ab000 JavaThread "SharedPool-Worker-319" daemon [_thread_blocked, id=2411, stack(0x00007f07120e2000,0x00007f0712123000)] 0x00007f0693ba2000 JavaThread "SharedPool-Worker-310" daemon [_thread_blocked, id=2410, stack(0x00007f0712123000,0x00007f0712164000)] 0x00007f074c978000 JavaThread "SharedPool-Worker-311" daemon [_thread_blocked, id=2409, stack(0x00007f0712164000,0x00007f07121a5000)] 0x00007f0691527800 JavaThread "SharedPool-Worker-316" daemon [_thread_blocked, id=2408, stack(0x00007f07121a5000,0x00007f07121e6000)] 0x00007f04b009e000 JavaThread "SharedPool-Worker-314" daemon [_thread_blocked, id=2407, stack(0x00007f07121e6000,0x00007f0712227000)] 0x00007f0787da9800 JavaThread "SharedPool-Worker-308" daemon [_thread_blocked, id=2406, stack(0x00007f0712227000,0x00007f0712268000)] 0x00007f0690943000 JavaThread "SharedPool-Worker-309" daemon [_thread_blocked, id=2405, stack(0x00007f0712268000,0x00007f07122a9000)] 0x00007f074c663800 JavaThread "SharedPool-Worker-307" daemon [_thread_blocked, id=2404, stack(0x00007f07122a9000,0x00007f07122ea000)] 0x00007f0693536000 JavaThread "SharedPool-Worker-306" daemon [_thread_blocked, id=2403, stack(0x00007f07122ea000,0x00007f071232b000)] 0x00007f06935ab800 JavaThread "SharedPool-Worker-305" daemon [_thread_blocked, id=2402, stack(0x00007f071232b000,0x00007f071236c000)] 0x00007f06906c0000 JavaThread "SharedPool-Worker-304" daemon [_thread_blocked, id=2401, stack(0x00007f071236c000,0x00007f07123ad000)] 0x00007f0787053800 JavaThread "SharedPool-Worker-303" daemon [_thread_blocked, id=2400, stack(0x00007f07123ad000,0x00007f07123ee000)] 0x00007f0787064000 JavaThread "SharedPool-Worker-301" daemon [_thread_blocked, id=2399, stack(0x00007f071f396000,0x00007f071f3d7000)] 0x00007f074d6bf800 JavaThread "SharedPool-Worker-302" daemon [_thread_blocked, id=2398, stack(0x00007f071f3d7000,0x00007f071f418000)] 0x00007f0693605800 JavaThread "SharedPool-Worker-300" daemon [_thread_blocked, id=2397, stack(0x00007f07202eb000,0x00007f072032c000)] 0x00007f04b0d1f000 JavaThread "SharedPool-Worker-298" daemon [_thread_blocked, id=2396, stack(0x00007f07208a9000,0x00007f07208ea000)] 0x00007f04b07bf000 JavaThread "SharedPool-Worker-299" daemon [_thread_blocked, id=2395, stack(0x00007f07208ea000,0x00007f072092b000)] 0x00007f0693506800 JavaThread "SharedPool-Worker-297" daemon [_thread_blocked, id=2394, stack(0x00007f0720ab4000,0x00007f0720af5000)] 0x00007f074c167800 JavaThread "SharedPool-Worker-296" daemon [_thread_blocked, id=2393, stack(0x00007f0707e81000,0x00007f0707ec2000)] 0x00007f04b1334800 JavaThread "SharedPool-Worker-295" daemon [_thread_blocked, id=2392, stack(0x00007f074565e000,0x00007f074569f000)] 0x00007f0787c5a000 JavaThread "SharedPool-Worker-294" daemon [_thread_blocked, id=2391, stack(0x00007f072538a000,0x00007f07253cb000)] 0x00007f04b0867000 JavaThread "SharedPool-Worker-292" daemon [_thread_blocked, id=1300, stack(0x00007f072540c000,0x00007f072544d000)] 0x00007f07860c1800 JavaThread "SharedPool-Worker-293" daemon [_thread_blocked, id=1299, stack(0x00007f0708007000,0x00007f0708048000)] 0x00007f0787d19000 JavaThread "SharedPool-Worker-291" daemon [_thread_blocked, id=1298, stack(0x00007f0707f44000,0x00007f0707f85000)] 0x00007f04b0338000 JavaThread "SharedPool-Worker-290" daemon [_thread_blocked, id=1297, stack(0x00007f07210e7000,0x00007f0721128000)] 0x00007f04b0595000 JavaThread "MessagingService-Incoming-/54.179.183.26" [_thread_in_native, id=1127, stack(0x00007f0707f03000,0x00007f0707f44000)] 0x00007f06938af800 JavaThread "MessagingService-Incoming-/54.179.183.26" [_thread_in_native, id=1122, stack(0x00007f0707fc6000,0x00007f0708007000)] 0x00007f06909da000 JavaThread "SharedPool-Worker-286" daemon [_thread_blocked, id=1119, stack(0x00007f0717c4c000,0x00007f0717c8d000)] 0x00007f0693833000 JavaThread "SharedPool-Worker-288" daemon [_thread_blocked, id=1118, stack(0x00007f0717c8d000,0x00007f0717cce000)] 0x00007f04b014a800 JavaThread "SharedPool-Worker-289" daemon [_thread_blocked, id=1117, stack(0x00007f0717cce000,0x00007f0717d0f000)] 0x00007f0787e53000 JavaThread "SharedPool-Worker-284" daemon [_thread_blocked, id=1116, stack(0x00007f0717d0f000,0x00007f0717d50000)] 0x00007f078767f800 JavaThread "SharedPool-Worker-283" daemon [_thread_blocked, id=1115, stack(0x00007f0717e95000,0x00007f0717ed6000)] 0x00007f074c3af800 JavaThread "SharedPool-Worker-285" daemon [_thread_blocked, id=1114, stack(0x00007f0717d91000,0x00007f0717dd2000)] 0x00007f04b020a000 JavaThread "SharedPool-Worker-287" daemon [_thread_blocked, id=1113, stack(0x00007f0717e54000,0x00007f0717e95000)] 0x00007f0693fcf800 JavaThread "SharedPool-Worker-281" daemon [_thread_blocked, id=1112, stack(0x00007f04afa23000,0x00007f04afa64000)] 0x00007f0787210800 JavaThread "SharedPool-Worker-282" daemon [_thread_blocked, id=1111, stack(0x00007f0749d3c000,0x00007f0749d7d000)] 0x00007f04b010a800 JavaThread "SharedPool-Worker-280" daemon [_thread_blocked, id=1110, stack(0x00007f0722003000,0x00007f0722044000)] 0x00007f0693832000 JavaThread "MessagingService-Incoming-/52.221.228.170" [_thread_in_native, id=1101, stack(0x00007f0717dd2000,0x00007f0717e13000)] 0x00007f06934f6000 JavaThread "MessagingService-Incoming-/52.221.228.170" [_thread_in_native, id=1096, stack(0x00007f0717ed6000,0x00007f0717f17000)] 0x00007f04b00d5000 JavaThread "SharedPool-Worker-279" daemon [_thread_blocked, id=1093, stack(0x00007f0717f17000,0x00007f0717f58000)] 0x00007f04b00d4000 JavaThread "SharedPool-Worker-278" daemon [_thread_blocked, id=1092, stack(0x00007f0717f58000,0x00007f0717f99000)] 0x00007f04b01df000 JavaThread "SharedPool-Worker-271" daemon [_thread_blocked, id=1091, stack(0x00007f07205b4000,0x00007f07205f5000)] 0x00007f04b01de000 JavaThread "SharedPool-Worker-274" daemon [_thread_blocked, id=1090, stack(0x00007f07205f5000,0x00007f0720636000)] 0x00007f074c1c7000 JavaThread "SharedPool-Worker-273" daemon [_thread_blocked, id=1089, stack(0x00007f0720636000,0x00007f0720677000)] 0x00007f074c1c6800 JavaThread "SharedPool-Worker-276" daemon [_thread_blocked, id=1088, stack(0x00007f0720677000,0x00007f07206b8000)] 0x00007f04b02a4800 JavaThread "SharedPool-Worker-277" daemon [_thread_blocked, id=1087, stack(0x00007f07206b8000,0x00007f07206f9000)] 0x00007f04b02a4000 JavaThread "SharedPool-Worker-269" daemon [_thread_blocked, id=1086, stack(0x00007f07206f9000,0x00007f072073a000)] 0x00007f074c2ae000 JavaThread "SharedPool-Worker-268" daemon [_thread_blocked, id=1085, stack(0x00007f072073a000,0x00007f072077b000)] 0x00007f074c2ad000 JavaThread "SharedPool-Worker-275" daemon [_thread_blocked, id=1084, stack(0x00007f072077b000,0x00007f07207bc000)] 0x00007f0693ec3000 JavaThread "SharedPool-Worker-272" daemon [_thread_blocked, id=1083, stack(0x00007f07207bc000,0x00007f07207fd000)] 0x00007f0693ec2000 JavaThread "SharedPool-Worker-270" daemon [_thread_blocked, id=1082, stack(0x00007f07207fd000,0x00007f072083e000)] 0x00007f0690983800 JavaThread "SharedPool-Worker-267" daemon [_thread_blocked, id=1081, stack(0x00007f072083e000,0x00007f072087f000)] 0x00007f0690983000 JavaThread "SharedPool-Worker-265" daemon [_thread_blocked, id=1080, stack(0x00007f0720be6000,0x00007f0720c27000)] 0x00007f04b0358000 JavaThread "SharedPool-Worker-266" daemon [_thread_blocked, id=1079, stack(0x00007f0720c27000,0x00007f0720c68000)] 0x00007f04b0357000 JavaThread "SharedPool-Worker-263" daemon [_thread_blocked, id=1078, stack(0x00007f0720c68000,0x00007f0720ca9000)] 0x00007f04b05e6800 JavaThread "SharedPool-Worker-264" daemon [_thread_blocked, id=1077, stack(0x00007f0720ca9000,0x00007f0720cea000)] 0x00007f04b05e5800 JavaThread "SharedPool-Worker-262" daemon [_thread_blocked, id=1076, stack(0x00007f0720cea000,0x00007f0720d2b000)] 0x00007f0786e27800 JavaThread "SharedPool-Worker-261" daemon [_thread_blocked, id=1074, stack(0x00007f0720d2b000,0x00007f0720d6c000)] 0x00007f0786e27000 JavaThread "SharedPool-Worker-258" daemon [_thread_blocked, id=1073, stack(0x00007f0720d6c000,0x00007f0720dad000)] 0x00007f069366c800 JavaThread "SharedPool-Worker-259" daemon [_thread_blocked, id=1072, stack(0x00007f0720dad000,0x00007f0720dee000)] 0x00007f069366c000 JavaThread "SharedPool-Worker-260" daemon [_thread_blocked, id=1071, stack(0x00007f0720dee000,0x00007f0720e2f000)] 0x00007f0693661000 JavaThread "SharedPool-Worker-257" daemon [_thread_blocked, id=1070, stack(0x00007f0720e2f000,0x00007f0720e70000)] 0x00007f0693660000 JavaThread "SharedPool-Worker-256" daemon [_thread_blocked, id=1069, stack(0x00007f0720e70000,0x00007f0720eb1000)] 0x00007f0693603000 JavaThread "SharedPool-Worker-255" daemon [_thread_blocked, id=1068, stack(0x00007f0720f61000,0x00007f0720fa2000)] 0x00007f0693602800 JavaThread "SharedPool-Worker-252" daemon [_thread_blocked, id=1067, stack(0x00007f0720eb1000,0x00007f0720ef2000)] 0x00007f0693fd7000 JavaThread "SharedPool-Worker-251" daemon [_thread_blocked, id=1066, stack(0x00007f0749d7d000,0x00007f0749dbe000)] 0x00007f04b0109800 JavaThread "SharedPool-Worker-254" daemon [_thread_blocked, id=1065, stack(0x00007f04af508000,0x00007f04af549000)] 0x00007f0787c49800 JavaThread "SharedPool-Worker-253" daemon [_thread_blocked, id=1064, stack(0x00007f04b4030000,0x00007f04b4071000)] 0x00007f04b00d9000 JavaThread "SharedPool-Worker-249" daemon [_thread_blocked, id=1063, stack(0x00007f04b772a000,0x00007f04b776b000)] ...
Complete error log is at https://dl.dropboxusercontent.com/u/1575409/Ably/logs/2017-05-10-cassandra-crash/ap-southeast-1/error.log
A screenshot load, memory usage etc. is at https://dl.dropboxusercontent.com/u/1575409/Ably/logs/2017-05-10-cassandra-crash/ap-southeast-1/Voila_Capture%202017-05-11_09-08-53_am.png