[CASSANDRA-885] OOM on Commit log Replay

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 0.6
• Severity: Normal
• Assignee: Jonathan Ellis
• Reporter: Paul Querna

Authors

Jonathan Ellis

• Created: 12/Mar/10 20:50
• Updated: 16/Apr/19 09:33
• Resolved: 19/Mar/10 21:32

Description

Running 0.5.

We had a node reboot, and when it came back up, it was unable to replay the commit logs, it would OOM every time.

We upped the max heap to 6 gigs, but it didn't help.

I have a heap dump and have it opened in Eclipse MAT.

Anything specific I should pull out?

Class Name | Shallow Heap | Retained Heap | Percentage
----------------------------------------------------------------------------------------------------------------------------------------

org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor @ 0x7fad06454f48 | 112 | 2,604,583,312 | 84.35%

java.util.concurrent.LinkedBlockingQueue @ 0x7fad06405b78	80	2,604,579,864	84.35%
java.util.HashSet @ 0x7fad071b2110	24	2,952	0.00%
java.lang.String @ 0x7fad071b3588 org.apache.cassandra.concurrent:type=ROW-MUTATION-STAGE	40	176	0.00%
org.apache.cassandra.concurrent.NamedThreadFactory @ 0x7fad0714e998	32	56	0.00%
java.util.concurrent.locks.ReentrantLock$NonfairSync @ 0x7fad0722b570	48	48	0.00%
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject @ 0x7fad071b3560	40	40	0.00%
java.util.concurrent.atomic.AtomicInteger @ 0x7fad071b20e0	24	24	0.00%
java.util.concurrent.locks.ReentrantLock @ 0x7fad071b20f8	24	24	0.00%
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy @ 0x7fad071b2128	16	16	0.00% '- Total: 9 entries

org.apache.cassandra.db.ColumnFamilyStore @ 0x7fad063b3390 | 104 | 95,871,520 | 3.10%
org.apache.cassandra.db.ColumnFamilyStore @ 0x7fad063b3460 | 104 | 82,056,536 | 2.66%
org.apache.cassandra.db.ColumnFamilyStore @ 0x7fad063b3188 | 104 | 72,894,176 | 2.36%
org.apache.cassandra.db.ColumnFamilyStore @ 0x7fad06331798 | 104 | 45,394,360 | 1.47%
org.apache.cassandra.db.ColumnFamilyStore @ 0x7fad06331660 | 104 | 33,895,560 | 1.10%
java.lang.Thread @ 0x7fad060e2320 main Native Stack, Thread | 168 | 33,573,328 | 1.09%
org.apache.cassandra.db.ColumnFamilyStore @ 0x7fad06287e50 | 104 | 33,528,680 | 1.09%
org.apache.cassandra.db.ColumnFamilyStore @ 0x7fad063b3530 | 104 | 33,423,720 | 1.08%
org.apache.cassandra.db.ColumnFamilyStore @ 0x7fad063b32c0 | 104 | 20,221,624 | 0.65%
org.apache.cassandra.db.Memtable @ 0x7fad06410120 | 96 | 9,383,880 | 0.30%
org.apache.cassandra.db.ColumnFamily @ 0x7fad33df3690 | 80 | 1,190,016 | 0.04%
org.apache.cassandra.io.SSTableReader @ 0x7fad0653a4a0 | 72 | 587,008 | 0.02%
org.apache.cassandra.io.SSTableReader @ 0x7fad06c41ef0 | 72 | 543,552 | 0.02%
org.apache.cassandra.io.SSTableReader @ 0x7fad06456450 | 72 | 541,320 | 0.02%
Total: 15 of 49,239 entries | | |
----------------------------------------------------------------------------------------------------------------------------------------