[CASSANDRA-3537] ExpiringMap timer is not exception-proof

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 1.1.0

Environment

当前堆大小: 
5,815,955 Kb
堆大小的最大值: 
6,045,696 Kb
分配的内存: 
6,045,696 Kb
暂挂结束操作: 
0 个对象
垃圾收集器: 
Name = 'ParNew', Collections = 3,294, Total time spent = 2 minutes
垃圾收集器: 
Name = 'ConcurrentMarkSweep', Collections = 5,909, Total time spent = 2 hours 17 minutes

操作系统: 
Linux 2.6.32.12-0.7-default
体系结构: 
amd64
处理器的数目: 
16
分配的虚拟内存: 
42,748,416 Kb
物理内存总量: 
24,568,836 Kb
可用物理内存: 
 7,136,380 Kb
交换空间总量: 
 2,104,472 Kb
可用交换空间: 
 1,970,800 Kb

VM 参数: 
-ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G -Xmn2400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dpasswd.properties=/opt/obs/cassandra/conf/passwd.properties -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true
类路径: 
/opt/obs/cassandra/conf:/opt/obs/cassandra/build/classes/main:/opt/obs/cassandra/build/classes/thrift:/opt/obs/cassandra/lib/antlr-3.2.jar:/opt/obs/cassandra/lib/apache-cassandra-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-clientutil-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-thrift-1.0.0.jar:/opt/obs/cassandra/lib/avro-1.4.0-fixes.jar:/opt/obs/cassandra/lib/avro-1.4.0-sources-fixes.jar:/opt/obs/cassandra/lib/cassandra_simple_authentication.jar:/opt/obs/cassandra/lib/commons-cli-1.1.jar:/opt/obs/cassandra/lib/commons-codec-1.2.jar:/opt/obs/cassandra/lib/commons-lang-2.4.jar:/opt/obs/cassandra/lib/compress-lzf-0.8.4.jar:/opt/obs/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/obs/cassandra/lib/guava-r08.jar:/opt/obs/cassandra/lib/high-scale-lib-1.1.2.jar:/opt/obs/cassandra/lib/jackson-core-asl-1.4.0.jar:/opt/obs/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/opt/obs/cassandra/lib/jamm-0.2.5.jar:/opt/obs/cassandra/lib/jline-0.9.94.jar:/opt/obs/cassandra/lib/json-simple-1.1.jar:/opt/obs/cassandra/lib/libthrift-0.6.jar:/opt/obs/cassandra/lib/log4j-1.2.16.jar:/opt/obs/cassandra/lib/servlet-api-2.5-20081211.jar:/opt/obs/cassandra/lib/slf4j-api-1.6.1.jar:/opt/obs/cassandra/lib/slf4j-log4j12-1.6.1.jar:/opt/obs/cassandra/lib/snakeyaml-1.6.jar:/opt/obs/cassandra/lib/snappy-java-1.0.3.jar

• Severity: Low
• Assignee: Jonathan Ellis Assign to me
• Reporter: MaHaiyang

Authors

Jonathan Ellis

Reviewers

Sylvain Lebresne

• Created: 29/Nov/11 07:24
• Updated: 16/Apr/19 09:32
• Resolved: 24/Feb/12 15:26

Description

I have 4 cassandra nodes ,and I put about 30G data to db for every nodes . It's just 4 days before I start the cluster ,but now every 4 nodes have the same problem ,JVM heap is full ,and GC take no effect ,There must be some memory leak . Jmap the memory as follow:

Object Histogram:

num #instances #bytes Class description
--------------------------------------------------------------------------
1: 15793606 758093088 java.nio.HeapByteBuffer
2: 2153811 320138208 java.lang.Object[]
3: 6163192 197222144 org.apache.cassandra.db.Column
4: 2543836 175890256 int[]
5: 2168816 155397192 long[]
6: 2078123 116374888 org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
7: 1847111 73884440 java.math.BigInteger
8: 1234243 59243664 java.util.Hashtable
9: 1770829 58233000 char[]
10: 1770627 56660064 java.lang.String
11: 1665886 39981264 org.apache.cassandra.db.DecoratedKey
12: 692706 38791536 org.cliffc.high_scale_lib.NonBlockingHashMap$CHM
13: 1234274 37172088 java.util.Hashtable$Entry[]
14: 1133541 36273312 java.net.Inet4Address
15: 738528 35449344 org.apache.cassandra.service.ReadCallback
16: 2078118 33249888 org.cliffc.high_scale_lib.Counter
17: 1373886 32973264 org.apache.cassandra.db.ReadResponse
18: 1234023 29616552 org.apache.cassandra.net.Message
19: 1234019 29616456 org.apache.cassandra.net.Header
20: 1846185 29538960 org.apache.cassandra.dht.BigIntegerToken
21: 891378 28524096 org.apache.cassandra.utils.ExpiringMap$CacheableObject
22: 692706 27708240 org.cliffc.high_scale_lib.NonBlockingHashMap
23: 1148252 27558048 java.util.Collections$SynchronizedSet
24: 541977 26014896 org.apache.cassandra.db.SliceFromReadCommand
25: 998001 23952024 java.util.concurrent.ConcurrentSkipListMap$Node
26: 928792 22291008 java.util.ArrayList
27: 692715 22166880 java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
28: 891378 21393072 org.apache.cassandra.net.CallbackInfo
29: 1148247 18371952 java.util.Hashtable$KeySet
30: 731859 17564616 org.apache.cassandra.db.Row
31: 529991 16959712 org.apache.cassandra.db.ArrayBackedSortedColumns
32: 691425 16594200 org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
33: 648580 15565920 org.apache.cassandra.db.filter.QueryPath
34: 648338 15560112 org.apache.cassandra.service.RowDigestResolver
35: 971376 15542016 java.util.concurrent.atomic.AtomicInteger
36: 837418 13398688 org.apache.cassandra.utils.SimpleCondition
37: 535614 12854736 org.apache.cassandra.db.ColumnFamily
38: 725634 11610144 java.util.concurrent.atomic.AtomicReference
39: 195117 9365616 org.apache.cassandra.db.ThreadSafeSortedColumns
40: 281921 9021472 java.util.concurrent.ConcurrentSkipListMap$HeadIndex
41: 277679 8885728 java.util.concurrent.locks.ReentrantLock$NonfairSync
42: 314424 7546176 java.util.concurrent.ConcurrentSkipListMap$Index
43: 275186 6604464 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
44: 270280 6486720 java.util.concurrent.LinkedBlockingQueue$Node
45: 219553 5269272 org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
46: 106436 5108928 java.util.TreeMap
47: 122185 4887400 org.apache.cassandra.db.ExpiringColumn
48: 189968 4559232 org.apache.cassandra.db.SuperColumn
49: 275659 4410544 java.util.concurrent.locks.ReentrantLock
50: 90213 4330224 java.util.concurrent.LinkedBlockingQueue
51: 107026 4281040 java.util.TreeMap$Entry
52: 30501 4222056 * ConstMethodKlass