[CASSANDRA-13546] Getting error while adding a node in existing cluster

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Cannot Reproduce
• Fix Version/s: 2.1.x
• Component/s: Local/Compaction

Environment

Unix

• Severity: Normal
• Assignee: Unassigned
• Reporter: Sonu Gupta
• Created: 23/May/17 06:56
• Updated: 16/Apr/19 09:30
• Resolved: 13/Feb/18 14:37

Description

Getting error after adding a node in existing cluster, error are coming after 1 hour after started service on seed node and due to this load got increased vastly on seed node not on new node.

Below errors from system.log on seed node:

ERROR [NonPeriodicTasks:1] 2017-05-18 10:03:01,819 CassandraDaemon.java:153 - Exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.AssertionError: null
        at org.apache.cassandra.io.util.Memory.free(Memory.java:300) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:645) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) ~[na:1.7.0_71]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.7.0_71]
        at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
INFO  [CompactionExecutor:162] 2017-05-18 10:03:01,820 ColumnFamilyStore.java:840 - Enqueuing flush of compactions_in_progress: 164 (0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:152] 2017-05-18 10:03:01,821 Memtable.java:325 - Writing Memtable-compactions_in_progress@731506998(0 serialized bytes, 1 ops, 0%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:152] 2017-05-18 10:03:01,825 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-52434-Data.db (42 bytes) for commitlog position ReplayPosition(segmentId=1495106933696, position=9156004)
ERROR [CompactionExecutor:162] 2017-05-18 10:03:01,829 CassandraDaemon.java:153 -
 Exception in thread Thread[CompactionExecutor:162,1,main]
java.lang.AssertionError: null
        at org.apache.cassandra.io.util.Memory.free(Memory.java:300) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:345) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.io.sstable.SSTableRewriter.abort(SSTableRewriter.java:198) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.7.0_71]
        at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
INFO  [main] 2017-05-18 11:06:56,032 CassandraDaemon.java:89 - Hostname: csdb2-dl.intermesh.net
INFO  [main] 2017-05-18 11:06:56,061 YamlConfigurationLoader.java:92 - Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml