[CASSANDRA-7892] Hashmap IllegalStateException in anticompaction

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1.0, 2.1.1

Environment

Cassandra 2.1 rc 5

• Severity: Normal
• Assignee: Mikhail Stepura Assign to me
• Reporter: Johan Bjork

Authors

Mikhail Stepura

Reviewers

Yuki Morishita

• Created: 05/Sep/14 20:10
• Updated: 16/Apr/19 09:31
• Resolved: 07/Sep/14 00:22

Description

ERROR [AntiEntropyStage:4] 2014-09-05 19:55:06,439 CassandraDaemon.java:166 - Exception in thread Thread[AntiEntropyStage:4,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
        at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:117) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_60]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
        at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
        at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:113) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
        ... 4 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_60]
        at java.util.concurrent.FutureTask.get(FutureTask.java:188) ~[na:1.7.0_60]
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
        ... 6 common frames omitted
Caused by: java.lang.IllegalStateException: null
        at java.util.HashMap$HashIterator.remove(HashMap.java:938) ~[na:1.7.0_60]
        at org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:425) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
        at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:381) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
        ... 3 common frames omitted

And on a different host:

ERROR [CompactionExecutor:43] 2014-09-05 19:55:06,376 CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:4
3,1,main]
java.lang.IllegalStateException: null
        at java.util.HashMap$HashIterator.remove(HashMap.java:938) ~[na:1.7.0_60]
        at org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:425) ~[apache-cassa
ndra-2.1.0-rc5.jar:2.1.0-rc5]
        at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:381) ~[apache-cassandra-2.1
.0-rc5.jar:2.1.0-rc5]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]

The exceptions happened in our test environment after I added a bunch of new-nodes at the same time that was in a different (previously non-existing) datacenter. The datacenter is determined using GossipingPropertyFileSnitch and started repairs on a bunch of nodes at the same time using

nodetool -h <node> repair -inc -par

The nodes in the old datacenter had previously been repaired using the -inc option.

Issue Links

• is related to: CASSANDRA-5351 Avoid repairing already-repaired data by default