[CASSANDRA-6403] Division by zero Exception in HintedHandoff and CompactionExecutor

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.0.4

Environment

Cassandra 2.0.3 RC, Linux (Ubuntu Precise), OpenJDK 7

• Severity: Normal
• Assignee: Unassigned
• Reporter: Gabriel Wicke
• Created: 23/Nov/13 21:53
• Updated: 16/Apr/19 09:31
• Resolved: 13/Mar/14 21:49

Description

In write load testing I'm getting division by zero exceptions after running for a while:

ERROR [HintedHandoff:2] 2013-11-23 20:44:41,411 CassandraDaemon.java (line 187) Exception in thread Thread[HintedHandoff:2,1,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.ArithmeticException: / by zero
at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:464)
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
at org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.util.concurrent.ExecutionException: java.lang.ArithmeticException: / by zero
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:460)
... 6 more

ERROR [CompactionExecutor:8] 2013-11-23 21:34:01,493 CassandraDaemon.java (line 187) Exception in thread Thread[CompactionExecutor:8,1,RMI Runtime]
java.lang.ArithmeticException: / by zero
at org.apache.cassandra.db.compaction.ParallelCompactionIterable.<init>(ParallelCompactionIterable.java:59)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:296)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

The nodes that encounter this error seem to hold onto a lot of memory which is not freed even after the write load is stopped. With the write load continuing they eventually run out of heap. nodetool compact dies with the same exception.