[CASSANDRA-10233] IndexOutOfBoundsException in HintedHandOffManager

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 2.1.12, 2.2.4

Environment

Cassandra 2.2.0

• Severity: Low
• Assignee: J.P. Eiti Kimura
• Reporter: Omri Iluz

Authors

J.P. Eiti Kimura

Reviewers

Paulo Motta

• Created: 31/Aug/15 18:44
• Updated: 16/Apr/19 09:30
• Resolved: 14/Oct/15 15:59

Description

After upgrading our cluster to 2.2.0, the following error started showing exectly every 10 minutes on every server in the cluster:

INFO  [CompactionExecutor:1381] 2015-08-31 18:31:55,506 CompactionTask.java:142 - Compacting (8e7e1520-500e-11e5-b1e3-e95897ba4d20) [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-540-big-Data.db:level=0, ]
INFO  [CompactionExecutor:1381] 2015-08-31 18:31:55,599 CompactionTask.java:224 - Compacted (8e7e1520-500e-11e5-b1e3-e95897ba4d20) 1 sstables to [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-541-big,] to level=0.  1,544,495 bytes to 1,544,495 (~100% of original) in 93ms = 15.838121MB/s.  0 total partitions merged to 4.  Partition merge counts were {1:4, }
ERROR [HintedHandoff:1] 2015-08-31 18:31:55,600 CassandraDaemon.java:182 - Exception in thread Thread[HintedHandoff:1,1,main]
java.lang.IndexOutOfBoundsException: null
	at java.nio.Buffer.checkIndex(Buffer.java:538) ~[na:1.7.0_79]
	at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:410) ~[na:1.7.0_79]
	at org.apache.cassandra.utils.UUIDGen.getUUID(UUIDGen.java:106) ~[apache-cassandra-2.2.0.jar:2.2.0]
	at org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:515) ~[apache-cassandra-2.2.0.jar:2.2.0]
	at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:88) ~[apache-cassandra-2.2.0.jar:2.2.0]
	at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:168) ~[apache-cassandra-2.2.0.jar:2.2.0]
	at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-2.2.0.jar:2.2.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_79]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_79]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_79]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]

Issue Links

• relates to: CASSANDRA-10485 Missing host ID on hinted handoff write