[CASSANDRA-9133] Getting validationexception while repairing nodes

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 2.0.15
• Severity: Normal
• Assignee: Yuki Morishita
• Reporter: srinivasu gottipati

Authors

Yuki Morishita

• Created: 08/Apr/15 00:28
• Updated: 16/Apr/19 09:31
• Resolved: 08/Apr/15 02:19

Description

Cassandra V:2.01.4

We are running repairs on nodes to run scrub on them later to get around issue (CASSANDRA-9126). While running nodetool repair on nodes, getting the validationexception error. We are seeing bunch of these validation exceptions on lot of nodes. Any help is appreciated?

java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair #b3ee6050-dd7a-11e4-adac-172f3292c10d on ...., (-1729382256910270568,-1537228672809129408]] Validation failed in /........
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #b3ee6050-dd7a-11e4-adac-172f3292c10d on ...., (-1729382256910270568,-1537228672809129408]] Validation failed in .....
at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:161)
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:228)
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)

Also, seeing the following exceptions:

ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 Validator.java (line 242) Failed creating a merkle tree for [repair #bf8d13c0-dd7a-11e4-80ff-810270bd0450 on ......, (-8262604116349070008,-8070450532247928848]], /.... (see log for details)
ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 CassandraDaemon.java (line 258) Exception in thread Thread[ValidationExecutor:33,1,main]
java.lang.AssertionError: row DecoratedKey(-8232281172744762721, 354036383831303736375f3330335f31355f474d4543454f) received out of order wrt DecoratedKey(-8070630157486006754, 3540383834383134305f3336315f31355f474d4543454f)
at org.apache.cassandra.repair.Validator.add(Validator.java:136)
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)	

Issue Links

• duplicates: CASSANDRA-9126 java.lang.RuntimeException: Last written key DecoratedKey >= current key DecoratedKey