Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-9133

Getting validationexception while repairing nodes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Duplicate
    • Fix Version/s: 2.0.15
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal

      Description

      Cassandra V:2.01.4

      We are running repairs on nodes to run scrub on them later to get around issue (CASSANDRA-9126). While running nodetool repair on nodes, getting the validationexception error. We are seeing bunch of these validation exceptions on lot of nodes. Any help is appreciated?

      java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair #b3ee6050-dd7a-11e4-adac-172f3292c10d on ...., (-1729382256910270568,-1537228672809129408]] Validation failed in /........
      at com.google.common.base.Throwables.propagate(Throwables.java:160)
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.cassandra.exceptions.RepairException: [repair #b3ee6050-dd7a-11e4-adac-172f3292c10d on ...., (-1729382256910270568,-1537228672809129408]] Validation failed in .....
      at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:161)
      at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:228)
      at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
      at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
      

      Also, seeing the following exceptions:

      ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 Validator.java (line 242) Failed creating a merkle tree for [repair #bf8d13c0-dd7a-11e4-80ff-810270bd0450 on ......, (-8262604116349070008,-8070450532247928848]], /.... (see log for details)
      ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 CassandraDaemon.java (line 258) Exception in thread Thread[ValidationExecutor:33,1,main]
      java.lang.AssertionError: row DecoratedKey(-8232281172744762721, 354036383831303736375f3330335f31355f474d4543454f) received out of order wrt DecoratedKey(-8070630157486006754, 3540383834383134305f3336315f31355f474d4543454f)
      at org.apache.cassandra.repair.Validator.add(Validator.java:136)
      at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
      at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
      at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)	

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                yukim Yuki Morishita
                Reporter:
                sgottipati srinivasu gottipati
                Authors:
                Yuki Morishita
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: