[CASSANDRA-10519] RepairException: [repair #... on .../..., (...,...]] Validation failed in /w.x.y.z

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Cannot Reproduce

Environment

CentOS 7, JDK 8u60, Cassandra 2.2.2 (upgraded from 2.1.5)

• Severity: Low
• Assignee: Unassigned
• Reporter: Gábor Auth
• Created: 14/Oct/15 06:34
• Updated: 16/Apr/19 09:30
• Resolved: 13/Nov/15 16:55

Description

Sometimes the repair fails:

ERROR [Repair#3:1] 2015-10-14 06:22:56,490 CassandraDaemon.java:185 - Exception in thread Thread[Repair#3:1,5,RMI Runtime]
com.google.common.util.concurrent.UncheckedExecutionException: org.apache.cassandra.exceptions.RepairException: [repair #018adc70-723c-11e5-b0d8-6b2151e4d388 on keyspace/table, (2414492737393085601,27880539413409
54029]] Validation failed in /w.y.x.z
        at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1387) ~[guava-16.0.jar:na]
        at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1373) ~[guava-16.0.jar:na]
        at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:169) ~[apache-cassandra-2.2.2.jar:2.2.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_60]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #018adc70-723c-11e5-b0d8-6b2151e4d388 on keyspace/table, (2414492737393085601,2788053941340954029]] Validation failed in /w.y.x.z
        at org.apache.cassandra.repair.ValidationTask.treeReceived(ValidationTask.java:64) ~[apache-cassandra-2.2.2.jar:2.2.2]
        at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:183) ~[apache-cassandra-2.2.2.jar:2.2.2]
        at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:399) ~[apache-cassandra-2.2.2.jar:2.2.2]
        at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:163) ~[apache-cassandra-2.2.2.jar:2.2.2]
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[apache-cassandra-2.2.2.jar:2.2.2]
        ... 3 common frames omitted

And here is the w.y.x.z side:

ERROR [ValidationExecutor:7] 2015-10-14 06:22:56,487 CompactionManager.java:1053 - Cannot start multiple repair sessions over the same sstables
ERROR [ValidationExecutor:7] 2015-10-14 06:22:56,487 Validator.java:246 - Failed creating a merkle tree for [repair #018adc70-723c-11e5-b0d8-6b2151e4d388 on keyspace/table, (2414492737393085601,2788053941340954029]], /a.b.c.d (see log for details)
ERROR [ValidationExecutor:7] 2015-10-14 06:22:56,488 CassandraDaemon.java:185 - Exception in thread Thread[ValidationExecutor:7,1,main]
java.lang.RuntimeException: Cannot start multiple repair sessions over the same sstables
        at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1054) ~[apache-cassandra-2.2.2.jar:2.2.2]
        at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:86) ~[apache-cassandra-2.2.2.jar:2.2.2]
        at org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:652) ~[apache-cassandra-2.2.2.jar:2.2.2]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
...
ERROR [Reference-Reaper:1] 2015-10-14 06:23:21,439 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@74fc054a) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1949471967:/home/cassandra/dsc-cassandra-2.2.2/bin/../data/data/keyspace/table-b15521b0621111e4bbedcdee5e027297/la-1195-big was not released before the reference was garbage collected

Issue Links

• is cloned by: CASSANDRA-12860 Nodetool repair fragile: cannot properly recover from single node failure. Has to restart all nodes in order to repair again