[CASSANDRA-11014] Repair fails with "not enough bytes"

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Duplicate

Environment

3 node cluster, debian jessie, cassandra 2.2.4

• Severity: Critical
• Assignee: Unassigned
• Reporter: Christian Schjolberg
• Created: 14/Jan/16 12:42
• Updated: 16/Apr/19 09:30
• Resolved: 27/Jul/16 14:06

Description

After upgrading to 2.2.4, nodetool repair fails every time with the error message "Not enough bytes". It appears no data is being repaired at all. Here's some output:

-@cas01:~$ nodetool repair
[2016-01-14 12:00:16,590] Starting repair command #1, repairing keyspace adsquare with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 768)
[2016-01-14 12:00:21,935] Repair session 61174f80-bab6-11e5-9fa9-11175757c857 for range (-3942884673882176939,-3929110923969659376] failed with error repair #61174f80-bab6-11e5-9fa9-11175757c857 on adsquare/device_lookup, (-3942884673882176939,-3929110923969659376] Validation failed in /10.10.100.61 (progress: 0%)

The system.log on the host in question shows
ERROR [ValidationExecutor:2] 2016-01-14 09:58:19,935 CassandraDaemon.java:185 - Exception in thread Thread[ValidationExecutor:2,1,main]
java.lang.IllegalArgumentException: Not enough bytes
at org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362) ~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.db.composites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98) ~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381) ~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365) ~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.2.4.jar:2.2.4]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:169) ~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202) ~[apache-cassandra-2.2.4.jar:2.2.4]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
at org.apache.cassandra.db.compaction.LazilyCompactedRow.update(LazilyCompactedRow.java:172) ~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.repair.Validator.rowHash(Validator.java:194) ~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.repair.Validator.add(Validator.java:143) ~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1118) ~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:73) ~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:671) ~[apache-cassandra-2.2.4.jar:2.2.4]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]

Thinking that there might be some corruption I ran nodetool scrub, which fills the log with statements like the one in the attachment.

Issue Links

• duplicates: CASSANDRA-10961 Not enough bytes error when add nodes to cluster