[CASSANDRA-12088] Upgrade corrupts SSTables

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Duplicate
• Fix Version/s: 3.0.14, 3.11.5

Environment

OS: CentOS release 6.7 (Final)
Cassandra version: 2.1, 3.7

• Severity: Critical
• Assignee: Sam Tunnicliffe
• Reporter: Chandra Sekar S

Authors

Sam Tunnicliffe

Reviewers

Jeff Jirsa

• Created: 25/Jun/16 04:21
• Updated: 01/Aug/21 12:31
• Resolved: 09/May/17 16:57

Description

When upgrading from 2.0 to 3.7, table was corrupted and an exception occurs when performing LWT from Java Driver. The server was upgraded from 2.0 to 2.1 and then to 3.7. "nodetool upgradesstables" was run after each step of upgrade.

Schema of affected table:

CREATE TABLE payment.tbl (
    c1 text,
    c2 timestamp,
    c3 text,
    s1 timestamp static,
    s2 int static,
    c4 text,
    PRIMARY KEY (c1, c2)
) WITH CLUSTERING ORDER BY (c2 ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

Insertion that fails:

insert into tbl (c1, s2) values ('value', 0) if not exists;

The stack trace in system.log of cassandra server,

INFO  [HANDSHAKE-maven-repo.corp.zeta.in/10.1.5.13] 2016-06-24 22:23:14,887 OutboundTcpConnection.java:514 - Handshaking version with maven-repo.corp.zeta.in/10.1.5.13
ERROR [MessagingService-Incoming-/10.1.5.13] 2016-06-24 22:23:14,889 CassandraDaemon.java:217 - Exception in thread Thread[MessagingService-Incoming-/10.1.5.13,5,main]
java.io.IOError: java.io.IOException: Corrupt flags value for unfiltered partition (isStatic flag set): 160
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224) ~[apache-cassandra-3.7.0.jar:3.7.0]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212) ~[apache-cassandra-3.7.0.jar:3.7.0]
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.0.jar:3.7.0]
	at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681) ~[apache-cassandra-3.7.0.jar:3.7.0]
	at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642) ~[apache-cassandra-3.7.0.jar:3.7.0]
	at org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:131) ~[apache-cassandra-3.7.0.jar:3.7.0]
	at org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:97) ~[apache-cassandra-3.7.0.jar:3.7.0]
	at org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:66) ~[apache-cassandra-3.7.0.jar:3.7.0]
	at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114) ~[apache-cassandra-3.7.0.jar:3.7.0]
	at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190) ~[apache-cassandra-3.7.0.jar:3.7.0]
	at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) ~[apache-cassandra-3.7.0.jar:3.7.0]
	at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[apache-cassandra-3.7.0.jar:3.7.0]
Caused by: java.io.IOException: Corrupt flags value for unfiltered partition (isStatic flag set): 160
	at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:380) ~[apache-cassandra-3.7.0.jar:3.7.0]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219) ~[apache-cassandra-3.7.0.jar:3.7.0]
	... 11 common frames omitted

Issue Links

• duplicates: CASSANDRA-13236 corrupt flag error after upgrade from 2.2 to 3.0.10
• is related to: CASSANDRA-13236 corrupt flag error after upgrade from 2.2 to 3.0.10