[CASSANDRA-13632] Digest mismatch if row is empty

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Cannot Reproduce
• Fix Version/s: 3.0.25
• Component/s: Legacy/Local Write-Read Paths
• Severity: Normal
• Assignee: Andrew Whang
• Reporter: Andrew Whang

Authors

Andrew Whang

• Created: 22/Jun/17 23:47
• Updated: 01/Aug/21 11:43
• Resolved: 29/Jun/21 15:45

Description

This issue is similar to CASSANDRA-12090. Quorum read queries that include a column selector (non-wildcard) result in digest mismatch when the row is empty (key does not exist). It seems the data serialization path checks if rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY flag). However, the digest serialization path does not perform this check and includes column names. The digest comparison results in a mismatch. The mismatch does not end up issuing a read repair mutation since the underlying data is the same.

The mismatch on the read path ends up doubling our p99 read latency. We discovered this issue while testing a 2.2.5 to 3.0.13 upgrade.

One thing to note is that we're using thrift, which ends up handling the ColumnFilter differently than the CQL path.

As with CASSANDRA-12090, fixing the digest seems sensible.