[CASSANDRA-15869] MemoryOutputStream overflow on large bloom filter

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-beta1, 4.0
• Component/s: Local/SSTable
• Bug Category: Correctness - Recoverable Corruption / Loss
• Severity: Normal
• Complexity: Normal
• Discovered By: Performance Regression Test
• Platform: All
• Since Version: 4.0-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/7cdad3ce61664cf2ffac75b6e537c910e573754c

Test and Documentation Plan

Large sstable can be loaded again.

 

CircleCi: https://circleci.com/workflow-run/69685b9a-f356-46ef-939f-851cc1d29a4f

• Assignee: Zhao Yang
• Reporter: Zhao Yang

Authors

Zhao Yang

Reviewers

Brandon Williams

• Created: 10/Jun/20 10:16
• Updated: 03/Jan/21 17:03
• Resolved: 12/Jun/20 14:17

Description

Since CASSANDRA-9067, Cassandra will use MemoryOutputStream to reconstruct BF Memory without re-ordering bytes, see OffsetBitSet#deserialize. But MemoryOutputStream use INT to track position and will overflow when BF size exceeds 2GB.

stacktrace

error: Illegal bounds [-2147483648..-2147483584); size: 4588588016
-- StackTrace --
java.lang.AssertionError: Illegal bounds [-2147483648..-2147483584); size: 4588588016
	at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:185)
	at org.apache.cassandra.io.util.Memory.setBytes(Memory.java:138)
	at org.apache.cassandra.io.util.MemoryOutputStream.write(MemoryOutputStream.java:45)

Issue Links

• is related to: CASSANDRA-9067 BloomFilter serialization format should not change byte ordering