[CASSANDRA-15880] Memory leak in CompressedChunkReader

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.11.9, 4.0-beta3, 4.0
• Component/s: Feature/Compression
• Bug Category: Degradation - Resource Management
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.6

Source Control Link

https://github.com/apache/cassandra/commit/3f73c16d50a80c574dd004c59bfaa6042dcd5781

Test and Documentation Plan

See PR

• Assignee: Berenguer Blasi
• Reporter: Jaroslaw Grabowski

Authors

Berenguer Blasi

Reviewers

Benjamin Lerer, Branimir Lambov

• Created: 18/Jun/20 09:04
• Updated: 03/Jan/21 17:03
• Resolved: 02/Oct/20 15:26

Description

CompressedChunkReader uses java.lang.ThreadLocal to reuse ByteBuffer for compressed data. ByteBuffers leak due to peculiar ThreadLocal quality.
ThreadLocals are stored in a map, where the key is a weak reference to a ThreadLocal and the value is the user's object (ByteBuffer in this case). When a last strong reference to a ThreadLocal is lost, weak reference to ThreadLocal (key) is removed but the value (ByteBuffer) is kept until cleaned by ThreadLocal heuristic expunge mechanism. See ThreadLocal's "stale entries" for details.

When a number of long-living threads is high enough this results in thousands of ByteBuffers stored as stale entries in ThreadLocals. In a not-so-lucky scenario we get OutOfMemoryException.

Issue Links

• links to: CASSANDRA-15880 3.11 PR
• links to: CASSANDRA-15880 4.0 PR