[CASSANDRA-8409] Node generating a huge number of tiny sstable_activity flushes

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 2.1.3

Environment

Cassandra 2.1.0, Oracle JDK 1.8.0_25, Ubuntu 12.04

• Severity: Normal
• Assignee: Unassigned
• Reporter: Fred Wulff
• Created: 02/Dec/14 21:51
• Updated: 16/Apr/19 09:31
• Resolved: 30/Dec/14 14:50

Description

On one of my nodes, I’m seeing hundreds per second of “INFO 21:28:05 Enqueuing flush of sstable_activity: 0 (0%) on-heap, 33 (0%) off-heap”. tpstats shows a steadily climbing # of pending MemtableFlushWriter/MemtablePostFlush until the node OOMs. When the flushes actually happen the sstable written is invariably 121 bytes. I’m writing pretty aggressively to one of my user tables (sev.mdb_group_pit), but that table's flushing behavior seems reasonable.

tpstats:

frew@hostname:~/s_dist/apache-cassandra-2.1.0$ bin/nodetool -h hostname tpstats
Pool Name Active Pending Completed Blocked All time blocked
MutationStage 128 4429 36810 0 0
ReadStage 0 0 1205 0 0
RequestResponseStage 0 0 24910 0 0
ReadRepairStage 0 0 26 0 0
CounterMutationStage 0 0 0 0 0
MiscStage 0 0 0 0 0
HintedHandoff 2 2 9 0 0
GossipStage 0 0 5157 0 0
CacheCleanupExecutor 0 0 0 0 0
InternalResponseStage 0 0 0 0 0
CommitLogArchiver 0 0 0 0 0
CompactionExecutor 4 28 429 0 0
ValidationExecutor 0 0 0 0 0
MigrationStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
PendingRangeCalculator 0 0 11 0 0
MemtableFlushWriter 8 38644 8987 0 0
MemtablePostFlush 1 38940 8735 0 0
MemtableReclaimMemory 0 0 8987 0 0

Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 0
MUTATION 10457
COUNTER_MUTATION 0
BINARY 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 208

I've attached one of the produced sstables.

Issue Links

• duplicates: CASSANDRA-8018 Cassandra seems to insert twice in custom PerColumnSecondaryIndex