[CASSANDRA-658] Hinted Handoff CF contention

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 0.6

Environment

debian lenny amd64 OpenJDK 64-Bit Server VM (build 1.6.0_0-b11, mixed mode)

• Severity: Normal
• Assignee: Jonathan Ellis
• Reporter: Brandon Williams

Authors

Jonathan Ellis

• Created: 30/Dec/09 02:44
• Updated: 16/Apr/19 09:33
• Resolved: 07/Jan/10 05:41

Description

Hinted handoff causes a lot of contention on the HH CF, causing insert speed to massively drop. Most of the row mutation stage threads end up blocking on each other at Memtable.resolve. This is because HH sends the hint to the closest node, which will always be the node handling the write.

To reproduce: start a cluster with even InitialTokens, and begin a constant stream of writes to one node, with an even key distribution. (I used 4 nodes and stress.py in random mode.) Take a node down, and the insert rate begin to drop, eventually settling between 100-300/s and sustaining there. Bringing the down node back up will restore the original insert rate.