[CASSANDRA-15752] Range read concurrency factor didn't consider range merger

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.21, 3.11.7, 4.0-beta1, 4.0
• Component/s: Legacy/Coordination
• Bug Category: Degradation - Performance Bug/Regression
• Severity: Normal
• Complexity: Normal
• Discovered By: Code Inspection
• Platform: All
• Since Version: 2.1 beta1

Source Control Link

https://github.com/apache/cassandra/commit/abdf5085d4381351054bc2c0976bc826f4ac82e2

Test and Documentation Plan

Added unit tests.

Circle Ci: https://circleci.com/workflow-run/440126c4-058d-4511-8301-16df2bddf5db
 
 

• Assignee: Zhao Yang
• Reporter: Zhao Yang

Authors

Zhao Yang

Reviewers

Andres de la Peña, Caleb Rackliffe

• Created: 22/Apr/20 16:04
• Updated: 03/Jan/21 17:02
• Resolved: 22/Jun/20 15:29

Description

During range read, coordinator computes concurrency factor which is the number of vnode ranges to contact in parallel for the next batch.

But in RangeCommandIterator, vnode ranges are merged by RangeMerger if vnode ranges share enough replicas to satisfy consistency level. eg. vnode range [a,b) has replica n1,n2,n3 and vnode range [b,c) has replica n2,n3,n4, so they can be merged as range [a,c) with replica n2, n3 for Quorum.

Currently it counts number of merged ranges towards concurrency factor. Coordinator may fetch more ranges than needed.

---

Another issue is that when executing range read on table with very small amount of data, concurrency factor can be bumped to size of total vnode ranges, eg. 10k, depending on the num of vnodes and cluster size. As a result, coordinator will send large number of concurrent range requests, potentially slowing down the cluster.. We should cap the max concurrency factor..