[SPARK-49544] Severe lock contention in SparkConnectExecutionManager

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect
• Labels: pull-request-available
• Assignee: Changgyoo Park
• Reporter: Changgyoo Park
• Created: 09/Sep/24 09:37
• Updated: 11/Sep/24 23:39
• Resolved: 11/Sep/24 23:39

Description

Critical sections protected by executionsLock can become too broad when there are too many ExecuteHolders, e.g., >= 10^4. The problem is aggravated when there are too many threads in the system: priority inversion. In order to minimise the chance of a thread getting pre-empted holding executionsLock, replace it with a concurrent hash map that internally partitions the data: coarse-grained locking -> fine-grained locking.

-> https://issues.apache.org/jira/browse/SPARK-49580 will be eventually needed.

Issue Links

• links to: GitHub Pull Request #48034