[SPARK-49544] Severe lock contention in SparkConnectExecutionManager - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

Delete

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 4.0.0
Fix Version/s: 4.0.0
Component/s: Connect
Labels:
- pull-request-available

Description

Critical sections protected by executionsLock can become too broad when there are too many ExecuteHolders, e.g., >= 10^4. The problem is aggravated when there are too many threads in the system: priority inversion. In order to minimise the chance of a thread getting pre-empted holding executionsLock, replace it with a concurrent hash map that internally partitions the data: coarse-grained locking -> fine-grained locking.

-> https://issues.apache.org/jira/browse/SPARK-49580 will be eventually needed.