[HUDI-5303] Allow users to control the concurrency to submit jobs in clustering

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: clustering, spark
• Labels: pull-request-available
• Assignee: Hui An
• Reporter: Hui An
• Created: 30/Nov/22 13:06
• Updated: 27/Jun/23 01:57
• Resolved: 27/Jun/23 01:57

Description

If there are sufficient resources in the clustering job, some clustering groups sometimes could still waits to be triggered, we use forkJoinPool to submit these jobs, and it's also difficult for clients to adjust this configure(--conf spark.driver.extraJavaOptions=-Djava.util.concurrent.ForkJoinPool.common.parallelism), and it could also affect other tasks using the forkJoinPool, so instead, we introduce a new threadPool to control the submitting job parallelism for the clustering.

Issue Links

• links to: GitHub Pull Request #7343