[FLINK-18035] Executors#newCachedThreadPool could not work as expected

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.10.2, 1.11.0
• Fix Version/s: 1.10.2, 1.11.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Chesnay Schepler
• Reporter: Yang Wang
• Created: 29/May/20 11:29
• Updated: 02/Jun/20 12:35
• Resolved: 31/May/20 21:45

Description

In FLINK-17558, we introduce Executors#newCachedThreadPool to create dedicated thread pool for TaskManager io. However, it could not work as expected.

The root cause is about the following constructor of ThreadPoolExecutor. Only when the workQueue is full, new thread will be started then. So if we set a LinkedBlockingQueue with Integer.MAX_VALUE capacity, only one thread will be started. It never grows up.

 

public ThreadPoolExecutor(int corePoolSize,
                          int maximumPoolSize,
                          long keepAliveTime,
                          TimeUnit unit,
                          BlockingQueue<Runnable> workQueue,
                          ThreadFactory threadFactory)

Issue Links

• is caused by: FLINK-17558 Partitions are released in TaskExecutor Main Thread
• links to: GitHub Pull Request #12409