[TEZ-4198] TezContainerLauncherImpl launch and stop Container work

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.1
• Labels: refactor
• Assignee: Unassigned
• Reporter: Gurleen S Dhody
• Created: 12/Jul/20 21:40
• Updated: 12/Jul/20 21:41

Description

https://github.com/apache/tez/blob/3f2373e2b2ab3825ef50e9f19b8704265542a8b2/tez-dag/src/main/java/org/apache/tez/dag/app/launcher/TezContainerLauncherImpl.java#L429

https://github.com/apache/tez/blob/3f2373e2b2ab3825ef50e9f19b8704265542a8b2/tez-dag/src/main/java/org/apache/tez/dag/app/launcher/TezContainerLauncherImpl.java#L326

We are putting all the events (container events) in the event queue (Which is a blocking queue). And then on a separate thread we just take items from this queue and add it to a thread pool executor, which also has an internal blocking queue.

This is a redundant creation of blocking queue and the thread to execute operations on the executor service. 

Also the executor service increases it's core pool size thread but never decreases it once the workload decreases. The core threads once started don't die unless we shutdown the service or we specify which we are also not doing. Also having the threads live for 1 hour is a stretch as we are potentially occupying resources that we don't use that often, especially when we are reusing containers ? 
We also don't need to specify INTEGER.MAX as max core pool size, but let it be the max value we want. The declaration of the thread pool service requires changes. https://github.com/apache/tez/blob/3f2373e2b2ab3825ef50e9f19b8704265542a8b2/tez-dag/src/main/java/org/apache/tez/dag/app/launcher/TezContainerLauncherImpl.java#L283