Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4198

TezContainerLauncherImpl launch and stop Container work

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.9.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      https://github.com/apache/tez/blob/3f2373e2b2ab3825ef50e9f19b8704265542a8b2/tez-dag/src/main/java/org/apache/tez/dag/app/launcher/TezContainerLauncherImpl.java#L429

      https://github.com/apache/tez/blob/3f2373e2b2ab3825ef50e9f19b8704265542a8b2/tez-dag/src/main/java/org/apache/tez/dag/app/launcher/TezContainerLauncherImpl.java#L326

      We are putting all the events (container events) in the event queue (Which is a blocking queue). And then on a separate thread we just take items from this queue and add it to a thread pool executor, which also has an internal blocking queue.

      This is a redundant creation of blocking queue and the thread to execute operations on the executor service. 

      Also the executor service increases it's core pool size thread but never decreases it once the workload decreases. The core threads once started don't die unless we shutdown the service or we specify which we are also not doing. Also having the threads live for 1 hour is a stretch as we are potentially occupying resources that we don't use that often, especially when we are reusing containers ? 
      We also don't need to specify INTEGER.MAX as max core pool size, but let it be the max value we want. The declaration of the thread pool service requires changes. https://github.com/apache/tez/blob/3f2373e2b2ab3825ef50e9f19b8704265542a8b2/tez-dag/src/main/java/org/apache/tez/dag/app/launcher/TezContainerLauncherImpl.java#L283

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              gdhody Gurleen S Dhody
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: