Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6265

Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control to launch/kill containers

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers
      Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at

        protected static final int INITIAL_POOL_SIZE = 10;
      

      We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
      Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
      We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node.

        Attachments

        1. MAPREDUCE-6265.000.patch
          9 kB
          Zhihai Xu
        2. MAPREDUCE-6265.001.patch
          9 kB
          Zhihai Xu
        3. MAPREDUCE-6265.002.patch
          9 kB
          Zhihai Xu

          Activity

            People

            • Assignee:
              zxu Zhihai Xu
              Reporter:
              zxu Zhihai Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: