Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6265

Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control to launch/kill containers

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.7.0
    • mrv2
    • None
    • Reviewed

    Description

      make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers
      Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at

        protected static final int INITIAL_POOL_SIZE = 10;
      

      We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
      Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
      We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node.

      Attachments

        1. MAPREDUCE-6265.000.patch
          9 kB
          Zhihai Xu
        2. MAPREDUCE-6265.001.patch
          9 kB
          Zhihai Xu
        3. MAPREDUCE-6265.002.patch
          9 kB
          Zhihai Xu

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zxu Zhihai Xu
            zxu Zhihai Xu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment