Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-543 [Umbrella] NodeManager localization related issues
  3. YARN-574

PrivateLocalizer does not support parallel resource download via ContainerLocalizer

    Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0, 2.8.0, 2.7.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Hide
      YARN-574. Allow parallel download of resources in PrivateLocalizer. Contributed by Zheng Shao.
      Show
      YARN-574 . Allow parallel download of resources in PrivateLocalizer. Contributed by Zheng Shao.

      Description

      At present private resources will be downloaded in parallel only if multiple containers request the same resource. However otherwise it will be serial. The protocol between PrivateLocalizer and ContainerLocalizer supports multiple downloads however it is not used and only one resource is sent for downloading at a time.

      I think we can increase / assure parallelism (even for single container requesting resource) for private/application resources by making multiple downloads per ContainerLocalizer.
      Total Parallelism before
      = number of threads allotted for PublicLocalizer [public resource] + number of containers[private and application resource]
      Total Parallelism after
      = number of threads allotted for PublicLocalizer [public resource] + number of containers * max downloads per container [private and application resource]

        Attachments

        1. YARN-574.1.patch
          6 kB
          Zheng Shao
        2. YARN-574.2.patch
          6 kB
          Zheng Shao
        3. YARN-574.03.patch
          14 kB
          Ajith S
        4. YARN-574.04.patch
          21 kB
          Ajith S
        5. YARN-574.05.patch
          23 kB
          Ajith S

          Activity

            People

            • Assignee:
              ajithshetty Ajith S
              Reporter:
              ojoshi Omkar Vinit Joshi
            • Votes:
              1 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated: