Hadoop YARN
  1. Hadoop YARN
  2. YARN-543 [Umbrella] NodeManager localization related issues
  3. YARN-574

PrivateLocalizer does not support parallel resource download via ContainerLocalizer

    Details

    • Type: Sub-task Sub-task
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0, 2.8.0, 2.7.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Hide
      YARN-574. Allow parallel download of resources in PrivateLocalizer. Contributed by Zheng Shao.
      Show
      YARN-574 . Allow parallel download of resources in PrivateLocalizer. Contributed by Zheng Shao.

      Description

      At present private resources will be downloaded in parallel only if multiple containers request the same resource. However otherwise it will be serial. The protocol between PrivateLocalizer and ContainerLocalizer supports multiple downloads however it is not used and only one resource is sent for downloading at a time.

      I think we can increase / assure parallelism (even for single container requesting resource) for private/application resources by making multiple downloads per ContainerLocalizer.
      Total Parallelism before
      = number of threads allotted for PublicLocalizer [public resource] + number of containers[private and application resource]
      Total Parallelism after
      = number of threads allotted for PublicLocalizer [public resource] + number of containers * max downloads per container [private and application resource]

      1. YARN-574.1.patch
        6 kB
        Zheng Shao
      2. YARN-574.2.patch
        6 kB
        Zheng Shao

        Activity

        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 17s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 37s There were no new javac warning messages.
        +1 javadoc 9m 37s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 18s The applied patch generated 2 new checkstyle issues (total was 213, now 214).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 2m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api.
        +1 yarn tests 6m 4s Tests passed in hadoop-yarn-server-nodemanager.
            47m 49s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12737994/YARN-574.1.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 7588585
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8199/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8199/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8199/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8199/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8199/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 17s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 18s The applied patch generated 2 new checkstyle issues (total was 213, now 214). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api. +1 yarn tests 6m 4s Tests passed in hadoop-yarn-server-nodemanager.     47m 49s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737994/YARN-574.1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7588585 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8199/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8199/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8199/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8199/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8199/console This message was automatically generated.
        Hide
        Zheng Shao added a comment -

        Fixed syntax error.

        Show
        Zheng Shao added a comment - Fixed syntax error.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 16m 21s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 36s There were no new javac warning messages.
        +1 javadoc 9m 38s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 11s The applied patch generated 1 new checkstyle issues (total was 213, now 213).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 2m 46s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 22s Tests passed in hadoop-yarn-api.
        +1 yarn tests 6m 3s Tests passed in hadoop-yarn-server-nodemanager.
            46m 38s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12738110/YARN-574.2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 71de367
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8206/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8206/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 21s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 11s The applied patch generated 1 new checkstyle issues (total was 213, now 213). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 46s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 22s Tests passed in hadoop-yarn-api. +1 yarn tests 6m 3s Tests passed in hadoop-yarn-server-nodemanager.     46m 38s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738110/YARN-574.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 71de367 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8206/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8206/console This message was automatically generated.

          People

          • Assignee:
            Omkar Vinit Joshi
            Reporter:
            Omkar Vinit Joshi
          • Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:

              Development