Hadoop YARN
  1. Hadoop YARN
  2. YARN-543 [Umbrella] NodeManager localization related issues
  3. YARN-574

PrivateLocalizer does not support parallel resource download via ContainerLocalizer

    Details

    • Type: Sub-task Sub-task
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0, 2.8.0, 2.7.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Hide
      YARN-574. Allow parallel download of resources in PrivateLocalizer. Contributed by Zheng Shao.
      Show
      YARN-574 . Allow parallel download of resources in PrivateLocalizer. Contributed by Zheng Shao.

      Description

      At present private resources will be downloaded in parallel only if multiple containers request the same resource. However otherwise it will be serial. The protocol between PrivateLocalizer and ContainerLocalizer supports multiple downloads however it is not used and only one resource is sent for downloading at a time.

      I think we can increase / assure parallelism (even for single container requesting resource) for private/application resources by making multiple downloads per ContainerLocalizer.
      Total Parallelism before
      = number of threads allotted for PublicLocalizer [public resource] + number of containers[private and application resource]
      Total Parallelism after
      = number of threads allotted for PublicLocalizer [public resource] + number of containers * max downloads per container [private and application resource]

      1. YARN-574.03.patch
        14 kB
        Ajith S
      2. YARN-574.1.patch
        6 kB
        Zheng Shao
      3. YARN-574.2.patch
        6 kB
        Zheng Shao

        Activity

        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 17s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 37s There were no new javac warning messages.
        +1 javadoc 9m 37s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 18s The applied patch generated 2 new checkstyle issues (total was 213, now 214).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 2m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api.
        +1 yarn tests 6m 4s Tests passed in hadoop-yarn-server-nodemanager.
            47m 49s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12737994/YARN-574.1.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 7588585
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8199/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8199/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8199/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8199/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8199/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 17s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 18s The applied patch generated 2 new checkstyle issues (total was 213, now 214). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api. +1 yarn tests 6m 4s Tests passed in hadoop-yarn-server-nodemanager.     47m 49s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737994/YARN-574.1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7588585 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8199/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8199/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8199/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8199/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8199/console This message was automatically generated.
        Hide
        Zheng Shao added a comment -

        Fixed syntax error.

        Show
        Zheng Shao added a comment - Fixed syntax error.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 16m 21s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 36s There were no new javac warning messages.
        +1 javadoc 9m 38s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 11s The applied patch generated 1 new checkstyle issues (total was 213, now 213).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 2m 46s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 22s Tests passed in hadoop-yarn-api.
        +1 yarn tests 6m 3s Tests passed in hadoop-yarn-server-nodemanager.
            46m 38s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12738110/YARN-574.2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 71de367
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8206/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8206/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 21s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 11s The applied patch generated 1 new checkstyle issues (total was 213, now 213). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 46s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 22s Tests passed in hadoop-yarn-api. +1 yarn tests 6m 3s Tests passed in hadoop-yarn-server-nodemanager.     46m 38s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738110/YARN-574.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 71de367 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8206/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8206/console This message was automatically generated.
        Hide
        Naganarasimha G R added a comment -

        Omkar Vinit Joshi, Seems like overall patch is fine except for the logic that in ContainerLocalizer.localizeFiles we can optimize to receive all the resources to be localized in one shot (already we receive it as list<ResourceLocalizationSpec> so just need to confirm the same in Resourceloclization service) and avoid polling/hb the server frequently just based on the number of threads in the Container Localizer.
        Retriggering the build to see whether old patch works!

        Show
        Naganarasimha G R added a comment - Omkar Vinit Joshi , Seems like overall patch is fine except for the logic that in ContainerLocalizer.localizeFiles we can optimize to receive all the resources to be localized in one shot (already we receive it as list<ResourceLocalizationSpec> so just need to confirm the same in Resourceloclization service) and avoid polling/hb the server frequently just based on the number of threads in the Container Localizer. Retriggering the build to see whether old patch works!
        Hide
        Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 6s YARN-574 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12738110/YARN-574.2.patch
        JIRA Issue YARN-574
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12803/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s YARN-574 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12738110/YARN-574.2.patch JIRA Issue YARN-574 Console output https://builds.apache.org/job/PreCommit-YARN-Build/12803/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 5s YARN-574 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12738110/YARN-574.2.patch
        JIRA Issue YARN-574
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12804/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s YARN-574 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12738110/YARN-574.2.patch JIRA Issue YARN-574 Console output https://builds.apache.org/job/PreCommit-YARN-Build/12804/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        Naganarasimha G R added a comment -

        Omkar Vinit Joshi seems like its not getting applied to trunk, Are you planning to work in it ?

        Show
        Naganarasimha G R added a comment - Omkar Vinit Joshi seems like its not getting applied to trunk, Are you planning to work in it ?
        Hide
        Ajith S added a comment -

        Have a requirement for this. Omkar Vinit Joshi can i work on this.?

        Show
        Ajith S added a comment - Have a requirement for this. Omkar Vinit Joshi can i work on this.?
        Hide
        Jason Lowe added a comment -

        Cancelling the patch as it no longer applies.

        Show
        Jason Lowe added a comment - Cancelling the patch as it no longer applies.
        Hide
        Ajith S added a comment -

        I will take it over, if you are working on it, please reassign

        Show
        Ajith S added a comment - I will take it over, if you are working on it, please reassign
        Hide
        Ajith S added a comment -

        I have rebased and added testcase for the patch. Please review

        Show
        Ajith S added a comment - I have rebased and added testcase for the patch. Please review
        Hide
        Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 7m 10s trunk passed
        +1 compile 6m 48s trunk passed
        +1 checkstyle 0m 54s trunk passed
        +1 mvnsite 1m 16s trunk passed
        +1 mvneclipse 0m 48s trunk passed
        +1 findbugs 2m 8s trunk passed
        +1 javadoc 0m 55s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 0m 46s the patch passed
        +1 compile 5m 17s the patch passed
        +1 javac 5m 17s the patch passed
        -0 checkstyle 0m 54s hadoop-yarn-project/hadoop-yarn: The patch generated 10 new + 397 unchanged - 1 fixed = 407 total (was 398)
        +1 mvnsite 1m 14s the patch passed
        +1 mvneclipse 0m 43s the patch passed
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 2m 19s the patch passed
        +1 javadoc 0m 54s the patch passed
        -1 unit 0m 35s hadoop-yarn-api in the patch failed.
        -1 unit 15m 57s hadoop-yarn-server-nodemanager in the patch failed.
        +1 asflicense 0m 38s The patch does not generate ASF License warnings.
        58m 22s



        Reason Tests
        Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields
          hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e809691
        JIRA Issue YARN-574
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837742/YARN-574.03.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f6471cf3e508 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b970446
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13807/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13807/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13807/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13807/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13807/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13807/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 7m 10s trunk passed +1 compile 6m 48s trunk passed +1 checkstyle 0m 54s trunk passed +1 mvnsite 1m 16s trunk passed +1 mvneclipse 0m 48s trunk passed +1 findbugs 2m 8s trunk passed +1 javadoc 0m 55s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 46s the patch passed +1 compile 5m 17s the patch passed +1 javac 5m 17s the patch passed -0 checkstyle 0m 54s hadoop-yarn-project/hadoop-yarn: The patch generated 10 new + 397 unchanged - 1 fixed = 407 total (was 398) +1 mvnsite 1m 14s the patch passed +1 mvneclipse 0m 43s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 2m 19s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 0m 35s hadoop-yarn-api in the patch failed. -1 unit 15m 57s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 58m 22s Reason Tests Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue YARN-574 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837742/YARN-574.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f6471cf3e508 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b970446 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13807/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13807/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13807/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13807/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13807/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13807/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Varun Saxena added a comment -

        Ajith S, thanks for the patch.
        Parallel downloads should speed up container localization phase.
        Coming to the patch, should the configuration value be 4 ? Or should we keep it as 1 i.e. current behavior. And then it can be tuned based on size and visibility of resources to be downloaded, if somebody wants to use it.
        Localizers are not treated as containers which means resources used by them are not accounted for, so it should not be that they all together end up eating up quite a bit of resources on the node with default value of 4.
        Thoughts ?

        Show
        Varun Saxena added a comment - Ajith S , thanks for the patch. Parallel downloads should speed up container localization phase. Coming to the patch, should the configuration value be 4 ? Or should we keep it as 1 i.e. current behavior. And then it can be tuned based on size and visibility of resources to be downloaded, if somebody wants to use it. Localizers are not treated as containers which means resources used by them are not accounted for, so it should not be that they all together end up eating up quite a bit of resources on the node with default value of 4. Thoughts ?
        Hide
        Naganarasimha G R added a comment -

        Thanks Ajith S, for the patch.
        As discussed offline, I think existing approach of sending only one resource as HB response is better as there can be some control at NM end to determine (if req in future) whether to give multiple resources to be localized or one at a time(or even none based on the NM's load). Further to it i would to add further points

        1. Agree to Varun Saxena's point having fixed 4 threads (core and max pool size ) is not ideal. But IMHO i would like to keep atleast 2(default max pool size), as users might not be able make use of the benifit if not.
        2. I think instead of using Executors.newFixedThreadPool(nThreads, tf) we can try to use ThreadPoolExecutor so that we can we can make use of ThreadPoolExecutor.getActiveCount() and compare it with max pool size to determine whether to request Heartbeat immediately or to wait 1000s.
        3. If NM sends LIVE and if no ResourceLocalizationSpecs are shared across then no need to check the current load on the executor, we can wait for the defined poll period and then do the HB.
        4. TestContainerLocalizer.java, ln no 301-306, Please add some proper message which can be shown on failure.
        5. TestYarnConfigurationFields.testCompareConfigurationClassAgainstXml is related to the patch
        Show
        Naganarasimha G R added a comment - Thanks Ajith S , for the patch. As discussed offline, I think existing approach of sending only one resource as HB response is better as there can be some control at NM end to determine (if req in future) whether to give multiple resources to be localized or one at a time(or even none based on the NM's load). Further to it i would to add further points Agree to Varun Saxena 's point having fixed 4 threads (core and max pool size ) is not ideal. But IMHO i would like to keep atleast 2(default max pool size), as users might not be able make use of the benifit if not. I think instead of using Executors.newFixedThreadPool(nThreads, tf) we can try to use ThreadPoolExecutor so that we can we can make use of ThreadPoolExecutor.getActiveCount() and compare it with max pool size to determine whether to request Heartbeat immediately or to wait 1000s. If NM sends LIVE and if no ResourceLocalizationSpecs are shared across then no need to check the current load on the executor, we can wait for the defined poll period and then do the HB. TestContainerLocalizer.java, ln no 301-306, Please add some proper message which can be shown on failure. TestYarnConfigurationFields.testCompareConfigurationClassAgainstXml is related to the patch
        Hide
        Naganarasimha G R added a comment -

        Also would like to get the feedback from Jason Lowe, Vinod Kumar Vavilapalli, Devaraj K & others in the watcher list.

        Show
        Naganarasimha G R added a comment - Also would like to get the feedback from Jason Lowe , Vinod Kumar Vavilapalli , Devaraj K & others in the watcher list.

          People

          • Assignee:
            Ajith S
            Reporter:
            Omkar Vinit Joshi
          • Votes:
            1 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

            • Created:
              Updated:

              Development