Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.2.0
    • None
    • None
    • Reviewed

    Description

      Add node/host to attribute mapping in yarn client API.

      Attachments

        1. YARN-8104-YARN-3409.006.patch
          59 kB
          Bibin Chundatt
        2. YARN-8104-YARN-3409.005.patch
          59 kB
          Bibin Chundatt
        3. YARN-8104-YARN-3409.004.patch
          59 kB
          Bibin Chundatt
        4. YARN-8104-YARN-3409.003.patch
          56 kB
          Bibin Chundatt
        5. YARN-8104-YARN-3409.002.patch
          56 kB
          Bibin Chundatt
        6. YARN-8104-YARN-3409.001.patch
          49 kB
          Bibin Chundatt

        Activity

          bibinchundatt Bibin Chundatt added a comment - - edited

          Attached first patch.
          sunil.govind@gmail.com/Naganarasimha/cheersyang please help in review

          bibinchundatt Bibin Chundatt added a comment - - edited Attached first patch. sunil.govind@gmail.com / Naganarasimha / cheersyang please help in review
          genericqa genericqa added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s YARN-8104 does not apply to YARN-3409. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          This message was automatically generated.

          genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s YARN-8104 does not apply to YARN-3409 . Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-8104 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12917543/YARN-8104-YARN-3409.001.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/20218/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          bibinchundatt Bibin Chundatt added a comment -

          Patch is failing since its over YARN-8100

          bibinchundatt Bibin Chundatt added a comment - Patch is failing since its over YARN-8100

          Hi bibinchundatt, Now that YARN-8100 is in can you rebase on this ?

          Also there was one point i thought of adding in 8100 , GetAttributesToNodesResponseProto could have had field attributeToNodes as attributesToNodes.

          Can you incorporate that here ?

          Naganarasimha Naganarasimha G R added a comment - Hi bibinchundatt , Now that YARN-8100 is in can you rebase on this ? Also there was one point i thought of adding in 8100 , GetAttributesToNodesResponseProto could have had field attributeToNodes as attributesToNodes. Can you incorporate that here ?
          bibinchundatt Bibin Chundatt added a comment -

          Thank you Naganarasimha for review

          Attached rebased patch and also handling your comments.

          bibinchundatt Bibin Chundatt added a comment - Thank you Naganarasimha for review Attached rebased patch and also handling your comments.
          genericqa genericqa added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 37s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                YARN-3409 Compile Tests
          0 mvndep 1m 25s Maven dependency ordering for branch
          +1 mvninstall 26m 23s YARN-3409 passed
          +1 compile 30m 3s YARN-3409 passed
          +1 checkstyle 3m 15s YARN-3409 passed
          +1 mvnsite 5m 0s YARN-3409 passed
          +1 shadedclient 19m 15s branch has no errors when building and testing our client artifacts.
          -1 findbugs 1m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-3409 has 1 extant Findbugs warnings.
          +1 javadoc 3m 50s YARN-3409 passed
                Patch Compile Tests
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 3m 54s the patch passed
          +1 compile 29m 34s the patch passed
          +1 cc 29m 34s the patch passed
          +1 javac 29m 34s the patch passed
          -0 checkstyle 3m 25s root: The patch generated 6 new + 261 unchanged - 0 fixed = 267 total (was 261)
          +1 mvnsite 4m 56s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 10m 28s patch has no errors when building and testing our client artifacts.
          -1 findbugs 1m 34s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 32s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
          -1 javadoc 0m 56s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common generated 1 new + 4189 unchanged - 0 fixed = 4190 total (was 4189)
                Other Tests
          +1 unit 0m 50s hadoop-yarn-api in the patch passed.
          +1 unit 3m 14s hadoop-yarn-common in the patch passed.
          +1 unit 2m 14s hadoop-yarn-server-common in the patch passed.
          +1 unit 66m 57s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 unit 27m 3s hadoop-yarn-client in the patch passed.
          +1 unit 1m 29s hadoop-yarn-server-router in the patch passed.
          -1 unit 127m 42s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 48s The patch does not generate ASF License warnings.
          387m 23s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Nullcheck of hostName at line 454 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.nodelabels.NodeAttributesManagerImpl.getNodesToAttributes(Set) At NodeAttributesManagerImpl.java:454 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.nodelabels.NodeAttributesManagerImpl.getNodesToAttributes(Set) At NodeAttributesManagerImpl.java:[line 454]



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b
          JIRA Issue YARN-8104
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12918329/YARN-8104-YARN-3409.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
          uname Linux 8a5fa31c02ec 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision YARN-3409 / efcca85
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20285/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20285/artifact/out/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20285/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/20285/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/20285/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/20285/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20285/testReport/
          Max. process+thread count 1211 (vs. ulimit of 10000)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/20285/console
          Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 37s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       YARN-3409 Compile Tests 0 mvndep 1m 25s Maven dependency ordering for branch +1 mvninstall 26m 23s YARN-3409 passed +1 compile 30m 3s YARN-3409 passed +1 checkstyle 3m 15s YARN-3409 passed +1 mvnsite 5m 0s YARN-3409 passed +1 shadedclient 19m 15s branch has no errors when building and testing our client artifacts. -1 findbugs 1m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-3409 has 1 extant Findbugs warnings. +1 javadoc 3m 50s YARN-3409 passed       Patch Compile Tests 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 3m 54s the patch passed +1 compile 29m 34s the patch passed +1 cc 29m 34s the patch passed +1 javac 29m 34s the patch passed -0 checkstyle 3m 25s root: The patch generated 6 new + 261 unchanged - 0 fixed = 267 total (was 261) +1 mvnsite 4m 56s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 10m 28s patch has no errors when building and testing our client artifacts. -1 findbugs 1m 34s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 32s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) -1 javadoc 0m 56s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common generated 1 new + 4189 unchanged - 0 fixed = 4190 total (was 4189)       Other Tests +1 unit 0m 50s hadoop-yarn-api in the patch passed. +1 unit 3m 14s hadoop-yarn-common in the patch passed. +1 unit 2m 14s hadoop-yarn-server-common in the patch passed. +1 unit 66m 57s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 27m 3s hadoop-yarn-client in the patch passed. +1 unit 1m 29s hadoop-yarn-server-router in the patch passed. -1 unit 127m 42s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 48s The patch does not generate ASF License warnings. 387m 23s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Nullcheck of hostName at line 454 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.nodelabels.NodeAttributesManagerImpl.getNodesToAttributes(Set) At NodeAttributesManagerImpl.java:454 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.nodelabels.NodeAttributesManagerImpl.getNodesToAttributes(Set) At NodeAttributesManagerImpl.java: [line 454] Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b JIRA Issue YARN-8104 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12918329/YARN-8104-YARN-3409.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc uname Linux 8a5fa31c02ec 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision YARN-3409 / efcca85 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20285/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20285/artifact/out/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20285/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/20285/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/20285/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/20285/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20285/testReport/ Max. process+thread count 1211 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20285/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

          Thanks for the patch bibinchundatt

          At high level

          • Can you inform why NodeToAttributesProto is moved from yarn_server_resourcemanager_service_protos.proto to yarn_protos.proto ?
          • Also would it make sense to provide overloaded method(getNodesToAttributes) here supporting for NodeID ?

          and few other comments :

          yarn_protos.protos

          • ln no 391: node => hostname ... similar to earlier naming convention

          yarn_service_protos.protos

          • ln no 282: nodeToAttributes => nodesToAttributes ... based on convention followed in other places

          GetNodesToAttributesRequest.java

          • line no 55 & 64 : setNodes & getNodes -> setHostNames & getHostNames

          GetNodesToAttributesRequestPBImpl

          • line no 120: initNodeAttributes => initNodesToAttributesRequest or just init
          • line no 126: nodeLabelsList => hostNamesList

          TestPBImplRecords

          • We need to invoke generateByNewInstance for all the new PB's in setup. can you please check.

          NodeAttributesManagerImpl

          • ln no 454-457: Here we do not have a mapping we are setting a hostname with empty set, is that better or just pass for the ones which have attributes is better?
            IMO for the ones having mapping is better so that we not bloating the response and its a map. Also if nonexistent of erroneous hostnames are given it still shows empty set

          TestClientRMService

          • ln no 2053: can we have a separate method to test node to attributes API ? or document what all api's will be tested. and rename it to testNodeAttributesQueryAPI... i would still prefer the former option itself though..

           

          Can you also check the new findbug issue reported, checktyle  and as well as the javadoc issues reported, as it seems related to patch and fixable ?

          Naganarasimha Naganarasimha G R added a comment - Thanks for the patch bibinchundatt At high level Can you inform why NodeToAttributesProto is moved from yarn_server_resourcemanager_service_protos.proto to yarn_protos.proto ? Also would it make sense to provide overloaded method(getNodesToAttributes) here supporting for NodeID ? and few other comments : yarn_protos.protos ln no 391: node => hostname ... similar to earlier naming convention yarn_service_protos.protos ln no 282: nodeToAttributes => nodesToAttributes ... based on convention followed in other places GetNodesToAttributesRequest.java line no 55 & 64 : setNodes & getNodes -> setHostNames & getHostNames GetNodesToAttributesRequestPBImpl line no 120: initNodeAttributes => initNodesToAttributesRequest or just init line no 126: nodeLabelsList => hostNamesList TestPBImplRecords We need to invoke generateByNewInstance for all the new PB's in setup. can you please check. NodeAttributesManagerImpl ln no 454-457: Here we do not have a mapping we are setting a hostname with empty set, is that better or just pass for the ones which have attributes is better? IMO for the ones having mapping is better so that we not bloating the response and its a map. Also if nonexistent of erroneous hostnames are given it still shows empty set TestClientRMService ln no 2053: can we have a separate method to test node to attributes API ? or document what all api's will be tested. and rename it to testNodeAttributesQueryAPI... i would still prefer the former option itself though..   Can you also check the new findbug issue reported, checktyle  and as well as the javadoc issues reported, as it seems related to patch and fixable ?
          bibinchundatt Bibin Chundatt added a comment -

          Thank you Naganarasimha for review

          Can you inform why NodeToAttributesProto is moved from yarn_server_resourcemanager_service_protos.proto to yarn_protos.proto

          GetNodesToAttributesResponseProto yarn_service.proto requires the same yarn_protos.

          Also would it make sense to provide overloaded method(getNodesToAttributes) here supporting for NodeID ?

          Currently IMHO don't make sense since NodeAttributeManagerImpl doesn't use NodeId. We can take up this point once we support in NodeAttributeManager.

          We need to invoke generateByNewInstance for all the new PB's in setup. can you please check.

          PB's are added as part of testGetNodesToAttributesRequestPBImpl and testGetNodesToAttributesResponsePBImpl

          GetNodesToAttributesRequestPBImpl

          Handled.

          NodeAttributesManagerImpl
          Here we do not have a mapping we are setting a hostname with empty set, is that better or just pass for the ones which have attributes is better?

          Avoiding for which attribute mapping are not set.

          TestClientRMService

          Separated testcase for each.

          Handled javadoc and checkstyle issues too.

          bibinchundatt Bibin Chundatt added a comment - Thank you Naganarasimha for review Can you inform why NodeToAttributesProto is moved from yarn_server_resourcemanager_service_protos.proto to yarn_protos.proto GetNodesToAttributesResponseProto yarn_service.proto requires the same yarn_protos. Also would it make sense to provide overloaded method(getNodesToAttributes) here supporting for NodeID ? Currently IMHO don't make sense since NodeAttributeManagerImpl doesn't use NodeId. We can take up this point once we support in NodeAttributeManager. We need to invoke generateByNewInstance for all the new PB's in setup. can you please check. PB's are added as part of testGetNodesToAttributesRequestPBImpl and testGetNodesToAttributesResponsePBImpl GetNodesToAttributesRequestPBImpl Handled. NodeAttributesManagerImpl Here we do not have a mapping we are setting a hostname with empty set, is that better or just pass for the ones which have attributes is better? Avoiding for which attribute mapping are not set. TestClientRMService Separated testcase for each. Handled javadoc and checkstyle issues too.
          genericqa genericqa added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 40s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                YARN-3409 Compile Tests
          0 mvndep 1m 21s Maven dependency ordering for branch
          +1 mvninstall 25m 19s YARN-3409 passed
          +1 compile 29m 20s YARN-3409 passed
          +1 checkstyle 3m 14s YARN-3409 passed
          +1 mvnsite 5m 2s YARN-3409 passed
          +1 shadedclient 19m 14s branch has no errors when building and testing our client artifacts.
          -1 findbugs 1m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-3409 has 1 extant Findbugs warnings.
          +1 javadoc 3m 44s YARN-3409 passed
                Patch Compile Tests
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 4m 2s the patch passed
          +1 compile 38m 55s the patch passed
          +1 cc 38m 55s the patch passed
          +1 javac 38m 55s the patch passed
          -0 checkstyle 3m 43s root: The patch generated 5 new + 261 unchanged - 0 fixed = 266 total (was 261)
          +1 mvnsite 6m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 12m 5s patch has no errors when building and testing our client artifacts.
          +1 findbugs 9m 48s the patch passed
          +1 javadoc 5m 0s the patch passed
                Other Tests
          +1 unit 0m 56s hadoop-yarn-api in the patch passed.
          +1 unit 3m 32s hadoop-yarn-common in the patch passed.
          +1 unit 2m 33s hadoop-yarn-server-common in the patch passed.
          +1 unit 71m 26s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 unit 26m 58s hadoop-yarn-client in the patch passed.
          +1 unit 1m 29s hadoop-yarn-server-router in the patch passed.
          -1 unit 127m 10s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 46s The patch does not generate ASF License warnings.
          405m 34s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b
          JIRA Issue YARN-8104
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12918512/YARN-8104-YARN-3409.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
          uname Linux 29a625d59cb3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision YARN-3409 / efcca85
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20298/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20298/artifact/out/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/20298/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20298/testReport/
          Max. process+thread count 1195 (vs. ulimit of 10000)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/20298/console
          Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 40s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       YARN-3409 Compile Tests 0 mvndep 1m 21s Maven dependency ordering for branch +1 mvninstall 25m 19s YARN-3409 passed +1 compile 29m 20s YARN-3409 passed +1 checkstyle 3m 14s YARN-3409 passed +1 mvnsite 5m 2s YARN-3409 passed +1 shadedclient 19m 14s branch has no errors when building and testing our client artifacts. -1 findbugs 1m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-3409 has 1 extant Findbugs warnings. +1 javadoc 3m 44s YARN-3409 passed       Patch Compile Tests 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 4m 2s the patch passed +1 compile 38m 55s the patch passed +1 cc 38m 55s the patch passed +1 javac 38m 55s the patch passed -0 checkstyle 3m 43s root: The patch generated 5 new + 261 unchanged - 0 fixed = 266 total (was 261) +1 mvnsite 6m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 12m 5s patch has no errors when building and testing our client artifacts. +1 findbugs 9m 48s the patch passed +1 javadoc 5m 0s the patch passed       Other Tests +1 unit 0m 56s hadoop-yarn-api in the patch passed. +1 unit 3m 32s hadoop-yarn-common in the patch passed. +1 unit 2m 33s hadoop-yarn-server-common in the patch passed. +1 unit 71m 26s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 26m 58s hadoop-yarn-client in the patch passed. +1 unit 1m 29s hadoop-yarn-server-router in the patch passed. -1 unit 127m 10s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 46s The patch does not generate ASF License warnings. 405m 34s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b JIRA Issue YARN-8104 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12918512/YARN-8104-YARN-3409.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc uname Linux 29a625d59cb3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision YARN-3409 / efcca85 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20298/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20298/artifact/out/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/20298/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20298/testReport/ Max. process+thread count 1195 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20298/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          bibinchundatt Bibin Chundatt added a comment -

          Attached patch to handle checkstyle issues too.

          bibinchundatt Bibin Chundatt added a comment - Attached patch to handle checkstyle issues too.
          genericqa genericqa added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 41s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                YARN-3409 Compile Tests
          0 mvndep 1m 29s Maven dependency ordering for branch
          +1 mvninstall 27m 4s YARN-3409 passed
          +1 compile 28m 49s YARN-3409 passed
          +1 checkstyle 3m 14s YARN-3409 passed
          +1 mvnsite 4m 57s YARN-3409 passed
          +1 shadedclient 19m 7s branch has no errors when building and testing our client artifacts.
          -1 findbugs 1m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-3409 has 1 extant Findbugs warnings.
          +1 javadoc 3m 45s YARN-3409 passed
                Patch Compile Tests
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 3m 41s the patch passed
          +1 compile 43m 32s the patch passed
          +1 cc 43m 32s the patch passed
          +1 javac 43m 32s the patch passed
          +1 checkstyle 6m 7s the patch passed
          +1 mvnsite 9m 49s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 10m 31s patch has no errors when building and testing our client artifacts.
          +1 findbugs 7m 53s the patch passed
          +1 javadoc 3m 46s the patch passed
                Other Tests
          +1 unit 0m 47s hadoop-yarn-api in the patch passed.
          +1 unit 3m 11s hadoop-yarn-common in the patch passed.
          +1 unit 2m 12s hadoop-yarn-server-common in the patch passed.
          +1 unit 66m 55s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 unit 27m 1s hadoop-yarn-client in the patch passed.
          +1 unit 1m 29s hadoop-yarn-server-router in the patch passed.
          -1 unit 124m 56s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 51s The patch does not generate ASF License warnings.
          404m 8s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b
          JIRA Issue YARN-8104
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12918587/YARN-8104-YARN-3409.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
          uname Linux 61ebc4154df9 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision YARN-3409 / efcca85
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20305/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/20305/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20305/testReport/
          Max. process+thread count 1205 (vs. ulimit of 10000)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/20305/console
          Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 41s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       YARN-3409 Compile Tests 0 mvndep 1m 29s Maven dependency ordering for branch +1 mvninstall 27m 4s YARN-3409 passed +1 compile 28m 49s YARN-3409 passed +1 checkstyle 3m 14s YARN-3409 passed +1 mvnsite 4m 57s YARN-3409 passed +1 shadedclient 19m 7s branch has no errors when building and testing our client artifacts. -1 findbugs 1m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-3409 has 1 extant Findbugs warnings. +1 javadoc 3m 45s YARN-3409 passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 3m 41s the patch passed +1 compile 43m 32s the patch passed +1 cc 43m 32s the patch passed +1 javac 43m 32s the patch passed +1 checkstyle 6m 7s the patch passed +1 mvnsite 9m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 10m 31s patch has no errors when building and testing our client artifacts. +1 findbugs 7m 53s the patch passed +1 javadoc 3m 46s the patch passed       Other Tests +1 unit 0m 47s hadoop-yarn-api in the patch passed. +1 unit 3m 11s hadoop-yarn-common in the patch passed. +1 unit 2m 12s hadoop-yarn-server-common in the patch passed. +1 unit 66m 55s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 27m 1s hadoop-yarn-client in the patch passed. +1 unit 1m 29s hadoop-yarn-server-router in the patch passed. -1 unit 124m 56s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 51s The patch does not generate ASF License warnings. 404m 8s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b JIRA Issue YARN-8104 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12918587/YARN-8104-YARN-3409.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc uname Linux 61ebc4154df9 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision YARN-3409 / efcca85 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20305/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/20305/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20305/testReport/ Max. process+thread count 1205 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20305/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

          Thanks for the patches bibinchundatt

          Currently IMHO don't make sense since NodeAttributeManagerImpl doesn't use NodeId. We can take up this point once we support in NodeAttributeManager.

          I never meant the comment should change the API in NodeAttributeManager neither are there any plans in that direction. Till now the apis exposed are either internal or admin related. But for user whether it would make sense to have API for NodeId is the question here.

           Also was wondering what would be the behavior if ip's are used instead of hostnames while mapping?

           GetNodesToAttributesResponseProto yarn_service.proto requires the same yarn_protos.

          agree

           

          hadoop-mapreduce-client-jobclient unit tests either failed or timed out can you have a local run and share the results ?

           

          Apart from that patch LGTM !

           

          Naganarasimha Naganarasimha G R added a comment - Thanks for the patches  bibinchundatt ,  Currently IMHO don't make sense since NodeAttributeManagerImpl doesn't use NodeId. We can take up this point once we support in NodeAttributeManager. I never meant the comment should change the API in NodeAttributeManager neither are there any plans in that direction. Till now the apis exposed are either internal or admin related. But for user whether it would make sense to have API for NodeId is the question here.  Also was wondering what would be the behavior if ip's are used instead of hostnames while mapping?   GetNodesToAttributesResponseProto yarn_service.proto requires the same yarn_protos. agree   hadoop-mapreduce-client-jobclient unit tests either failed or timed out can you have a local run and share the results ?   Apart from that patch LGTM !  
          bibinchundatt Bibin Chundatt added a comment -

          Naganarasimha

          Got confused with NodeAttributesManagerImpl#getNodeToAttributes. IIUC you are focusing on YarnClient#getNodeToAttributes.

          It does make sense from user's perspective since all other api to him are NodeID. But if user adds two nodemanagers in same machine.
          (How valid is the deployment scenarios i am also not sure.) this API might create additional confusion rt ??
          In distributed mode one Nodemanager can change attributes of second NM on same node?

          IMHO need further discussion on this point. Thoughts on having follow up jira after discussion for Node ID related clarifications ?.

          Regarding test timeout

          org.apache.hadoop.mapred.pipes.TestPipeApplication is getting timeout in trunk too .

          bibinchundatt Bibin Chundatt added a comment - Naganarasimha Got confused with NodeAttributesManagerImpl#getNodeToAttributes . IIUC you are focusing on YarnClient#getNodeToAttributes . It does make sense from user's perspective since all other api to him are NodeID. But if user adds two nodemanagers in same machine. (How valid is the deployment scenarios i am also not sure.) this API might create additional confusion rt ?? In distributed mode one Nodemanager can change attributes of second NM on same node? IMHO need further discussion on this point. Thoughts on having follow up jira after discussion for Node ID related clarifications ?. Regarding test timeout org.apache.hadoop.mapred.pipes.TestPipeApplication is getting timeout in trunk too .

          IMHO need further discussion on this point. Thoughts on having follow up jira after discussion for Node ID related clarifications ?.

          It was discussed in detail and then only we came down to this decision, in none of enterprise clusters we have two NM's in the same node so it's only a testing scenario. Hence we have knowingly removed it. So i do not see any need to handle for a test scenario as of now unless we have good scenario we need not address it.

          Naganarasimha Naganarasimha G R added a comment - IMHO need further discussion on this point. Thoughts on having follow up jira after discussion for Node ID related clarifications ?. It was discussed in detail and then only we came down to this decision, in none of enterprise clusters we have two NM's in the same node so it's only a testing scenario. Hence we have knowingly removed it. So i do not see any need to handle for a test scenario as of now unless we have good scenario we need not address it.
          bibinchundatt Bibin Chundatt added a comment -

          Thank you Naganarasimha for sharing.. Hope no further updates are required.

          bibinchundatt Bibin Chundatt added a comment - Thank you Naganarasimha for sharing.. Hope no further updates are required.

          Yes mostly fine with me but would like sunil.govind@gmail.com to review once before i commit. Also for our offline discussion on hostname, IIUC its not necessary that nodes needs to be configured with hostnames, ip's can also be configured so in that sense would the api be good with hostname as the variable ? sunil.govind@gmail.com correct me if i am wrong.

          Naganarasimha Naganarasimha G R added a comment - Yes mostly fine with me but would like sunil.govind@gmail.com to review once before i commit. Also for our offline discussion on hostname, IIUC its not necessary that nodes needs to be configured with hostnames, ip's can also be configured so in that sense would the api be good with hostname as the variable ? sunil.govind@gmail.com correct me if i am wrong.
          bibinchundatt Bibin Chundatt added a comment -

          Naganarasimha

          IIUC ContainerManagerImpl#serviceStart registeration will happen always with hostname.

              NodeId nodeId = buildNodeId(connectAddress, hostOverride);
              ((NodeManager.NMContext)context).setNodeId(nodeId);
          
          
            private NodeId buildNodeId(InetSocketAddress connectAddress,
                String hostOverride) {
              if (hostOverride != null) {
                connectAddress = NetUtils.getConnectAddress(
                    new InetSocketAddress(hostOverride, connectAddress.getPort()));
              }
              return NodeId.newInstance(
                  connectAddress.getAddress().getCanonicalHostName(),
                  connectAddress.getPort());
            }
          
          bibinchundatt Bibin Chundatt added a comment - Naganarasimha IIUC ContainerManagerImpl#serviceStart registeration will happen always with hostname. NodeId nodeId = buildNodeId(connectAddress, hostOverride); ((NodeManager.NMContext)context).setNodeId(nodeId); private NodeId buildNodeId(InetSocketAddress connectAddress, String hostOverride) { if (hostOverride != null ) { connectAddress = NetUtils.getConnectAddress( new InetSocketAddress(hostOverride, connectAddress.getPort())); } return NodeId.newInstance( connectAddress.getAddress().getCanonicalHostName(), connectAddress.getPort()); }
          bibinchundatt Bibin Chundatt added a comment -

          As discussed offline updating patch with minor variable name change in proto

          bibinchundatt Bibin Chundatt added a comment - As discussed offline updating patch with minor variable name change in proto
          genericqa genericqa added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 49s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                YARN-3409 Compile Tests
          0 mvndep 2m 5s Maven dependency ordering for branch
          +1 mvninstall 23m 53s YARN-3409 passed
          +1 compile 27m 14s YARN-3409 passed
          +1 checkstyle 2m 39s YARN-3409 passed
          +1 mvnsite 4m 26s YARN-3409 passed
          +1 shadedclient 16m 37s branch has no errors when building and testing our client artifacts.
          -1 findbugs 1m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-3409 has 1 extant Findbugs warnings.
          +1 javadoc 3m 38s YARN-3409 passed
                Patch Compile Tests
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 3m 30s the patch passed
          +1 compile 25m 37s the patch passed
          +1 cc 25m 37s the patch passed
          +1 javac 25m 37s the patch passed
          +1 checkstyle 2m 43s the patch passed
          +1 mvnsite 4m 47s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 9m 25s patch has no errors when building and testing our client artifacts.
          +1 findbugs 7m 40s the patch passed
          +1 javadoc 3m 45s the patch passed
                Other Tests
          +1 unit 0m 46s hadoop-yarn-api in the patch passed.
          +1 unit 3m 20s hadoop-yarn-common in the patch passed.
          +1 unit 2m 18s hadoop-yarn-server-common in the patch passed.
          +1 unit 67m 30s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 unit 27m 18s hadoop-yarn-client in the patch passed.
          +1 unit 1m 28s hadoop-yarn-server-router in the patch passed.
          -1 unit 129m 21s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 35s The patch does not generate ASF License warnings.
          373m 55s



          Reason Tests
          Failed junit tests hadoop.mapred.TestMRTimelineEventHandling



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b
          JIRA Issue YARN-8104
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12918969/YARN-8104-YARN-3409.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
          uname Linux 1973c71f1dd7 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision YARN-3409 / efcca85
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_162
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20339/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/20339/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20339/testReport/
          Max. process+thread count 1293 (vs. ulimit of 10000)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/20339/console
          Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 49s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       YARN-3409 Compile Tests 0 mvndep 2m 5s Maven dependency ordering for branch +1 mvninstall 23m 53s YARN-3409 passed +1 compile 27m 14s YARN-3409 passed +1 checkstyle 2m 39s YARN-3409 passed +1 mvnsite 4m 26s YARN-3409 passed +1 shadedclient 16m 37s branch has no errors when building and testing our client artifacts. -1 findbugs 1m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-3409 has 1 extant Findbugs warnings. +1 javadoc 3m 38s YARN-3409 passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 3m 30s the patch passed +1 compile 25m 37s the patch passed +1 cc 25m 37s the patch passed +1 javac 25m 37s the patch passed +1 checkstyle 2m 43s the patch passed +1 mvnsite 4m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 9m 25s patch has no errors when building and testing our client artifacts. +1 findbugs 7m 40s the patch passed +1 javadoc 3m 45s the patch passed       Other Tests +1 unit 0m 46s hadoop-yarn-api in the patch passed. +1 unit 3m 20s hadoop-yarn-common in the patch passed. +1 unit 2m 18s hadoop-yarn-server-common in the patch passed. +1 unit 67m 30s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 27m 18s hadoop-yarn-client in the patch passed. +1 unit 1m 28s hadoop-yarn-server-router in the patch passed. -1 unit 129m 21s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 373m 55s Reason Tests Failed junit tests hadoop.mapred.TestMRTimelineEventHandling Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b JIRA Issue YARN-8104 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12918969/YARN-8104-YARN-3409.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc uname Linux 1973c71f1dd7 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision YARN-3409 / efcca85 maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20339/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/20339/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20339/testReport/ Max. process+thread count 1293 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20339/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See https://builds.apache.org/job/Hadoop-trunk-Commit/14931/)
          YARN-8104. Add API to fetch node to attribute mapping. Contributed by (sunilg: rev 5dc7d6e0f3f46277f0615d61755970a7aa28b8d7)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNodesToAttributesRequestPBImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributesManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNodesToAttributesRequest.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAttributesToNodesResponsePBImpl.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNodesToAttributesResponsePBImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ApplicationClientProtocolPBClientImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodesToAttributesMappingRequestPBImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/DefaultClientRequestInterceptor.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/MockResourceManagerFacade.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNodesToAttributesResponse.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/PassThroughClientRequestInterceptor.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeToAttributesPBImpl.java
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See https://builds.apache.org/job/Hadoop-trunk-Commit/14931/ ) YARN-8104 . Add API to fetch node to attribute mapping. Contributed by (sunilg: rev 5dc7d6e0f3f46277f0615d61755970a7aa28b8d7) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNodesToAttributesRequestPBImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributesManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNodesToAttributesRequest.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAttributesToNodesResponsePBImpl.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNodesToAttributesResponsePBImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ApplicationClientProtocolPBClientImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodesToAttributesMappingRequestPBImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/DefaultClientRequestInterceptor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/MockResourceManagerFacade.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNodesToAttributesResponse.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/PassThroughClientRequestInterceptor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeToAttributesPBImpl.java

          People

            bibinchundatt Bibin Chundatt
            bibinchundatt Bibin Chundatt
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: