Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6484

Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: client, security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. This will cause HDFS token renew failure for renewer "nobody" if the rules from hadoop.security.auth_to_local exclude the client address in HDFS DelegationTokenIdentifier.
      The reason why the local address is returned is: When HA is enabled, "yarn.resourcemanager.address" may not be set, if HOSTNAME_PATTERN("_HOST") is used in "yarn.resourcemanager.principal", the default address "0.0.0.0:8032" will be used, Based on the following code at SecurityUtil.java, the local address will be used to replace "0.0.0.0".

        private static String replacePattern(String[] components, String hostname)
            throws IOException {
          String fqdn = hostname;
          if (fqdn == null || fqdn.isEmpty() || fqdn.equals("0.0.0.0")) {
            fqdn = getLocalHostName();
          }
          return components[0] + "/" + fqdn.toLowerCase(Locale.US) + "@" + components[2];
        }
        static String getLocalHostName() throws UnknownHostException {
          return InetAddress.getLocalHost().getCanonicalHostName();
        }
        public static String getServerPrincipal(String principalConfig,
            InetAddress addr) throws IOException {
          String[] components = getComponents(principalConfig);
          if (components == null || components.length != 3
              || !components[1].equals(HOSTNAME_PATTERN)) {
            return principalConfig;
          } else {
            if (addr == null) {
              throw new IOException("Can't replace " + HOSTNAME_PATTERN
                  + " pattern since client address is null");
            }
            return replacePattern(components, addr.getCanonicalHostName());
          }
        }
      

      The following is the exception which cause the job fail:

      15/09/12 16:27:24 WARN security.UserGroupInformation: PriviledgedActionException as:test@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: Failed to run job : yarn tries to renew a token with renewer nobody
      at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:464)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem.java:7109)
      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewDelegationToken(NameNodeRpcServer.java:512)
      at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.renewDelegationToken(AuthorizationProviderProxyClientProtocol.java:648)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:975)
      at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
      java.io.IOException: Failed to run job : yarn tries to renew a token with renewer nobody
      at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:464)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem.java:7109)
      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewDelegationToken(NameNodeRpcServer.java:512)
      at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.renewDelegationToken(AuthorizationProviderProxyClientProtocol.java:648)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:975)
      at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
      at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:300)
      at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:438)
      at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
      at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
      at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
      at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313)
      at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
      at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
      at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      
      1. MAPREDUCE-6484.001.patch
        4 kB
        zhihai xu
      2. YARN-4187.000.patch
        4 kB
        zhihai xu

        Activity

        Hide
        hitesh Hitesh Shah added a comment -

        Arun Suresh thanks for the pointer. Any reason why MR does not use that function in that case?

        Show
        hitesh Hitesh Shah added a comment - Arun Suresh thanks for the pointer. Any reason why MR does not use that function in that case?
        Hide
        asuresh Arun Suresh added a comment - - edited

        I feel this is useful only to MR. Correct me if im wrong, but I believe this is used specifically by the mapreduce TokenCache and JobRunner. I also believe ClientRMProxy handles this correctly when it does a getTokenService . zhihai xu, Thoughts ?

        Show
        asuresh Arun Suresh added a comment - - edited I feel this is useful only to MR. Correct me if im wrong, but I believe this is used specifically by the mapreduce TokenCache and JobRunner. I also believe ClientRMProxy handles this correctly when it does a getTokenService . zhihai xu , Thoughts ?
        Hide
        hitesh Hitesh Shah added a comment -

        Arun Suresh zhihai xu It seems like the getMasterAddress() functionality ideally belongs in YARN and not in MR so that other applications that make use of YARN can always leverage the same functionality. Would you agree?

        Show
        hitesh Hitesh Shah added a comment - Arun Suresh zhihai xu It seems like the getMasterAddress() functionality ideally belongs in YARN and not in MR so that other applications that make use of YARN can always leverage the same functionality. Would you agree?
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #414 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/414/)
        MAPREDUCE-6484. Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #414 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/414/ ) MAPREDUCE-6484 . Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2380 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2380/)
        MAPREDUCE-6484. Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2380 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2380/ ) MAPREDUCE-6484 . Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #1175 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1175/)
        MAPREDUCE-6484. Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1175 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1175/ ) MAPREDUCE-6484 . Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2353 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2353/)
        MAPREDUCE-6484. Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2353 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2353/ ) MAPREDUCE-6484 . Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #442 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/442/)
        MAPREDUCE-6484. Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #442 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/442/ ) MAPREDUCE-6484 . Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8513 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8513/)
        MAPREDUCE-6484. Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8513 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8513/ ) MAPREDUCE-6484 . Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #435 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/435/)
        MAPREDUCE-6484. Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #435 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/435/ ) MAPREDUCE-6484 . Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu (zxu: rev 97a08807ec328a6f455d941fe3d9a469d885caa6) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
        Hide
        zxu zhihai xu added a comment -

        Committed it to branch-2 and trunk.

        Show
        zxu zhihai xu added a comment - Committed it to branch-2 and trunk.
        Hide
        zxu zhihai xu added a comment -

        Thanks for the review Arun Suresh! The new patch passed jenkins. I will commit it tomorrow if no one objects.

        Show
        zxu zhihai xu added a comment - Thanks for the review Arun Suresh ! The new patch passed jenkins. I will commit it tomorrow if no one objects.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 5s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 59s There were no new javac warning messages.
        +1 javadoc 10m 11s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 48s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 32s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 mapreduce tests 1m 47s Tests passed in hadoop-mapreduce-client-core.
            41m 50s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12762046/MAPREDUCE-6484.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 06d1c90
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6011/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6011/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6011/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 59s There were no new javac warning messages. +1 javadoc 10m 11s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 48s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 47s Tests passed in hadoop-mapreduce-client-core.     41m 50s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762046/MAPREDUCE-6484.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 06d1c90 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6011/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6011/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6011/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 22s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 11s There were no new javac warning messages.
        +1 javadoc 10m 20s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 49s The applied patch generated 1 new checkstyle issues (total was 9, now 9).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 mapreduce tests 1m 46s Tests passed in hadoop-mapreduce-client-core.
            42m 23s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12761995/MAPREDUCE-6484.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 1f707ec
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6010/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6010/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6010/testReport/
        Java 1.7.0_55
        uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6010/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 22s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 11s There were no new javac warning messages. +1 javadoc 10m 20s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 49s The applied patch generated 1 new checkstyle issues (total was 9, now 9). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 46s Tests passed in hadoop-mapreduce-client-core.     42m 23s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761995/MAPREDUCE-6484.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1f707ec checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6010/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6010/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6010/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6010/console This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        The latest patch looks good.. thanks zhihai xu

        +1, pending jenkins

        Show
        asuresh Arun Suresh added a comment - The latest patch looks good.. thanks zhihai xu +1, pending jenkins
        Hide
        zxu zhihai xu added a comment -

        thanks for the review Arun Suresh! That is a good suggestion. I attached a new patch MAPREDUCE-6484.001.patch, which addressed your comment.

        Show
        zxu zhihai xu added a comment - thanks for the review Arun Suresh ! That is a good suggestion. I attached a new patch MAPREDUCE-6484 .001.patch, which addressed your comment.
        Hide
        asuresh Arun Suresh added a comment -

        Thanks for the patch, zhihai xu.. Makes sense..

        Minor nit :
        Instead of using yarnConf.getStringCollection() and then doing an rmIds.toArrays(), you can probably just use yarnConf.getStrings() which returns an array itself.

        +1, pending that

        Show
        asuresh Arun Suresh added a comment - Thanks for the patch, zhihai xu .. Makes sense.. Minor nit : Instead of using yarnConf.getStringCollection() and then doing an rmIds.toArrays() , you can probably just use yarnConf.getStrings() which returns an array itself. +1, pending that
        Hide
        zxu zhihai xu added a comment -

        Moved the JIRA from YARN to MapReduce since all the changes are in Map Reduce project.

        Show
        zxu zhihai xu added a comment - Moved the JIRA from YARN to MapReduce since all the changes are in Map Reduce project.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 54s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 9m 45s There were no new javac warning messages.
        +1 javadoc 12m 28s There were no new javadoc warning messages.
        +1 release audit 0m 31s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 4s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 53s mvn install still works.
        +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse.
        +1 findbugs 1m 35s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 mapreduce tests 1m 59s Tests passed in hadoop-mapreduce-client-core.
            49m 52s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12761248/YARN-4187.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 66b46d0
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6006/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6006/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6006/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 9m 45s There were no new javac warning messages. +1 javadoc 12m 28s There were no new javadoc warning messages. +1 release audit 0m 31s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 4s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 53s mvn install still works. +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse. +1 findbugs 1m 35s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 59s Tests passed in hadoop-mapreduce-client-core.     49m 52s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761248/YARN-4187.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 66b46d0 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6006/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6006/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6006/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 6s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 59s There were no new javac warning messages.
        +1 javadoc 10m 15s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 46s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 27s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 mapreduce tests 1m 41s Tests failed in hadoop-mapreduce-client-core.
            41m 38s  



        Reason Tests
        Failed unit tests hadoop.mapreduce.lib.input.TestLineRecordReader
          hadoop.mapred.TestLineRecordReader



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12761237/YARN-4187.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 94dec5a
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6005/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6005/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6005/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 6s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 59s There were no new javac warning messages. +1 javadoc 10m 15s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 46s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 1m 41s Tests failed in hadoop-mapreduce-client-core.     41m 38s   Reason Tests Failed unit tests hadoop.mapreduce.lib.input.TestLineRecordReader   hadoop.mapred.TestLineRecordReader Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761237/YARN-4187.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 94dec5a hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6005/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6005/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6005/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 14s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 4s There were no new javac warning messages.
        +1 javadoc 10m 15s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 50s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 mapreduce tests 1m 48s Tests passed in hadoop-mapreduce-client-core.
            42m 9s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12761237/YARN-4187.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 94dec5a
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6004/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6004/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6004/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 14s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 10m 15s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 48s Tests passed in hadoop-mapreduce-client-core.     42m 9s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761237/YARN-4187.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 94dec5a hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6004/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6004/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6004/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 57s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 9m 37s There were no new javac warning messages.
        +1 javadoc 12m 16s There were no new javadoc warning messages.
        +1 release audit 0m 29s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 55s The applied patch generated 1 new checkstyle issues (total was 9, now 10).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 47s mvn install still works.
        +1 eclipse:eclipse 0m 46s The patch built with eclipse:eclipse.
        +1 findbugs 1m 37s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 mapreduce tests 2m 22s Tests passed in hadoop-mapreduce-client-core.
            48m 50s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12761227/YARN-4187.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 94dec5a
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9216/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-YARN-Build/9216/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9216/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9216/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 9m 37s There were no new javac warning messages. +1 javadoc 12m 16s There were no new javadoc warning messages. +1 release audit 0m 29s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 55s The applied patch generated 1 new checkstyle issues (total was 9, now 10). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 47s mvn install still works. +1 eclipse:eclipse 0m 46s The patch built with eclipse:eclipse. +1 findbugs 1m 37s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 2m 22s Tests passed in hadoop-mapreduce-client-core.     48m 50s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761227/YARN-4187.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 94dec5a checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9216/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-YARN-Build/9216/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9216/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9216/console This message was automatically generated.
        Hide
        zxu zhihai xu added a comment -

        I attached a patch YARN-4187.000.patch which use YarnConfiguration instead of JobConf(Configuration) to call getSocketAddr, because Configuration.getSocketAddr won't support RM HA. Also we need check whether RM_HA_ID is configured, if RM_HA_ID is not configured, the first one in RM_HA_IDS will be used, otherwise it will fall back to use the client local address.

        Show
        zxu zhihai xu added a comment - I attached a patch YARN-4187 .000.patch which use YarnConfiguration instead of JobConf(Configuration) to call getSocketAddr , because Configuration.getSocketAddr won't support RM HA. Also we need check whether RM_HA_ID is configured, if RM_HA_ID is not configured, the first one in RM_HA_IDS will be used, otherwise it will fall back to use the client local address.

          People

          • Assignee:
            zxu zhihai xu
            Reporter:
            zxu zhihai xu
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development