Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4833

For Queue AccessControlException client retries multiple times on both RM

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Submit application to queue where ACL is enabled and submitted user is not having access. Client retries till failMaxattempt 10 times.

      16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking submitApplication of class ApplicationClientProtocolPBClientImpl over rm1. Trying to fail over immediately.
      org.apache.hadoop.security.AccessControlException: User hdfs does not have permission to submit application_1458273884145_0001 to queue default
              at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
              at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
              at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
              at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
              at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
      
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
              at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
              at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
              at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
              at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
              at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
              at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
              at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
              at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
              at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
              at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
              at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
              at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
              at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
              at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
              at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
              at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.hadoop.util.RunJar.run(RunJar.java:222)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): User hdfs does not have permission to submit application_1458273884145_0001 to queue default
              at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
              at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
              at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
              at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
              at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
      
              at org.apache.hadoop.ipc.Client.call(Client.java:1449)
              at org.apache.hadoop.ipc.Client.call(Client.java:1386)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
              at com.sun.proxy.$Proxy22.submitApplication(Unknown Source)
              at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:269)
              ... 35 more
      16/03/18 10:01:06 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
      16/03/18 10:01:06 WARN ipc.Client: Failed to connect to server: host-10-19-92-187/10.19.91.146:45022: retries get failed due to exceeded maximum allowed retries number: 0
      java.net.ConnectException: Connection refused
              at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
              at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
              at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
              at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
              at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
              at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:634)
              at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:733)
              at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378)
              at org.apache.hadoop.ipc.Client.getConnection(Client.java:1510)
              at org.apache.hadoop.ipc.Client.call(Client.java:1425)
              at org.apache.hadoop.ipc.Client.call(Client.java:1386)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
              at com.sun.proxy.$Proxy22.submitApplication(Unknown Source)
              at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:269)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
              at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
              at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
              at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
              at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
              at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
              at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
              at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
              at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
              at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
              at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
              at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
              at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.hadoop.util.RunJar.run(RunJar.java:222)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking submitApplication of class ApplicationClientProtocolPBClientImpl over rm2 after 1 fail over attempts. Trying to fail over after sleeping for 22450ms.
      java.net.ConnectException: Call From host-10-19-92-199/10.19.91.157 to host-10-19-92-187:45022 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
              at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
              at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
              at org.apache.hadoop.ipc.Client.call(Client.java:1453)
              at org.apache.hadoop.ipc.Client.call(Client.java:1386)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
              at com.sun.proxy.$Proxy22.submitApplication(Unknown Source)
              at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:269)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
              at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
              at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
              at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
              at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
              at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
              at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
              at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
              at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
              at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
              at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
              at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
              at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.hadoop.util.RunJar.run(RunJar.java:222)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      Caused by: java.net.ConnectException: Connection refused
              at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
              at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
              at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
              at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
              at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
              at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:634)
              at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:733)
              at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378)
              at org.apache.hadoop.ipc.Client.getConnection(Client.java:1510)
              at org.apache.hadoop.ipc.Client.call(Client.java:1425)
              ... 39 more
      
      

      Solution

      1. As per discussion with Jian He should handle the AccessControlException in RetryPolicy and should go to fallbackPolicy
      2. Wrap AccessControl exception to YarnException in RMAppManager#submitApplication
        Thoughts?
      1. YARN-4833-branch-2.0001.patch
        6 kB
        Bibin A Chundatt
      2. YARN-4833.0005.patch
        6 kB
        Bibin A Chundatt
      3. YARN-4833.0004.patch
        7 kB
        Bibin A Chundatt
      4. YARN-4833.0003.patch
        6 kB
        Bibin A Chundatt
      5. YARN-4833.0002.patch
        2 kB
        Bibin A Chundatt
      6. YARN-4833.0001.patch
        2 kB
        Bibin A Chundatt
      7. 0001-YARN-4833.patch
        2 kB
        Bibin A Chundatt

        Issue Links

          Activity

          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Committed to branch-2 too
          Thanks Bibin A Chundatt

          Show
          Naganarasimha Naganarasimha G R added a comment - Committed to branch-2 too Thanks Bibin A Chundatt
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Naganarasimha G R
          Test failures are not related patch attached.. Will check any other jira is raised related to the TC failure for branch-2.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Naganarasimha G R Test failures are not related patch attached.. Will check any other jira is raised related to the TC failure for branch-2.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 34s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 48s branch-2 passed
          +1 compile 0m 27s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 32s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 0m 23s branch-2 passed
          +1 mvnsite 0m 40s branch-2 passed
          +1 mvneclipse 0m 18s branch-2 passed
          +1 findbugs 1m 11s branch-2 passed
          +1 javadoc 0m 19s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 0m 24s branch-2 passed with JDK v1.7.0_101
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.8.0_101
          +1 javac 0m 26s the patch passed
          +1 compile 0m 30s the patch passed with JDK v1.7.0_101
          +1 javac 0m 30s the patch passed
          +1 checkstyle 0m 20s the patch passed
          +1 mvnsite 0m 35s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 18s the patch passed
          +1 javadoc 0m 18s the patch passed with JDK v1.8.0_101
          +1 javadoc 0m 23s the patch passed with JDK v1.7.0_101
          +1 unit 33m 7s hadoop-yarn-server-resourcemanager in the patch passed with JDK v1.8.0_101.
          -1 unit 34m 23s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_101.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          98m 5s



          Reason Tests
          JDK v1.7.0_101 Failed junit tests hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823404/YARN-4833-branch-2.0001.patch
          JIRA Issue YARN-4833
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8984c7fec256 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 6d53e09
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12753/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12753/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12753/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12753/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 34s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 48s branch-2 passed +1 compile 0m 27s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 32s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 23s branch-2 passed +1 mvnsite 0m 40s branch-2 passed +1 mvneclipse 0m 18s branch-2 passed +1 findbugs 1m 11s branch-2 passed +1 javadoc 0m 19s branch-2 passed with JDK v1.8.0_101 +1 javadoc 0m 24s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 30s the patch passed +1 compile 0m 26s the patch passed with JDK v1.8.0_101 +1 javac 0m 26s the patch passed +1 compile 0m 30s the patch passed with JDK v1.7.0_101 +1 javac 0m 30s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 18s the patch passed with JDK v1.8.0_101 +1 javadoc 0m 23s the patch passed with JDK v1.7.0_101 +1 unit 33m 7s hadoop-yarn-server-resourcemanager in the patch passed with JDK v1.8.0_101. -1 unit 34m 23s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 98m 5s Reason Tests JDK v1.7.0_101 Failed junit tests hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823404/YARN-4833-branch-2.0001.patch JIRA Issue YARN-4833 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8984c7fec256 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 6d53e09 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/12753/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12753/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12753/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12753/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching branch-2 patch

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching branch-2 patch
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          NGarla_Unused
          Thank you for commit. Will upload the patch soon for branch 2

          Show
          bibinchundatt Bibin A Chundatt added a comment - NGarla_Unused Thank you for commit. Will upload the patch soon for branch 2
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10265 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10265/)
          YARN-4833. For Queue AccessControlException client retries multiple (naganarasimha_gr: rev 874577a67df8a49243586909d866034df4e3b276)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10265 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10265/ ) YARN-4833 . For Queue AccessControlException client retries multiple (naganarasimha_gr: rev 874577a67df8a49243586909d866034df4e3b276) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          Hide
          Naganarasimha Naganarasimha G R added a comment - - edited

          Bibin A Chundatt, Committed to trunk but seems to fail for branch-2, can you provide a patch for it ?

          Show
          Naganarasimha Naganarasimha G R added a comment - - edited Bibin A Chundatt , Committed to trunk but seems to fail for branch-2, can you provide a patch for it ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          LGTM, Committing the patch !

          Show
          Naganarasimha Naganarasimha G R added a comment - LGTM, Committing the patch !
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 56s trunk passed
          +1 compile 0m 33s trunk passed
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 0m 43s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 1m 8s trunk passed
          +1 javadoc 0m 22s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 0m 37s the patch passed
          +1 javac 0m 37s the patch passed
          +1 checkstyle 0m 21s the patch passed
          +1 mvnsite 0m 42s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 15s the patch passed
          +1 javadoc 0m 22s the patch passed
          +1 unit 34m 20s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          51m 6s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823300/YARN-4833.0005.patch
          JIRA Issue YARN-4833
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 80119874edb6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8fbb57f
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12746/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12746/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 56s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 43s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 8s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 0m 37s the patch passed +1 javac 0m 37s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 42s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 15s the patch passed +1 javadoc 0m 22s the patch passed +1 unit 34m 20s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 51m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823300/YARN-4833.0005.patch JIRA Issue YARN-4833 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 80119874edb6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8fbb57f Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12746/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12746/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Updating patch after removing unused import.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Updating patch after removing unused import.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G
          Looked into the testcase error to me it looks not related to patch attached

          https://builds.apache.org/job/PreCommit-YARN-Build/12726/testReport/org.apache.hadoop.yarn.server.resourcemanager/TestSubmitApplicationWithRMHA/testHandleRMHADuringSubmitApplicationCallWithoutSavedApplicationState/

          2016-08-10 19:42:11,627 ERROR [Curator-Framework-0] curator.ConnectionState (ConnectionState.java:checkTimeouts(200)) - Connection timed out for connection string (127.0.0.1:40253) and timeout (15000) / elapsed (5036)
          org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
          	at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197)
          	at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87)
          	at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
          	at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:806)
          	at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792)
          	at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
          	at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:745)
          
          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G Looked into the testcase error to me it looks not related to patch attached https://builds.apache.org/job/PreCommit-YARN-Build/12726/testReport/org.apache.hadoop.yarn.server.resourcemanager/TestSubmitApplicationWithRMHA/testHandleRMHADuringSubmitApplicationCallWithoutSavedApplicationState/ 2016-08-10 19:42:11,627 ERROR [Curator-Framework-0] curator.ConnectionState (ConnectionState.java:checkTimeouts(200)) - Connection timed out for connection string (127.0.0.1:40253) and timeout (15000) / elapsed (5036) org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87) at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:806) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          thanks for the input Jian He and as i think there is already jira raised for test failure TestSubmitApplicationWithRMHA timeout.
          But just one nit Bibin A Chundatt please correct the checkstyle error (unused import)

          Show
          Naganarasimha Naganarasimha G R added a comment - thanks for the input Jian He and as i think there is already jira raised for test failure TestSubmitApplicationWithRMHA timeout . But just one nit Bibin A Chundatt please correct the checkstyle error (unused import)
          Hide
          sunilg Sunil G added a comment -

          +1 for the latest patch. Thanks Bibin A Chundatt. Test case seems not related. Could you please confirm that once.

          Show
          sunilg Sunil G added a comment - +1 for the latest patch. Thanks Bibin A Chundatt . Test case seems not related. Could you please confirm that once.
          Hide
          jianhe Jian He added a comment -

          lgtm too, thanks

          Show
          jianhe Jian He added a comment - lgtm too, thanks
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 24s trunk passed
          +1 compile 0m 33s trunk passed
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 38s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 55s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 31s the patch passed
          +1 compile 0m 29s the patch passed
          +1 javac 0m 29s the patch passed
          -1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 97 unchanged - 0 fixed = 98 total (was 97)
          +1 mvnsite 0m 34s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 2s the patch passed
          +1 javadoc 0m 18s the patch passed
          -1 unit 37m 22s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          52m 31s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823117/YARN-4833.0004.patch
          JIRA Issue YARN-4833
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux dc75f85b47b5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 89c0bff
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12726/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12726/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12726/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12726/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12726/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 24s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 55s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed -1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 97 unchanged - 0 fixed = 98 total (was 97) +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 2s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 37m 22s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 52m 31s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823117/YARN-4833.0004.patch JIRA Issue YARN-4833 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dc75f85b47b5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89c0bff Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12726/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12726/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12726/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12726/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12726/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G/Naganarasimha G R
          Thank you for review .. Attaching patch after handling test related comment

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G / Naganarasimha G R Thank you for review .. Attaching patch after handling test related comment
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Bibin A Chundatt, latest patch is fine and test case also is fine except for the nit pointed out by Sunil G.
          Jian He, any comments on the approach ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Bibin A Chundatt , latest patch is fine and test case also is fine except for the nit pointed out by Sunil G . Jian He , any comments on the approach ?
          Hide
          sunilg Sunil G added a comment -

          I think current approach in latest patch is better and looks fine. Also since AccessControlException is removed from throwable list, client need not have to worry for such exceptions.
          Bibin A Chundatt, One nit in test case. Could you pls unwrap and verify if its an AccessControlException and the message too. This can help to confirm the scenario.

          Show
          sunilg Sunil G added a comment - I think current approach in latest patch is better and looks fine. Also since AccessControlException is removed from throwable list, client need not have to worry for such exceptions. Bibin A Chundatt , One nit in test case. Could you pls unwrap and verify if its an AccessControlException and the message too. This can help to confirm the scenario.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Testcase failure is not related to patch attached.Tried running failed testcase after applying patch locally and is passing fine.
          YARN-5491 raised to handle random failure

          Show
          bibinchundatt Bibin A Chundatt added a comment - Testcase failure is not related to patch attached.Tried running failed testcase after applying patch locally and is passing fine. YARN-5491 raised to handle random failure
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 12s trunk passed
          +1 compile 0m 38s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 43s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 1m 7s trunk passed
          +1 javadoc 0m 25s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 0m 36s the patch passed
          +1 javac 0m 36s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 0m 44s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 2s the patch passed
          +1 javadoc 0m 19s the patch passed
          -1 unit 38m 6s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          54m 59s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822829/YARN-4833.0003.patch
          JIRA Issue YARN-4833
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 49aa228ddbc7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4aba858
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12694/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12694/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12694/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12694/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 12s trunk passed +1 compile 0m 38s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 43s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 7s trunk passed +1 javadoc 0m 25s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 0m 36s the patch passed +1 javac 0m 36s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 44s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 2s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 38m 6s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 54m 59s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822829/YARN-4833.0003.patch JIRA Issue YARN-4833 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 49aa228ddbc7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4aba858 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/12694/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12694/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12694/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12694/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching patch with testcase

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching patch with testcase
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          java.security.AccessControlException is used in ClientRMService. Should we change the same to org.apache.hadoop.security.AccessControlException.??

          Show
          bibinchundatt Bibin A Chundatt added a comment - java.security.AccessControlException is used in ClientRMService. Should we change the same to org.apache.hadoop.security.AccessControlException .??
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Jian He,Sunil G & Varun Saxena,
          Initial reason to avoid the approach in the Patch 1 was to have the proper error message in the REST call when Application is submitted in the above scenario. But Seems like in RMWebServices handles this properly by throwing

           catch (UndeclaredThrowableException ue) {
                if (ue.getCause() instanceof YarnException) {
                  throw new BadRequestException(ue.getCause().getMessage());
                }
                LOG.info("Submit app request failed", ue);
                throw ue;
              }
          

          Had offline discussion with Bibin A Chundatt, and based on our discussion what we felt is in YARN-4522 adds

             /**
          @@ -276,7 +281,7 @@ protected synchronized void checkAppNumCompletedLimit() {
             @SuppressWarnings("unchecked")
             protected void submitApplication(
                 ApplicationSubmissionContext submissionContext, long submitTime,
          -      String user) throws YarnException {
          +      String user) throws YarnException, AccessControlException {
               ApplicationId applicationId = submissionContext.getApplicationId();
           
               RMAppImpl application =
          @@ -325,7 +330,8 @@ protected void recoverApplication(ApplicationStateData appState,
           
             private RMAppImpl createAndPopulateNewRMApp(
                 ApplicationSubmissionContext submissionContext, long submitTime,
          -      String user, boolean isRecovery) throws YarnException {
          +      String user, boolean isRecovery)
          +      throws YarnException, AccessControlException 
          

          IMHO We could have kept the interface intact and throw the wrapped exception directly as per the approach 1 . Thoughts ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Jian He , Sunil G & Varun Saxena , Initial reason to avoid the approach in the Patch 1 was to have the proper error message in the REST call when Application is submitted in the above scenario. But Seems like in RMWebServices handles this properly by throwing catch (UndeclaredThrowableException ue) { if (ue.getCause() instanceof YarnException) { throw new BadRequestException(ue.getCause().getMessage()); } LOG.info( "Submit app request failed" , ue); throw ue; } Had offline discussion with Bibin A Chundatt , and based on our discussion what we felt is in YARN-4522 adds /** @@ -276,7 +281,7 @@ protected synchronized void checkAppNumCompletedLimit() { @SuppressWarnings( "unchecked" ) protected void submitApplication( ApplicationSubmissionContext submissionContext, long submitTime, - String user) throws YarnException { + String user) throws YarnException, AccessControlException { ApplicationId applicationId = submissionContext.getApplicationId(); RMAppImpl application = @@ -325,7 +330,8 @@ protected void recoverApplication(ApplicationStateData appState, private RMAppImpl createAndPopulateNewRMApp( ApplicationSubmissionContext submissionContext, long submitTime, - String user, boolean isRecovery) throws YarnException { + String user, boolean isRecovery) + throws YarnException, AccessControlException IMHO We could have kept the interface intact and throw the wrapped exception directly as per the approach 1 . Thoughts ?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Naganarasimha G R/Sunil G

          Another issue in ClientRMService is for AccessControlException in below code is of type import java.security.AccessControlException

          {    if (!checkAccess(callerUGI, application.getUser(),
                  ApplicationAccessType.MODIFY_APP, application)) {
                RMAuditLogger.logFailure(callerUGI.getShortUserName(),
                    AuditConstants.UPDATE_APP_PRIORITY,
                    "User doesn't have permissions to "
                        + ApplicationAccessType.MODIFY_APP.toString(), "ClientRMService",
                    AuditConstants.UNAUTHORIZED_USER, applicationId);
                throw RPCUtil.getRemoteException(new AccessControlException("User "
                    + callerUGI.getShortUserName() + " cannot perform operation "
                    + ApplicationAccessType.MODIFY_APP.name() + " on " + applicationId));
              }}
          

          Will update in YARN-4833.0001.patch . In ClientRMService all other location we are handling AccessControlException after wrapping in RemoteException.
          No need to handle in RetryPolicies#shouldRetry and behaviour will be same as earlier

          Show
          bibinchundatt Bibin A Chundatt added a comment - Naganarasimha G R / Sunil G Another issue in ClientRMService is for AccessControlException in below code is of type import java.security.AccessControlException { if (!checkAccess(callerUGI, application.getUser(), ApplicationAccessType.MODIFY_APP, application)) { RMAuditLogger.logFailure(callerUGI.getShortUserName(), AuditConstants.UPDATE_APP_PRIORITY, "User doesn't have permissions to " + ApplicationAccessType.MODIFY_APP.toString(), "ClientRMService" , AuditConstants.UNAUTHORIZED_USER, applicationId); throw RPCUtil.getRemoteException( new AccessControlException( "User " + callerUGI.getShortUserName() + " cannot perform operation " + ApplicationAccessType.MODIFY_APP.name() + " on " + applicationId)); }} Will update in YARN-4833 .0001.patch . In ClientRMService all other location we are handling AccessControlException after wrapping in RemoteException . No need to handle in RetryPolicies#shouldRetry and behaviour will be same as earlier
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Bob.zhao
          Thank you for looking into the patch .. It is YarnException

          public class RPCUtil {
          
            /**
             * Returns an instance of {@link YarnException}
             */
            public static YarnException getRemoteException(Throwable t) {
              return new YarnException(t);
            }
          
            /**
          
          
          Show
          bibinchundatt Bibin A Chundatt added a comment - Bob.zhao Thank you for looking into the patch .. It is YarnException public class RPCUtil { /** * Returns an instance of {@link YarnException} */ public static YarnException getRemoteException(Throwable t) { return new YarnException(t); } /**
          Hide
          varun_saxena Varun Saxena added a comment - - edited

          True. Actually we can handle it the 1st way (amongst the 2 approaches) by adding it to some exception map and then handling it while retrying on client side. That is we adopt approach 1.
          It was just a followup comment to Naga's comment.

          Your patch is also fine.

          Show
          varun_saxena Varun Saxena added a comment - - edited True. Actually we can handle it the 1st way (amongst the 2 approaches) by adding it to some exception map and then handling it while retrying on client side. That is we adopt approach 1. It was just a followup comment to Naga's comment. Your patch is also fine.
          Hide
          Jobo Bob.zhao added a comment -

          Hi, Bibin, thanks for you patch!
          Here one thing want to know, why not directly throw YarnException directly but warp with RemoteException?

          Show
          Jobo Bob.zhao added a comment - Hi, Bibin, thanks for you patch! Here one thing want to know, why not directly throw YarnException directly but warp with RemoteException?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 49s trunk passed
          +1 compile 0m 33s trunk passed
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 42s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 57s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 31s the patch passed
          +1 javac 0m 31s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvnsite 0m 35s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 0s the patch passed
          +1 javadoc 0m 19s the patch passed
          +1 unit 37m 37s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          53m 9s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822650/YARN-4833.0002.patch
          JIRA Issue YARN-4833
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 088c1affb815 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 58e1523
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12678/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12678/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 49s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 42s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 0s the patch passed +1 javadoc 0m 19s the patch passed +1 unit 37m 37s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 53m 9s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822650/YARN-4833.0002.patch JIRA Issue YARN-4833 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 088c1affb815 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 58e1523 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12678/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12678/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching patch handling the exception in ClientRMService .
          Tested the same manually its working fine

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching patch handling the exception in ClientRMService . Tested the same manually its working fine
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          AccessControlException in RetryPolicy and should go to fallbackPolicy

          Want to keep the behaviour similar to earlier.

          Show
          bibinchundatt Bibin A Chundatt added a comment - AccessControlException in RetryPolicy and should go to fallbackPolicy Want to keep the behaviour similar to earlier.
          Hide
          bibinchundatt Bibin A Chundatt added a comment - - edited

          Varun Saxena
          For HA the flow will be different.

              // Handle HA case first
              if (isHAEnabled) {
          ...
                return RetryPolicies.failoverOnNetworkException(
                    RetryPolicies.TRY_ONCE_THEN_FAIL, maxFailoverAttempts,
                    failoverSleepBaseMs, failoverSleepMaxMs);
          }
          
          Show
          bibinchundatt Bibin A Chundatt added a comment - - edited Varun Saxena For HA the flow will be different. // Handle HA case first if (isHAEnabled) { ... return RetryPolicies.failoverOnNetworkException( RetryPolicies.TRY_ONCE_THEN_FAIL, maxFailoverAttempts, failoverSleepBaseMs, failoverSleepMaxMs); }
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 9m 20s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 48s trunk passed
          +1 mvneclipse 0m 21s trunk passed
          +1 findbugs 0m 58s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 30s the patch passed
          +1 javac 0m 30s the patch passed
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 0m 36s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 5s the patch passed
          +1 javadoc 0m 17s the patch passed
          +1 unit 34m 23s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          52m 11s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822608/YARN-4833.0001.patch
          JIRA Issue YARN-4833
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fa78c9709539 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 37d939a
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12672/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12672/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 20s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 48s trunk passed +1 mvneclipse 0m 21s trunk passed +1 findbugs 0m 58s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 5s the patch passed +1 javadoc 0m 17s the patch passed +1 unit 34m 23s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 52m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822608/YARN-4833.0001.patch JIRA Issue YARN-4833 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fa78c9709539 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 37d939a Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12672/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12672/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment - - edited

          We can add AccessControlException with fail retry policy in exceptiontopolicymap (in RMProxy) as well. I do not think ACL related errors will change upon retrying.

          Show
          varun_saxena Varun Saxena added a comment - - edited We can add AccessControlException with fail retry policy in exceptiontopolicymap (in RMProxy) as well. I do not think ACL related errors will change upon retrying.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching rebased patch.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching rebased patch.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Bibin A Chundatt,
          Why not the first approach (should handle the AccessControlException in RetryPolicy and should go to fallbackPolicy) which you had initially mentioned ?,
          And if its approach2 as per the patch i would suggest to handle it in ClientRMService rather than RMAppManager

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Bibin A Chundatt , Why not the first approach (should handle the AccessControlException in RetryPolicy and should go to fallbackPolicy) which you had initially mentioned ?, And if its approach2 as per the patch i would suggest to handle it in ClientRMService rather than RMAppManager
          Hide
          sunilg Sunil G added a comment -

          Bibin A Chundatt ,could you please rebase the patch.

          Show
          sunilg Sunil G added a comment - Bibin A Chundatt ,could you please rebase the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s YARN-4833 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801651/0001-YARN-4833.patch
          JIRA Issue YARN-4833
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12671/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s YARN-4833 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801651/0001-YARN-4833.patch JIRA Issue YARN-4833 Console output https://builds.apache.org/job/PreCommit-YARN-Build/12671/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 42s trunk passed
          +1 compile 0m 25s trunk passed with JDK v1.8.0_91
          +1 compile 0m 30s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 35s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 4s trunk passed
          +1 javadoc 0m 22s trunk passed with JDK v1.8.0_91
          +1 javadoc 0m 27s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 24s the patch passed with JDK v1.8.0_91
          +1 javac 0m 24s the patch passed
          +1 compile 0m 27s the patch passed with JDK v1.7.0_95
          +1 javac 0m 27s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 14s the patch passed
          +1 javadoc 0m 20s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95
          -1 unit 48m 37s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
          -1 unit 48m 36s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 16s Patch does not generate ASF License warnings.
          113m 39s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.8.0_91 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801651/0001-YARN-4833.patch
          JIRA Issue YARN-4833
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7186d08aea1c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 971af60
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11296/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11296/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11296/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11296/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11296/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11296/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 42s trunk passed +1 compile 0m 25s trunk passed with JDK v1.8.0_91 +1 compile 0m 30s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 4s trunk passed +1 javadoc 0m 22s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 27s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 30s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_91 +1 javac 0m 24s the patch passed +1 compile 0m 27s the patch passed with JDK v1.7.0_95 +1 javac 0m 27s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 14s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95 -1 unit 48m 37s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. -1 unit 48m 36s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 16s Patch does not generate ASF License warnings. 113m 39s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.8.0_91 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801651/0001-YARN-4833.patch JIRA Issue YARN-4833 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7186d08aea1c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 971af60 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11296/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11296/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11296/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11296/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11296/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11296/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching patch for the same

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching patch for the same
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G
          Thank you for looking into the issue.

          Actually by Point 2 was thinking to throw RPC.getRemoteException which is YarnException.
          Will try the same and upload a patch soon

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G Thank you for looking into the issue. Actually by Point 2 was thinking to throw RPC.getRemoteException which is YarnException. Will try the same and upload a patch soon
          Hide
          sunilg Sunil G added a comment -

          Hi Bibin A Chundatt and Jian He

          I have some doubts here. AccessControlException is used in various places and in many places its been caught from caller as IOException. May be I didnt fully understand the point 2 in the mentioned solution "Wrap AccessControl exception to YarnException". Do you mean that instead of throwing AccessControlException, it will be thrown as YarnExcpetion with the message from AccessControlException OR are u planning to have AccessControlException as YarnException in its inheritance hierarchy?

          Since submitApplication is user facing, I am not very sure about change to YarnException in these ACL specific cases.

          RPC RemoteException is handled with correct RetryPolicy. So is it possible to throw AccessControlException wrapped with RPC RemotException. May be I missed/overlooked something, pls help to correct me if I am wrong.

          Show
          sunilg Sunil G added a comment - Hi Bibin A Chundatt and Jian He I have some doubts here. AccessControlException is used in various places and in many places its been caught from caller as IOException. May be I didnt fully understand the point 2 in the mentioned solution "Wrap AccessControl exception to YarnException". Do you mean that instead of throwing AccessControlException, it will be thrown as YarnExcpetion with the message from AccessControlException OR are u planning to have AccessControlException as YarnException in its inheritance hierarchy? Since submitApplication is user facing, I am not very sure about change to YarnException in these ACL specific cases. RPC RemoteException is handled with correct RetryPolicy. So is it possible to throw AccessControlException wrapped with RPC RemotException. May be I missed/overlooked something, pls help to correct me if I am wrong.

            People

            • Assignee:
              bibinchundatt Bibin A Chundatt
              Reporter:
              bibinchundatt Bibin A Chundatt
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development