Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4288

NodeManager restart should keep retrying to register to RM while connection exception happens during RM failed over.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When NM get restarted, NodeStatusUpdaterImpl will try to register to RM with RPC which could throw following exceptions when RM get restarted at the same time, like following exception shows:

      2015-08-17 14:35:59,434 ERROR nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:rebootNodeStatusUpdaterAndRegisterWithRM(222)) - Unexpected error rebooting NodeStatusUpdater
      java.io.IOException: Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "172.27.62.28"; destination host is: "172.27.62.57":8025;
              at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
              at org.apache.hadoop.ipc.Client.call(Client.java:1473)
              at org.apache.hadoop.ipc.Client.call(Client.java:1400)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
              at com.sun.proxy.$Proxy36.registerNodeManager(Unknown Source)
              at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:68)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
              at com.sun.proxy.$Proxy37.registerNodeManager(Unknown Source)
              at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:257)
              at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.rebootNodeStatusUpdaterAndRegisterWithRM(NodeStatusUpdaterImpl.java:215)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager$2.run(NodeManager.java:304)
      Caused by: java.io.IOException: Connection reset by peer
              at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
              at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
              at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
              at sun.nio.ch.IOUtil.read(IOUtil.java:197)
              at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
              at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
              at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
              at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
              at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
              at java.io.FilterInputStream.read(FilterInputStream.java:133)
              at java.io.FilterInputStream.read(FilterInputStream.java:133)
              at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:514)
              at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
              at java.io.DataInputStream.readInt(DataInputStream.java:387)
              at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
              at org.apache.hadoop.ipc.Client$Connection.run(Client.java:967)
      2015-08-17 14:35:59,436 FATAL nodemanager.NodeManager (NodeManager.java:run(307)) - Error while rebooting NodeStatusUpdater.
      org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException: Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "172.27.62.28"; destination host is: "172.27.62.57":8025;
              at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.rebootNodeStatusUpdaterAndRegisterWithRM(NodeStatusUpdaterImpl.java:223)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager$2.run(NodeManager.java:304)
      Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "ebdp-ch2-172.27.62.28"; destination host is: "172.27.62.57":8025;
              at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
              at org.apache.hadoop.ipc.Client.call(Client.java:1473)
              at org.apache.hadoop.ipc.Client.call(Client.java:1400)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
              at com.sun.proxy.$Proxy36.registerNodeManager(Unknown Source)
              at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:68)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
              at com.sun.proxy.$Proxy37.registerNodeManager(Unknown Source)
              at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:257)
              at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.rebootNodeStatusUpdaterAndRegisterWithRM(NodeStatusUpdaterImpl.java:215)
              ... 1 more
      Caused by: java.io.IOException: Connection reset by peer
              at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
              at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
              at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
              at sun.nio.ch.IOUtil.read(IOUtil.java:197)
              at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
              at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
              at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
              at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
              at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
              at java.io.FilterInputStream.read(FilterInputStream.java:133)
              at java.io.FilterInputStream.read(FilterInputStream.java:133)
              at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:514)
              at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
              at java.io.DataInputStream.readInt(DataInputStream.java:387)
              at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
              at org.apache.hadoop.ipc.Client$Connection.run(Client.java:967)
      2015-08-17 14:35:59,445 INFO  mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
      2015-08-17 14:35:59,547 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(512)) - Applications still running : [application_1439417357296_45357, application_1439417357296_45403, application_1439417357296_45355, application_1439417357296_45111, application_1439417357296_45452, application_1439417357296_45350, application_1439417357296_45499, application_1439417357296_45205, application_1439417357296_21009]
      2015-08-17 14:35:59,548 INFO  ipc.Server (Server.java:stop(2469)) - Stopping server on 45454
      2015-08-17 14:35:59,551 INFO  ipc.Server (Server.java:run(717)) - Stopping IPC Server listener on 45454
      2015-08-17 14:35:59,551 INFO  logaggregation.LogAggregationService (LogAggregationService.java:serviceStop(141)) - org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService waiting for pending aggregation during exit
      2015-08-17 14:35:59,552 INFO  ipc.Server (Server.java:run(843)) - Stopping IPC Server Responder
      

      It will make NM restart get failed. We should have a simple fix to allow this register to RM can retry with connection failures.

      1. YARN-4288.patch
        6 kB
        Junping Du
      2. YARN-4288-v2.patch
        8 kB
        Junping Du
      3. YARN-4288-v3.patch
        8 kB
        Junping Du

        Issue Links

          Activity

          Hide
          lichangleo Chang Li added a comment -

          Hi Junping Du, I have worked on YARN-4132, would that also address this issue?

          Show
          lichangleo Chang Li added a comment - Hi Junping Du , I have worked on YARN-4132 , would that also address this issue?
          Hide
          djp Junping Du added a comment -

          Thanks Chang Li for point that JIRA out.
          I think YARN-4132 may not help in this case because RMProxy will use RetryPolicies.TRY_ONCE_THEN_FAIL when HA is enabled so it still get failed without any retry. We should have a way to address retry for NM to register to RM in RM HA case.

          Show
          djp Junping Du added a comment - Thanks Chang Li for point that JIRA out. I think YARN-4132 may not help in this case because RMProxy will use RetryPolicies.TRY_ONCE_THEN_FAIL when HA is enabled so it still get failed without any retry. We should have a way to address retry for NM to register to RM in RM HA case.
          Hide
          djp Junping Du added a comment -

          Upload a quick patch to fix it.

          Show
          djp Junping Du added a comment - Upload a quick patch to fix it.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 22m 49s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 10m 41s There were no new javac warning messages.
          +1 javadoc 13m 48s There were no new javadoc warning messages.
          +1 release audit 0m 32s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 49s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 57s mvn install still works.
          +1 eclipse:eclipse 0m 43s The patch built with eclipse:eclipse.
          +1 findbugs 1m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 9m 34s Tests failed in hadoop-yarn-server-nodemanager.
              62m 32s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768048/YARN-4288.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 2798723
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9523/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-nodemanager.html
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9523/artifact/patchprocess/whitespace.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9523/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9523/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9523/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 22m 49s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 10m 41s There were no new javac warning messages. +1 javadoc 13m 48s There were no new javadoc warning messages. +1 release audit 0m 32s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 49s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 57s mvn install still works. +1 eclipse:eclipse 0m 43s The patch built with eclipse:eclipse. +1 findbugs 1m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 9m 34s Tests failed in hadoop-yarn-server-nodemanager.     62m 32s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768048/YARN-4288.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 2798723 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9523/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-nodemanager.html whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9523/artifact/patchprocess/whitespace.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9523/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9523/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9523/console This message was automatically generated.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Couple of questions

          • Why isn't existing RMProxy framework taking care of this?
          • Why are we putting special code in NodeStatusUpdater? Shouldn't we use something in the RMProxy framework? See ServerProxy for example that gets used by NMClients.
          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Couple of questions Why isn't existing RMProxy framework taking care of this? Why are we putting special code in NodeStatusUpdater? Shouldn't we use something in the RMProxy framework? See ServerProxy for example that gets used by NMClients.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Just looked at YARN-4132 too, we should definitely see if we can merge these two together.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Just looked at YARN-4132 too, we should definitely see if we can merge these two together.
          Hide
          djp Junping Du added a comment -

          Thanks Vinod Kumar Vavilapalli for the comments.

          Why isn't existing RMProxy framework taking care of this?

          RMProxy is supposed to take care of this. However, the way that RMProxy to do is to do retry on specific (known) exceptions but fail directly for other exceptions. Like this case, IOException get thrown will get failed directly without any retry (for non-HA case). We are a little risky if more potential exception could get thrown during RM down time. For this particular case, I can add the IOException (other than RemoteException) to be handled directly which sounds a easy way of fix.

          Why are we putting special code in NodeStatusUpdater? Shouldn't we use something in the RMProxy framework? See ServerProxy for example that gets used by NMClients.

          As I mentioned above, having a white list of exceptions to retry doesn't sound robust enough: if any exception we don't meet before, we could skip the retry unintentionally. Isn't it? Anyway, I could fix the problem with following existing retry policy framework but hopefully we could improve the framework in other JIRA.

          Just looked at YARN-4132 too, we should definitely see if we can merge these two together.

          This is a bug that NM doesn't retry in some cases. YARN-4132 talk about another problem that NM retry should be longer than general RMProxy client which is a more general improvement. I think we'd better separate them out. Thoughts?

          Show
          djp Junping Du added a comment - Thanks Vinod Kumar Vavilapalli for the comments. Why isn't existing RMProxy framework taking care of this? RMProxy is supposed to take care of this. However, the way that RMProxy to do is to do retry on specific (known) exceptions but fail directly for other exceptions. Like this case, IOException get thrown will get failed directly without any retry (for non-HA case). We are a little risky if more potential exception could get thrown during RM down time. For this particular case, I can add the IOException (other than RemoteException) to be handled directly which sounds a easy way of fix. Why are we putting special code in NodeStatusUpdater? Shouldn't we use something in the RMProxy framework? See ServerProxy for example that gets used by NMClients. As I mentioned above, having a white list of exceptions to retry doesn't sound robust enough: if any exception we don't meet before, we could skip the retry unintentionally. Isn't it? Anyway, I could fix the problem with following existing retry policy framework but hopefully we could improve the framework in other JIRA. Just looked at YARN-4132 too, we should definitely see if we can merge these two together. This is a bug that NM doesn't retry in some cases. YARN-4132 talk about another problem that NM retry should be longer than general RMProxy client which is a more general improvement. I think we'd better separate them out. Thoughts?
          Hide
          djp Junping Du added a comment -

          Update v2 patch to fix issue on RMProxy only.

          Show
          djp Junping Du added a comment - Update v2 patch to fix issue on RMProxy only.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 19m 17s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 8m 15s There were no new javac warning messages.
          +1 javadoc 10m 42s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 45s The applied patch generated 3 new checkstyle issues (total was 41, now 44).
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 3m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 6m 40s Tests failed in hadoop-common.
          +1 yarn tests 2m 4s Tests passed in hadoop-yarn-common.
              55m 2s  



          Reason Tests
          Failed unit tests hadoop.ipc.TestDecayRpcScheduler



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12769058/YARN-4288-v2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 68ce93c
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9591/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9591/artifact/patchprocess/diffcheckstylehadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9591/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9591/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9591/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9591/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9591/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 19m 17s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 8m 15s There were no new javac warning messages. +1 javadoc 10m 42s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 45s The applied patch generated 3 new checkstyle issues (total was 41, now 44). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 3m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 6m 40s Tests failed in hadoop-common. +1 yarn tests 2m 4s Tests passed in hadoop-yarn-common.     55m 2s   Reason Tests Failed unit tests hadoop.ipc.TestDecayRpcScheduler Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12769058/YARN-4288-v2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 68ce93c Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9591/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9591/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9591/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9591/artifact/patchprocess/testrun_hadoop-common.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9591/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9591/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9591/console This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Fix the whitespace and findbug issues in v3. The test failure is not related and already be tracked in HADOOP-11636.

          Show
          djp Junping Du added a comment - Fix the whitespace and findbug issues in v3. The test failure is not related and already be tracked in HADOOP-11636 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 3m 38s trunk passed
          +1 compile 4m 35s trunk passed with JDK v1.8.0_60
          +1 compile 4m 23s trunk passed with JDK v1.7.0_79
          +1 checkstyle 1m 0s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          -1 findbugs 1m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 22s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 40s trunk passed with JDK v1.7.0_79
          +1 mvninstall 2m 3s the patch passed
          +1 compile 4m 25s the patch passed with JDK v1.8.0_60
          +1 javac 4m 25s the patch passed
          +1 compile 4m 16s the patch passed with JDK v1.7.0_79
          +1 javac 4m 16s the patch passed
          -1 checkstyle 0m 57s Patch generated 1 new checkstyle issues in root (total was 44, now 45).
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 9s the patch passed
          +1 javadoc 1m 20s the patch passed with JDK v1.8.0_60
          +1 javadoc 1m 38s the patch passed with JDK v1.7.0_79
          +1 unit 6m 57s hadoop-common in the patch passed with JDK v1.8.0_60.
          +1 unit 1m 47s hadoop-yarn-common in the patch passed with JDK v1.8.0_60.
          +1 unit 7m 18s hadoop-common in the patch passed with JDK v1.7.0_79.
          +1 unit 2m 7s hadoop-yarn-common in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          58m 22s



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-28
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769294/YARN-4288-v3.patch
          JIRA Issue YARN-4288
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux a5588f4967ed 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh
          git revision trunk / 27414da
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9596/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9596/artifact/patchprocess/diff-checkstyle-root.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9596/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: .
          Max memory used 225MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9596/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 3m 38s trunk passed +1 compile 4m 35s trunk passed with JDK v1.8.0_60 +1 compile 4m 23s trunk passed with JDK v1.7.0_79 +1 checkstyle 1m 0s trunk passed +1 mvneclipse 0m 29s trunk passed -1 findbugs 1m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 22s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 40s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 3s the patch passed +1 compile 4m 25s the patch passed with JDK v1.8.0_60 +1 javac 4m 25s the patch passed +1 compile 4m 16s the patch passed with JDK v1.7.0_79 +1 javac 4m 16s the patch passed -1 checkstyle 0m 57s Patch generated 1 new checkstyle issues in root (total was 44, now 45). +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 9s the patch passed +1 javadoc 1m 20s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 38s the patch passed with JDK v1.7.0_79 +1 unit 6m 57s hadoop-common in the patch passed with JDK v1.8.0_60. +1 unit 1m 47s hadoop-yarn-common in the patch passed with JDK v1.8.0_60. +1 unit 7m 18s hadoop-common in the patch passed with JDK v1.7.0_79. +1 unit 2m 7s hadoop-yarn-common in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 58m 22s Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-28 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769294/YARN-4288-v3.patch JIRA Issue YARN-4288 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux a5588f4967ed 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh git revision trunk / 27414da Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9596/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9596/artifact/patchprocess/diff-checkstyle-root.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9596/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: . Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9596/console This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          The findbug warning and checkstyle is not related.

          Show
          djp Junping Du added a comment - The findbug warning and checkstyle is not related.
          Hide
          jianhe Jian He added a comment -

          lgtm

          Show
          jianhe Jian He added a comment - lgtm
          Hide
          jianhe Jian He added a comment -

          Committed to trunk and branch-2, thanks Junping !

          Show
          jianhe Jian He added a comment - Committed to trunk and branch-2, thanks Junping !
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8723 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8723/)
          YARN-4288. Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8723 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8723/ ) YARN-4288 . Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #599 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/599/)
          YARN-4288. Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #599 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/599/ ) YARN-4288 . Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java hadoop-yarn-project/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2542 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2542/)
          YARN-4288. Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2542 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2542/ ) YARN-4288 . Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188) hadoop-yarn-project/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1335 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1335/)
          YARN-4288. Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1335 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1335/ ) YARN-4288 . Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #612 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/612/)
          YARN-4288. Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #612 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/612/ ) YARN-4288 . Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188) hadoop-yarn-project/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2488 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2488/)
          YARN-4288. Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2488 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2488/ ) YARN-4288 . Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #550 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/550/)
          YARN-4288. Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #550 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/550/ ) YARN-4288 . Fixed RMProxy to retry on IOException from local host. (jianhe: rev c41699965e78ce5e87669d17923ab84e494c4188) hadoop-yarn-project/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Junping! We've seen AMRMClientImpl die with connection reset by peer instead of retrying in the RM proxy layer on 2.7, so I committed this to branch-2.7 as well.

          Show
          jlowe Jason Lowe added a comment - Thanks, Junping! We've seen AMRMClientImpl die with connection reset by peer instead of retrying in the RM proxy layer on 2.7, so I committed this to branch-2.7 as well.
          Hide
          djp Junping Du added a comment -

          Sure. Please go ahead. Thanks Jason!

          Show
          djp Junping Du added a comment - Sure. Please go ahead. Thanks Jason!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              djp Junping Du
              Reporter:
              djp Junping Du
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development