Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2917

Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I encoutered scenario where RM hanged while shutting down and keep on logging 2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Waiting for AsyncDispatcher to drain.

      1. 0001-YARN-2917.patch
        0.8 kB
        Rohith Sharma K S
      2. 0002-YARN-2917.patch
        1 kB
        Rohith Sharma K S

        Activity

        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Attaching thread dump when RM hanged

        "Thread-1" prio=10 tid=0x00000000006e1000 nid=0x55a4 in Object.wait() [0x00007f2ce9493000]
           java.lang.Thread.State: TIMED_WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0x00000000f26b0d48> (a java.lang.Object)
        	at org.apache.hadoop.yarn.event.AsyncDispatcher.serviceStop(AsyncDispatcher.java:141)
        	- locked <0x00000000f26b0d48> (a java.lang.Object)
        	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
        	- locked <0x00000000f26b0aa8> (a java.lang.Object)
        	at org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.stopDispatcher(CommonNodeLabelsManager.java:232)
        	at org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStop(CommonNodeLabelsManager.java:238)
        	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
        	- locked <0x00000000f26b0968> (a java.lang.Object)
        	at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
        	at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
        	at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
        	at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
        	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStop(ResourceManager.java:599)
        	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
        	- locked <0x00000000f2842458> (a java.lang.Object)
        	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stopActiveServices(ResourceManager.java:1002)
        	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToStandby(ResourceManager.java:1057)
        	- locked <0x00000000c0c96c98> (a org.apache.hadoop.yarn.server.resourcemanager.ResourceManager)
        	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1104)
        	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
        	- locked <0x00000000c0cab280> (a java.lang.Object)
        	at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
        	at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
        	at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:65)
        	at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:183)
        	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
        
        "AsyncDispatcher event handler" daemon prio=10 tid=0x00007f2cf0b81000 nid=0x54a1 in Object.wait() [0x00007f2cf7bfa000]
           java.lang.Thread.State: WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0x00000000c01b83e8> (a org.apache.hadoop.util.ShutdownHookManager$1)
        	at java.lang.Thread.join(Thread.java:1281)
        	- locked <0x00000000c01b83e8> (a org.apache.hadoop.util.ShutdownHookManager$1)
        	at java.lang.Thread.join(Thread.java:1355)
        	at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
        	at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
        	at java.lang.Shutdown.runHooks(Shutdown.java:123)
        	at java.lang.Shutdown.sequence(Shutdown.java:167)
        	at java.lang.Shutdown.exit(Shutdown.java:212)
        	- locked <0x00000000c04ae9c0> (a java.lang.Class for java.lang.Shutdown)
        	at java.lang.Runtime.exit(Runtime.java:109)
        	at java.lang.System.exit(System.java:962)
        	at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:185)
        	at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
        	at java.lang.Thread.run(Thread.java:745)
        
        Show
        rohithsharma Rohith Sharma K S added a comment - Attaching thread dump when RM hanged " Thread -1" prio=10 tid=0x00000000006e1000 nid=0x55a4 in Object .wait() [0x00007f2ce9493000] java.lang. Thread .State: TIMED_WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0x00000000f26b0d48> (a java.lang. Object ) at org.apache.hadoop.yarn.event.AsyncDispatcher.serviceStop(AsyncDispatcher.java:141) - locked <0x00000000f26b0d48> (a java.lang. Object ) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) - locked <0x00000000f26b0aa8> (a java.lang. Object ) at org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.stopDispatcher(CommonNodeLabelsManager.java:232) at org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStop(CommonNodeLabelsManager.java:238) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) - locked <0x00000000f26b0968> (a java.lang. Object ) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStop(ResourceManager.java:599) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) - locked <0x00000000f2842458> (a java.lang. Object ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stopActiveServices(ResourceManager.java:1002) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToStandby(ResourceManager.java:1057) - locked <0x00000000c0c96c98> (a org.apache.hadoop.yarn.server.resourcemanager.ResourceManager) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1104) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) - locked <0x00000000c0cab280> (a java.lang. Object ) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:65) at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:183) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) "AsyncDispatcher event handler" daemon prio=10 tid=0x00007f2cf0b81000 nid=0x54a1 in Object .wait() [0x00007f2cf7bfa000] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0x00000000c01b83e8> (a org.apache.hadoop.util.ShutdownHookManager$1) at java.lang. Thread .join( Thread .java:1281) - locked <0x00000000c01b83e8> (a org.apache.hadoop.util.ShutdownHookManager$1) at java.lang. Thread .join( Thread .java:1355) at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106) at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46) at java.lang.Shutdown.runHooks(Shutdown.java:123) at java.lang.Shutdown.sequence(Shutdown.java:167) at java.lang.Shutdown.exit(Shutdown.java:212) - locked <0x00000000c04ae9c0> (a java.lang. Class for java.lang.Shutdown) at java.lang. Runtime .exit( Runtime .java:109) at java.lang. System .exit( System .java:962) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:185) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106) at java.lang. Thread .run( Thread .java:745)
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        The main problem is
        Thread-1 : CommonNodeLabelManager#handle() throw back exception to AsyncDispatcher. Intern, asyncDispatcher calls shutdown hook and waiting for shutdown hook to complete.
        Thread-2 : ShutodownHook stops RM gracefully.But gracefull stop wait for drainng events from AsyncDispatcher.

        Show
        rohithsharma Rohith Sharma K S added a comment - The main problem is Thread-1 : CommonNodeLabelManager#handle() throw back exception to AsyncDispatcher. Intern, asyncDispatcher calls shutdown hook and waiting for shutdown hook to complete. Thread-2 : ShutodownHook stops RM gracefully.But gracefull stop wait for drainng events from AsyncDispatcher.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Scenario to reoccur this issue

        1. Start Hadoop cluster(NN DN RM NM) by enabling node-label manager. configuration yarn.node-labels.manager-class=org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager and yarn.node-labels.fs-store.root-dir=/tmp/node-label
        2. Stop DataNode's
        3. Perform -addToClusterNodeLabels operation
        Show
        rohithsharma Rohith Sharma K S added a comment - Scenario to reoccur this issue Start Hadoop cluster(NN DN RM NM) by enabling node-label manager. configuration yarn.node-labels.manager-class=org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager and yarn.node-labels.fs-store.root-dir=/tmp/node-label Stop DataNode's Perform -addToClusterNodeLabels operation
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Looking into deeply AsyncDispatcher, there is potential deadlock acros AsyncDispatcher#dispatch()(System.exit) and AsyncDispatcher#serviceStop().
        When ever dispatcher thread self exit on exception from the thread eventHandlingThread, eventHandlingThread wait internally by jvm i.e ApplicationShutdownHooks which runs ShutdownHook and join the shudownhook thread. But in AsyncDispatcher#serviceStop(), eventHandlingThread is checked for isAlive() which return always true result is while loop forever.

        Show
        rohithsharma Rohith Sharma K S added a comment - Looking into deeply AsyncDispatcher, there is potential deadlock acros AsyncDispatcher#dispatch()(System.exit) and AsyncDispatcher#serviceStop() . When ever dispatcher thread self exit on exception from the thread eventHandlingThread , eventHandlingThread wait internally by jvm i.e ApplicationShutdownHooks which runs ShutdownHook and join the shudownhook thread. But in AsyncDispatcher#serviceStop(), eventHandlingThread is checked for isAlive() which return always true result is while loop forever.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Updated summary as per the analysis.

        Show
        rohithsharma Rohith Sharma K S added a comment - Updated summary as per the analysis.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        I verified the patch deploying in real cluster, syster is exitting. Please review tha patch, I did not write test case since system.exit called.

        Show
        rohithsharma Rohith Sharma K S added a comment - I verified the patch deploying in real cluster, syster is exitting. Please review tha patch, I did not write test case since system.exit called.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12685697/0001-YARN-2917.patch
        against trunk revision 120e1de.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6029//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6029//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12685697/0001-YARN-2917.patch against trunk revision 120e1de. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6029//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6029//console This message is automatically generated.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Hi Karthik Kambatla, Jian He, Vinod Kumar Vavilapalli Kindly review the analysis and patch. This issue is causing RM to hang.

        Show
        rohithsharma Rohith Sharma K S added a comment - Hi Karthik Kambatla , Jian He , Vinod Kumar Vavilapalli Kindly review the analysis and patch. This issue is causing RM to hang.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Rohith Sharma K S,
        Good catch! Thanks for thinking about this.

        My take is this will happen when:
        Step 1 : Thread #1 (event dispatcher thread) has some exception when dispatching, will call System.exit
        Step 2 : Thread #2 (RM main thread) registered ShutdownHook, and will finally call AsyncDispatcher.serviceExit
        Step 3 : Thread #1 Is waiting for System.exit(-1) returns and Thread #2 is waiting for thread #1 exit at the same time. It's a pair of deadlock.

        But my question is: is it correct to set drainEventsOnStop to be false when such fatal error happens? Shouldn't we wait for it to be drained even if fatal error happens?
        Any thoughts?

        Show
        leftnoteasy Wangda Tan added a comment - Rohith Sharma K S , Good catch! Thanks for thinking about this. My take is this will happen when: Step 1 : Thread #1 (event dispatcher thread) has some exception when dispatching, will call System.exit Step 2 : Thread #2 (RM main thread) registered ShutdownHook, and will finally call AsyncDispatcher.serviceExit Step 3 : Thread #1 Is waiting for System.exit(-1) returns and Thread #2 is waiting for thread #1 exit at the same time. It's a pair of deadlock. But my question is: is it correct to set drainEventsOnStop to be false when such fatal error happens? Shouldn't we wait for it to be drained even if fatal error happens? Any thoughts?
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        is it correct to set drainEventsOnStop to be false when such fatal error happens? Shouldn't we wait for it to be drained even if fatal error happens?

        Yes because however JVM is going to exit either gracefully or abnormally.Setting flag is indication that shutdowhook is called by self. It need not necessarily wait for draining events since JVM will be killed aggressively.

        Show
        rohithsharma Rohith Sharma K S added a comment - is it correct to set drainEventsOnStop to be false when such fatal error happens? Shouldn't we wait for it to be drained even if fatal error happens? Yes because however JVM is going to exit either gracefully or abnormally.Setting flag is indication that shutdowhook is called by self. It need not necessarily wait for draining events since JVM will be killed aggressively.
        Hide
        jianhe Jian He added a comment -

        Rohith Sharma K S, thanks for reporting this !

        how about creating a new thread and call sys exit ?

                LOG.info("Exiting, bbye..");
                System.exit(-1);
        
        Show
        jianhe Jian He added a comment - Rohith Sharma K S , thanks for reporting this ! how about creating a new thread and call sys exit ? LOG.info( "Exiting, bbye.." ); System .exit(-1);
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        how about creating a new thread and call sys exit ?

        This also good. I see many places System.exit are called in RM/NM or other daemons. There can be centralized single thread that does system.exit. Any thought?

        Show
        rohithsharma Rohith Sharma K S added a comment - how about creating a new thread and call sys exit ? This also good. I see many places System.exit are called in RM/NM or other daemons. There can be centralized single thread that does system.exit. Any thought?
        Hide
        jianhe Jian He added a comment -

        There can be centralized single thread that does system.exit.

        Then we need to send events to notify this single thread? IMO, we can keep it simple and just create a new thread to do sys exit.

        Show
        jianhe Jian He added a comment - There can be centralized single thread that does system.exit. Then we need to send events to notify this single thread? IMO, we can keep it simple and just create a new thread to do sys exit.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Then we need to send events to notify this single thread?

        No via events, directly call in static method. ExitUtil.terminate() starts new thread.

        Show
        rohithsharma Rohith Sharma K S added a comment - Then we need to send events to notify this single thread? No via events, directly call in static method. ExitUtil.terminate() starts new thread.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Jian He In between, Thinking why to spawn new thread when jvm is going exit?

        Show
        rohithsharma Rohith Sharma K S added a comment - Jian He In between, Thinking why to spawn new thread when jvm is going exit?
        Hide
        jianhe Jian He added a comment -

        IIUC, the problem is that System.exit(-1); in AsyncDispatcher#dispatch will wait for the shutdown hook to exit, but in the meanwhile the shutdown hook is waiting for the dispatcher to stop. please correct me if I'm wrong.

        Show
        jianhe Jian He added a comment - IIUC, the problem is that System.exit(-1); in AsyncDispatcher#dispatch will wait for the shutdown hook to exit, but in the meanwhile the shutdown hook is waiting for the dispatcher to stop. please correct me if I'm wrong.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Jian He your understanding is correct. I was meant to say that setting drainEventsOnStop to false would solve the issue by bypassing wait for draining events in while loop. But is there any specific reason to create new thread and call sys exit?

        Show
        rohithsharma Rohith Sharma K S added a comment - Jian He your understanding is correct. I was meant to say that setting drainEventsOnStop to false would solve the issue by bypassing wait for draining events in while loop. But is there any specific reason to create new thread and call sys exit?
        Hide
        jianhe Jian He added a comment -

        this way we can still drain the events when calling the sys exiting. setting drainEventsOnStop will not drain the events. right?

        Show
        jianhe Jian He added a comment - this way we can still drain the events when calling the sys exiting. setting drainEventsOnStop will not drain the events. right?
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        setting drainEventsOnStop will not drain the events. right?

        Yes, it will not drain. I am thinking that when system is exiting is it really necessary to drain the event? I agree to draining of events when rm is transitioning to standby,but here JVM itself is exiting. Consider other case suppose JVM is killed using SIGKILL? Here also, events are not drained. But SIGKILL does not cause any issues further after RM restart.

        Show
        rohithsharma Rohith Sharma K S added a comment - setting drainEventsOnStop will not drain the events. right? Yes, it will not drain. I am thinking that when system is exiting is it really necessary to drain the event? I agree to draining of events when rm is transitioning to standby,but here JVM itself is exiting. Consider other case suppose JVM is killed using SIGKILL? Here also, events are not drained. But SIGKILL does not cause any issues further after RM restart.
        Hide
        jianhe Jian He added a comment -

        agree that if RM is abruptly killed, no events will be drained. In fact, things continue to work irrespective of whether events are drained or not either when RM is transiting or restarting. Draining events is just one optimization to process the remaining events before RM is shutting down. Similarly in this jira, there's no correctness concern whether to drain or not, it's just a optimization, unless I miss some other side effects. do you think if there's any ?

        Show
        jianhe Jian He added a comment - agree that if RM is abruptly killed, no events will be drained. In fact, things continue to work irrespective of whether events are drained or not either when RM is transiting or restarting. Draining events is just one optimization to process the remaining events before RM is shutting down. Similarly in this jira, there's no correctness concern whether to drain or not, it's just a optimization, unless I miss some other side effects. do you think if there's any ?
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Thanks for your explanation about necessacity of draining events. I do not think any other side effects. I will upload new patch soon.

        Show
        rohithsharma Rohith Sharma K S added a comment - Thanks for your explanation about necessacity of draining events. I do not think any other side effects. I will upload new patch soon.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Kindly review the attached patch. I have manually verified problematic scenario as I mentioned in my previous comment by deploying in 1 node cluster. It is able to shutdown gracefully calling ShutdownHook.

        Not related to this jira : I observed that ResourceManager#rmDispatcher does not drain ,is it bug?

        Show
        rohithsharma Rohith Sharma K S added a comment - Kindly review the attached patch. I have manually verified problematic scenario as I mentioned in my previous comment by deploying in 1 node cluster. It is able to shutdown gracefully calling ShutdownHook. Not related to this jira : I observed that ResourceManager#rmDispatcher does not drain ,is it bug?
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12686586/0002-YARN-2917.patch
        against trunk revision 390642a.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 25 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6083//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6083//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6083//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12686586/0002-YARN-2917.patch against trunk revision 390642a. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 25 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6083//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6083//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6083//console This message is automatically generated.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Findbug warnings are unrelated to this patch

        Show
        rohithsharma Rohith Sharma K S added a comment - Findbug warnings are unrelated to this patch
        Hide
        jianhe Jian He added a comment -

        patch looks good, thanks Rohith !
        Committing this.

        I observed that ResourceManager#rmDispatcher does not drain ,is it bug?

        I think the decision was to make the state-store relevant dispatcher drained. rmDispatcher is a global dispatcher. draining that may take more time depending on how busy the cluster is. I think it's still fine to drain the rmDispatcher, but need to evaluate the pros and cons.

        Show
        jianhe Jian He added a comment - patch looks good, thanks Rohith ! Committing this. I observed that ResourceManager#rmDispatcher does not drain ,is it bug? I think the decision was to make the state-store relevant dispatcher drained. rmDispatcher is a global dispatcher. draining that may take more time depending on how busy the cluster is. I think it's still fine to drain the rmDispatcher, but need to evaluate the pros and cons.
        Hide
        jianhe Jian He added a comment -

        Committed to trunk and branch-2, thanks Rohith !

        Show
        jianhe Jian He added a comment - Committed to trunk and branch-2, thanks Rohith !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #6698 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6698/)
        YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6698 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6698/ ) YARN-2917 . Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #38 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/38/)
        YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #38 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/38/ ) YARN-2917 . Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #773 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/773/)
        YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #773 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/773/ ) YARN-2917 . Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1970 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1970/)
        YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1970 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1970/ ) YARN-2917 . Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #36 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/36/)
        YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #36 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/36/ ) YARN-2917 . Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #40 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/40/)
        YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #40 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/40/ ) YARN-2917 . Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1990 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1990/)
        YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1990 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1990/ ) YARN-2917 . Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Pulled this into 2.6.1. Ran compilation before the push. Patch applied cleanly.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.6.1. Ran compilation before the push. Patch applied cleanly.

          People

          • Assignee:
            rohithsharma Rohith Sharma K S
            Reporter:
            rohithsharma Rohith Sharma K S
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development