Hadoop YARN
  1. Hadoop YARN
  2. YARN-139

Interrupted Exception within AsyncDispatcher leads to user confusion

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha, 0.23.4
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: api
    • Labels:
      None

      Description

      Successful applications tend to get InterruptedExceptions during shutdown. The exception is harmless but it leads to lots of user confusion and therefore could be cleaned up.

      2012-09-28 14:50:12,477 WARN [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Interrupted Exception while stopping
      java.lang.InterruptedException
      at java.lang.Object.wait(Native Method)
      at java.lang.Thread.join(Thread.java:1143)
      at java.lang.Thread.join(Thread.java:1196)
      at org.apache.hadoop.yarn.event.AsyncDispatcher.stop(AsyncDispatcher.java:105)
      at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99)
      at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:437)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:402)
      at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
      at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
      at java.lang.Thread.run(Thread.java:619)
      2012-09-28 14:50:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is stopped.
      2012-09-28 14:50:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.mapreduce.v2.app.MRAppMaster is stopped.
      2012-09-28 14:50:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Exiting MR AppMaster..GoodBye

      1. YARN-139-20121023.txt
        8 kB
        Vinod Kumar Vavilapalli
      2. YARN-139-20121019.txt
        4 kB
        Vinod Kumar Vavilapalli
      3. YARN-139-20121019.1.txt
        7 kB
        Vinod Kumar Vavilapalli
      4. YARN-139.txt
        1 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1236 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1236/)
          YARN-139. Interrupted Exception within AsyncDispatcher leads to user confusion. Contributed by Vinod Kumar Vavilapalli (Revision 1401726)

          Result = FAILURE
          jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401726
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1236 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1236/ ) YARN-139 . Interrupted Exception within AsyncDispatcher leads to user confusion. Contributed by Vinod Kumar Vavilapalli (Revision 1401726) Result = FAILURE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401726 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1206 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1206/)
          YARN-139. Interrupted Exception within AsyncDispatcher leads to user confusion. Contributed by Vinod Kumar Vavilapalli (Revision 1401726)

          Result = SUCCESS
          jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401726
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1206 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1206/ ) YARN-139 . Interrupted Exception within AsyncDispatcher leads to user confusion. Contributed by Vinod Kumar Vavilapalli (Revision 1401726) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401726 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #415 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/415/)
          svn merge -c 1401726 FIXES: YARN-139. Interrupted Exception within AsyncDispatcher leads to user confusion. Contributed by Vinod Kumar Vavilapalli (Revision 1401731)

          Result = SUCCESS
          jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401731
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #415 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/415/ ) svn merge -c 1401726 FIXES: YARN-139 . Interrupted Exception within AsyncDispatcher leads to user confusion. Contributed by Vinod Kumar Vavilapalli (Revision 1401731) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401731 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #16 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/16/)
          YARN-139. Interrupted Exception within AsyncDispatcher leads to user confusion. Contributed by Vinod Kumar Vavilapalli (Revision 1401726)

          Result = SUCCESS
          jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401726
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #16 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/16/ ) YARN-139 . Interrupted Exception within AsyncDispatcher leads to user confusion. Contributed by Vinod Kumar Vavilapalli (Revision 1401726) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401726 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #2923 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2923/)
          YARN-139. Interrupted Exception within AsyncDispatcher leads to user confusion. Contributed by Vinod Kumar Vavilapalli (Revision 1401726)

          Result = SUCCESS
          jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401726
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #2923 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2923/ ) YARN-139 . Interrupted Exception within AsyncDispatcher leads to user confusion. Contributed by Vinod Kumar Vavilapalli (Revision 1401726) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401726 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
          Hide
          Jason Lowe added a comment -

          Thanks again, Vinod. I committed this to trunk, branch-2, and branch-0.23.

          Show
          Jason Lowe added a comment - Thanks again, Vinod. I committed this to trunk, branch-2, and branch-0.23.
          Hide
          Jason Lowe added a comment -

          +1, thanks Vinod! I'll commit this shortly.

          Show
          Jason Lowe added a comment - +1, thanks Vinod! I'll commit this shortly.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550561/YARN-139-20121023.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/123//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/123//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550561/YARN-139-20121023.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/123//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/123//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Thanks Jason, was thinking of fixing that separately but here it goes.

          Show
          Vinod Kumar Vavilapalli added a comment - Thanks Jason, was thinking of fixing that separately but here it goes.
          Hide
          Jason Lowe added a comment -

          Filed MAPREDUCE-4741 to track the other WARN and ERROR messages being logged by the AM during shutdown.

          Show
          Jason Lowe added a comment - Filed MAPREDUCE-4741 to track the other WARN and ERROR messages being logged by the AM during shutdown.
          Hide
          Jason Lowe added a comment -

          This fixes the "Interrupted Exception while stopping" stacktrace, but now we have another:

          2012-10-22 14:39:24,760 WARN [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher thread interrupted
          java.lang.InterruptedException
          	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961)
          	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
          	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
          	at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:69)
          	at java.lang.Thread.run(Thread.java:662)
          

          We can simply check for !stopped before logging the exception to nix that.

          I'll file a separate JIRA to fix the other WARN and ERROR messages being logged during AM shutdown.

          Show
          Jason Lowe added a comment - This fixes the "Interrupted Exception while stopping" stacktrace, but now we have another: 2012-10-22 14:39:24,760 WARN [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher thread interrupted java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:69) at java.lang.Thread.run(Thread.java:662) We can simply check for !stopped before logging the exception to nix that. I'll file a separate JIRA to fix the other WARN and ERROR messages being logged during AM shutdown.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550123/YARN-139-20121019.1.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/109//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/109//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550123/YARN-139-20121019.1.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/109//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/109//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Fixing the test issue.

          Show
          Vinod Kumar Vavilapalli added a comment - Fixing the test issue.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550115/YARN-139-20121019.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:

          org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/107//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/107//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550115/YARN-139-20121019.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/107//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/107//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Tried out the patch, but it doesn't completely fix the problem. It does prevent spurious exception traces in the handler threads when they are interrupted, but the exception trace reported in the description is coming from the thread calling stop() on the AsyncDispatcher service. It was interrupted while trying to join with the handling thread, and that exception is still being logged.

          I completely misread the JIRA description. Let's track the AsyncDispatcher logs separately.

          Here's a patch to fix this. Can you please take a look?

          Show
          Vinod Kumar Vavilapalli added a comment - Tried out the patch, but it doesn't completely fix the problem. It does prevent spurious exception traces in the handler threads when they are interrupted, but the exception trace reported in the description is coming from the thread calling stop() on the AsyncDispatcher service. It was interrupted while trying to join with the handling thread, and that exception is still being logged. I completely misread the JIRA description. Let's track the AsyncDispatcher logs separately. Here's a patch to fix this. Can you please take a look?
          Hide
          Jason Lowe added a comment -

          Tried out the patch, but it doesn't completely fix the problem. It does prevent spurious exception traces in the handler threads when they are interrupted, but the exception trace reported in the description is coming from the thread calling stop() on the AsyncDispatcher service. It was interrupted while trying to join with the handling thread, and that exception is still being logged.

          Also related to the user confusion are the other errors and warnings being logged during normal AM shutdown. Some examples:

          2012-10-02 13:58:50,247 ERROR [ContainerLauncher Event Handler] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Returning, interrupted : java.lang.InterruptedException
          [...]
          2012-10-02 13:58:50,248 ERROR [Thread-47] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Returning, interrupted : java.lang.InterruptedException
          2012-10-02 13:58:50,248 WARN [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Allocated thread interrupted. Returning.
          [...]
          2012-10-02 13:58:50,367 ERROR [TaskCleaner Event Handler] org.apache.hadoop.mapreduce.v2.app.taskclean.TaskCleanerImpl: Returning, interrupted : java.lang.InterruptedException
          

          I can file another JIRA if we'd rather handle those separately.

          Show
          Jason Lowe added a comment - Tried out the patch, but it doesn't completely fix the problem. It does prevent spurious exception traces in the handler threads when they are interrupted, but the exception trace reported in the description is coming from the thread calling stop() on the AsyncDispatcher service. It was interrupted while trying to join with the handling thread, and that exception is still being logged. Also related to the user confusion are the other errors and warnings being logged during normal AM shutdown. Some examples: 2012-10-02 13:58:50,247 ERROR [ContainerLauncher Event Handler] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Returning, interrupted : java.lang.InterruptedException [...] 2012-10-02 13:58:50,248 ERROR [Thread-47] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Returning, interrupted : java.lang.InterruptedException 2012-10-02 13:58:50,248 WARN [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Allocated thread interrupted. Returning. [...] 2012-10-02 13:58:50,367 ERROR [TaskCleaner Event Handler] org.apache.hadoop.mapreduce.v2.app.taskclean.TaskCleanerImpl: Returning, interrupted : java.lang.InterruptedException I can file another JIRA if we'd rather handle those separately.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12547305/YARN-139.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/64//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/64//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547305/YARN-139.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/64//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/64//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Trivial patch. Logging the exception only if the dispatcher isn't throwing it during shutdown.

          Show
          Vinod Kumar Vavilapalli added a comment - Trivial patch. Logging the exception only if the dispatcher isn't throwing it during shutdown.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Yes, this indeed is annoying. Should be a trivial fix.

          Show
          Vinod Kumar Vavilapalli added a comment - Yes, this indeed is annoying. Should be a trivial fix.

            People

            • Assignee:
              Vinod Kumar Vavilapalli
              Reporter:
              Nathan Roberts
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development