Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4913

TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha, 0.23.6
    • Fix Version/s: 3.0.0, 2.0.3-alpha, 0.23.6
    • Component/s: mr-am
    • Labels:
      None

      Description

      testMRAppMasterMissingStaging will sometimes cause the JVM to exit due to this error from AsyncDispatcher:

      2013-01-05 02:14:54,682 FATAL [AsyncDispatcher event handler] event.AsyncDispatcher (AsyncDispatcher.java:dispatch(137)) - Error in dispatcher thread
      java.lang.Exception: No handler for registered for class org.apache.hadoop.mapreduce.jobhistory.EventType, cannot deliver EventType: AM_STARTED
              at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:132)
              at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
              at java.lang.Thread.run(Thread.java:662)
      2013-01-05 02:14:54,682 INFO  [AsyncDispatcher event handler] event.AsyncDispatcher (AsyncDispatcher.java:dispatch(140)) - Exiting, bbye..
      

      This can cause a build to fail since the test process exits without unregistering from surefire which treats it as a build error rather than a test failure.

        Issue Links

          Activity

          Hide
          Jason Lowe added a comment -

          There is a race condition in the MRAppMaster when handling a missing staging directory during recovery. In that case we do not register an event handler for jobhistory.EventType but MRAppMaster sends AM_STARTED later during its start() method.

          If the AsyncDispatcher sees the unexpected event before we shutdown from the missing staging directory error then the AsyncDispatcher will call System.exit() due to the unexpected event.

          Show
          Jason Lowe added a comment - There is a race condition in the MRAppMaster when handling a missing staging directory during recovery. In that case we do not register an event handler for jobhistory.EventType but MRAppMaster sends AM_STARTED later during its start() method. If the AsyncDispatcher sees the unexpected event before we shutdown from the missing staging directory error then the AsyncDispatcher will call System.exit() due to the unexpected event.
          Hide
          Jason Lowe added a comment -

          Patch to register the NoopEventHandler for jobhistory.EventType when we're in the staging directory missing case and are just trying to cleanup. No unit test since this is fixing an existing test.

          Show
          Jason Lowe added a comment - Patch to register the NoopEventHandler for jobhistory.EventType when we're in the staging directory missing case and are just trying to cleanup. No unit test since this is fixing an existing test.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12563390/MAPREDUCE-4913.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3203//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3203//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563390/MAPREDUCE-4913.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3203//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3203//console This message is automatically generated.
          Hide
          Thomas Graves added a comment -

          +1. thanks Jason!

          Show
          Thomas Graves added a comment - +1. thanks Jason!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3180 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3180/)
          MAPREDUCE-4913. TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits (Jason Lowe via tgraves) (Revision 1429371)

          Result = SUCCESS
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1429371
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3180 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3180/ ) MAPREDUCE-4913 . TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits (Jason Lowe via tgraves) (Revision 1429371) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1429371 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #88 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/88/)
          MAPREDUCE-4913. TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits (Jason Lowe via tgraves) (Revision 1429371)

          Result = SUCCESS
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1429371
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #88 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/88/ ) MAPREDUCE-4913 . TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits (Jason Lowe via tgraves) (Revision 1429371) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1429371 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #486 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/486/)
          MAPREDUCE-4913. TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits (Jason Lowe via tgraves) (Revision 1429377)

          Result = FAILURE
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1429377
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #486 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/486/ ) MAPREDUCE-4913 . TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits (Jason Lowe via tgraves) (Revision 1429377) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1429377 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1277 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1277/)
          MAPREDUCE-4913. TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits (Jason Lowe via tgraves) (Revision 1429371)

          Result = FAILURE
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1429371
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1277 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1277/ ) MAPREDUCE-4913 . TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits (Jason Lowe via tgraves) (Revision 1429371) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1429371 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1307 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1307/)
          MAPREDUCE-4913. TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits (Jason Lowe via tgraves) (Revision 1429371)

          Result = SUCCESS
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1429371
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1307 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1307/ ) MAPREDUCE-4913 . TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits (Jason Lowe via tgraves) (Revision 1429371) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1429371 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java

            People

            • Assignee:
              Jason Lowe
              Reporter:
              Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development