Hadoop Common
  1. Hadoop Common
  2. HADOOP-4112

Got ArrayOutOfBound exception while analyzing the job history

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.

      1. patch-4112.txt
        21 kB
        Amareshwari Sriramadasu
      2. patch-4112.txt
        20 kB
        Amareshwari Sriramadasu
      3. patch-4112.txt
        29 kB
        Amareshwari Sriramadasu
      4. patch-4112.txt
        35 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12389987/patch-4112.txt
          against trunk revision 695604.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3262/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12389987/patch-4112.txt against trunk revision 695604. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3262/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Is there any reason that TestHighRAMJobs fails? It also failed in my machine.

          Due to HADOOP-4129

          Show
          Amareshwari Sriramadasu added a comment - Is there any reason that TestHighRAMJobs fails? It also failed in my machine. Due to HADOOP-4129
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > All core tests and contrib tests passed on my mahcine except TestHighRAMJobs.

          Is there any reason that TestHighRAMJobs fails? It also failed in my machine.

          Show
          Tsz Wo Nicholas Sze added a comment - > All core tests and contrib tests passed on my mahcine except TestHighRAMJobs. Is there any reason that TestHighRAMJobs fails? It also failed in my machine.
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Amareshwari!

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Amareshwari!
          Hide
          Amareshwari Sriramadasu added a comment -

          test-patch result on trunk :

               [exec] -1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
               [exec]                         Please justify why no tests are needed for this patch.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]
          

          All core tests and contrib tests passed on my mahcine except TestHighRAMJobs.

          Show
          Amareshwari Sriramadasu added a comment - test-patch result on trunk : [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] All core tests and contrib tests passed on my mahcine except TestHighRAMJobs.
          Hide
          Amareshwari Sriramadasu added a comment -

          The patch removes refactoring done in earlier patch. A boolean parameter, isCleanup, is added to the methods in MapAttempt and ReduceAttempt and old methods are deprecated.

          Show
          Amareshwari Sriramadasu added a comment - The patch removes refactoring done in earlier patch. A boolean parameter, isCleanup, is added to the methods in MapAttempt and ReduceAttempt and old methods are deprecated.
          Hide
          Amar Kamat added a comment -

          I think the refactoring of JobHistory apis should be done as a part of HADOOP-4122.

          Show
          Amar Kamat added a comment - I think the refactoring of JobHistory apis should be done as a part of HADOOP-4122 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12389804/patch-4112.txt
          against trunk revision 693705.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3229/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3229/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3229/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3229/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12389804/patch-4112.txt against trunk revision 693705. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3229/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3229/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3229/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3229/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          test-patch result on the trunk:

               [exec] -1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
               [exec]                         Please justify why no tests are needed for this patch.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]
               [exec]
          

          All unit tests except TestHighRAMJobs passed on my machine. TestHighRAMJobs failure is due to HADOOP-4129

          Show
          Amareshwari Sriramadasu added a comment - test-patch result on the trunk: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] All unit tests except TestHighRAMJobs passed on my machine. TestHighRAMJobs failure is due to HADOOP-4129
          Hide
          Amareshwari Sriramadasu added a comment -

          Here is patch removing CleanupAttempt introduced in earlier patch. The cleanup tasks are logged with CLEANUP as TASK_TYPE. The patch adds the following methods to TaskAttempt

              public static void logStarted(TaskAttemptID taskAttemptId, long startTime,  String hostName, boolean isMap, boolean isCleanup) 
              public static void logFinished(TaskAttemptID taskAttemptId, long finishTime,  String hostName, boolean isMap, boolean isCleanup) 
              public static void logKilled(TaskAttemptID taskAttemptId, long timestamp, String hostName, String error, boolean isFailure, boolean isMap, boolean isCleanup) 
          

          The methods in MapAttempt and ReduceAttempt are modified to use the new methods. And JobInProgress's updating history is modified to use the new methods.
          There is no change for the JobTasksParseListener now. And there are no changes in jobhistory.jsp and HistoryViewer from the earlier patch.

          Show
          Amareshwari Sriramadasu added a comment - Here is patch removing CleanupAttempt introduced in earlier patch. The cleanup tasks are logged with CLEANUP as TASK_TYPE. The patch adds the following methods to TaskAttempt public static void logStarted(TaskAttemptID taskAttemptId, long startTime, String hostName, boolean isMap, boolean isCleanup) public static void logFinished(TaskAttemptID taskAttemptId, long finishTime, String hostName, boolean isMap, boolean isCleanup) public static void logKilled(TaskAttemptID taskAttemptId, long timestamp, String hostName, String error, boolean isFailure, boolean isMap, boolean isCleanup) The methods in MapAttempt and ReduceAttempt are modified to use the new methods. And JobInProgress's updating history is modified to use the new methods. There is no change for the JobTasksParseListener now. And there are no changes in jobhistory.jsp and HistoryViewer from the earlier patch.
          Hide
          Amareshwari Sriramadasu added a comment -

          We need not have CleanupAttempt as separate TaskAttempt, making TASK_TYPE as CLEANUP will solve the problem

          Show
          Amareshwari Sriramadasu added a comment - We need not have CleanupAttempt as separate TaskAttempt, making TASK_TYPE as CLEANUP will solve the problem
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12389729/patch-4112.txt
          against trunk revision 693404.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3216/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3216/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3216/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3216/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12389729/patch-4112.txt against trunk revision 693404. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3216/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3216/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3216/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3216/console This message is automatically generated.
          Hide
          Amar Kamat added a comment -

          Changes made to JobHistory and JobInProgress look fine to me.

          Show
          Amar Kamat added a comment - Changes made to JobHistory and JobInProgress look fine to me.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12389725/patch-4112.txt
          against trunk revision 693360.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3214/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3214/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3214/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3214/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12389725/patch-4112.txt against trunk revision 693360. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3214/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3214/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3214/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3214/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Thanks for the review.

          logFailed() and logKilled() differ in one string. I assume you have done just to keep the code consistent. Someday we should collapse both the api's into something like logComplete(). There is a lot of duplicate/redundant code.

          Yes. I raised HADOOP-4122 for the same.

          Why is job-failed log removed? Redundancy?

          Yes. It is a redundant log. Moreover, JobInfo.logFailed log closes the log file. Then the cleanupAttempt logs will be missed, because killJob launches a cleanup task and after the completion of cleanup, job is marked Failed.

          Patch incorporates other comments

          Show
          Amareshwari Sriramadasu added a comment - Thanks for the review. logFailed() and logKilled() differ in one string. I assume you have done just to keep the code consistent. Someday we should collapse both the api's into something like logComplete(). There is a lot of duplicate/redundant code. Yes. I raised HADOOP-4122 for the same. Why is job-failed log removed? Redundancy? Yes. It is a redundant log. Moreover, JobInfo.logFailed log closes the log file. Then the cleanupAttempt logs will be missed, because killJob launches a cleanup task and after the completion of cleanup, job is marked Failed. Patch incorporates other comments
          Hide
          Amar Kamat added a comment -

          Few comments

          JobHistory :
          1) The comment for CleanupAttempt class needs to be changed.
          2) logFailed() and logKilled() differ in one string. I assume you have done just to keep the code consistent. Someday we should collapse both the api's into something like logComplete(). There is a lot of duplicate/redundant code.

          JobInProgress :
          1) Why is job-failed log removed? Redundancy?
          2) I think there are some diffs because of formatting changes

          Note that I have just checked JobHistory.java and JobInProgress.java.

          Show
          Amar Kamat added a comment - Few comments JobHistory : 1) The comment for CleanupAttempt class needs to be changed. 2) logFailed() and logKilled() differ in one string. I assume you have done just to keep the code consistent. Someday we should collapse both the api's into something like logComplete() . There is a lot of duplicate/redundant code. JobInProgress : 1) Why is job-failed log removed? Redundancy? 2) I think there are some diffs because of formatting changes Note that I have just checked JobHistory.java and JobInProgress.java .
          Hide
          Amareshwari Sriramadasu added a comment -

          Here is patch doing the following:
          1. Adds CleanupAttempt to JobHistory. The cleanup tasks are logged in history as CleanupAttempts. And the TASK_TYPE is CLEANUP. This change also fixes ArrayOutOfBoundException in analyzeJob link.
          2. Updates JobTasksParseListener about CleanupAttempt
          3. JobInProgress is modified to log the cleanup task as CleanupAttempt. Also removes an extra JobFailed log from history.
          4. Updates jobhistory.jsp and HistoryViewer.java to show cleanup task.

          Show
          Amareshwari Sriramadasu added a comment - Here is patch doing the following: 1. Adds CleanupAttempt to JobHistory. The cleanup tasks are logged in history as CleanupAttempts. And the TASK_TYPE is CLEANUP. This change also fixes ArrayOutOfBoundException in analyzeJob link. 2. Updates JobTasksParseListener about CleanupAttempt 3. JobInProgress is modified to log the cleanup task as CleanupAttempt. Also removes an extra JobFailed log from history. 4. Updates jobhistory.jsp and HistoryViewer.java to show cleanup task.

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Amar Kamat
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development