Hadoop Common
  1. Hadoop Common
  2. HADOOP-5278

Finish time of a TIP is incorrectly logged to the jobhistory upon jobtracker restart

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Upon recovery, the jobtracker replays the attempt information to the JIP. Upon seeing the attempt-end info, the JIP completes the TIP (which sets the finish time to current time) and logs it to the new jobhistory file (the .recover file). After seeing the task(tip) finish line, the recovery manager changes the finish time to correct finish time but the new jobhistory file still contains the wrong value for the finish time seen while completing the TIP.

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #778 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/778/)
          . Moved the comment about 5278 to branch-0.20 section.

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #778 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/778/ ) . Moved the comment about 5278 to branch-0.20 section.
          Hide
          Devaraj Das added a comment -

          I committed this to the 0.20 branch as well.

          Show
          Devaraj Das added a comment - I committed this to the 0.20 branch as well.
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #766 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/766/ )
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Amar!

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Amar!
          Hide
          Amar Kamat added a comment -

          Result of test-patch :

          [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
               [exec]                         Please justify why no tests are needed for this patch.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
          
          Show
          Amar Kamat added a comment - Result of test-patch : [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Ravi Gummadi added a comment -

          I tested this patch with HADOOP-4191 and I see that correct value of finish time of task(tip) is logged to history file with this patch.

          +1

          Show
          Ravi Gummadi added a comment - I tested this patch with HADOOP-4191 and I see that correct value of finish time of task(tip) is logged to history file with this patch. +1
          Hide
          Amar Kamat added a comment -

          Attaching a patch the add another update line that overwrites the wrong finish-time with the correct finish-time.

          Show
          Amar Kamat added a comment - Attaching a patch the add another update line that overwrites the wrong finish-time with the correct finish-time.
          Hide
          Amar Kamat added a comment -

          Ravi actually found this while testing HADOOP-4191.

          Show
          Amar Kamat added a comment - Ravi actually found this while testing HADOOP-4191 .

            People

            • Assignee:
              Amar Kamat
              Reporter:
              Amar Kamat
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development