Hadoop Common
  1. Hadoop Common
  2. HADOOP-4731

Job is not removed from the waiting jobs queue upon completion.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.19.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When a job completes, the JobQueueManager removes this job from it's internal structures, which is a queue of the running jobs and waiting jobs. While the removal from the running jobs happens correctly, the removal from the waiting jobs does not happen, if the job is still in the PREP state. This was found while implementing HADOOP-4035, as described by Vinod here. The change was introduced in HADOOP-4471, which I should have caught in the review. Sorry about that !

      1. HADOOP-4731-v1.patch
        4 kB
        Amar Kamat
      2. HADOOP-4731-v1.1.patch
        4 kB
        Amar Kamat

        Activity

        Hide
        Hemanth Yamijala added a comment -

        Possibly there is no test case that completes a job without actually running it, which is why this issue was never found. So, I would recommend introducing a test that kills a queued job, and makes sure it is not there in the list.

        Show
        Hemanth Yamijala added a comment - Possibly there is no test case that completes a job without actually running it, which is why this issue was never found. So, I would recommend introducing a test that kills a queued job, and makes sure it is not there in the list.
        Hide
        Amar Kamat added a comment -

        Attaching patch that fixes the issue. Result of test-patch on my box

        [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec] 
             [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
        
        Show
        Amar Kamat added a comment - Attaching patch that fixes the issue. Result of test-patch on my box [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
        Hide
        Hemanth Yamijala added a comment -

        In testJobCompletion(),

        taskTrackerManager.finishTask("tt1", tasks.get(0).getTaskID().toString(), 
                                          fjob1);
        

        should be

        taskTrackerManager.finishTask("tt1", tasks.get(0).getTaskID().toString(), 
                                          fjob2);
        
        Show
        Hemanth Yamijala added a comment - In testJobCompletion() , taskTrackerManager.finishTask( "tt1" , tasks.get(0).getTaskID().toString(), fjob1); should be taskTrackerManager.finishTask( "tt1" , tasks.get(0).getTaskID().toString(), fjob2);
        Hide
        Amar Kamat added a comment -

        Incorporated Hemanth's comments.

        Show
        Amar Kamat added a comment - Incorporated Hemanth's comments.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12394834/HADOOP-4731-v1.1.patch
        against trunk revision 721415.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3672/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12394834/HADOOP-4731-v1.1.patch against trunk revision 721415. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3672/console This message is automatically generated.
        Hide
        Amar Kamat added a comment -

        HADOOP-4513 fixes this issue for hadoop-0.20. The attached patch fixes the bug in hadoop-0.19 and also provides a testcase. HADOOP-4513 has removed JobQueuesManager.getWaitingJobQueue() api and hence a new test case needs to be submitted for hadoop-0.20.

        Show
        Amar Kamat added a comment - HADOOP-4513 fixes this issue for hadoop-0.20. The attached patch fixes the bug in hadoop-0.19 and also provides a testcase. HADOOP-4513 has removed JobQueuesManager.getWaitingJobQueue() api and hence a new test case needs to be submitted for hadoop-0.20.
        Hide
        Amar Kamat added a comment -

        Tested the patch with hadoop-0.19.
        test-patch :

        [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
        

        Running ant-test now.

        Show
        Amar Kamat added a comment - Tested the patch with hadoop-0.19. test-patch : [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. Running ant-test now.
        Hide
        Amar Kamat added a comment -

        ant-test passed on my local box.

        Show
        Amar Kamat added a comment - ant-test passed on my local box.
        Hide
        Hemanth Yamijala added a comment -

        I just committed this to the Hadoop 0.19 branch. Thanks, Amar.

        As noted above, this fix is required only for the Hadoop 0.19 branch. In Hadoop 0.20, other changes to the code which are committed (HADOOP-4513 specifically) make this issue non-existent.

        Show
        Hemanth Yamijala added a comment - I just committed this to the Hadoop 0.19 branch. Thanks, Amar. As noted above, this fix is required only for the Hadoop 0.19 branch. In Hadoop 0.20, other changes to the code which are committed ( HADOOP-4513 specifically) make this issue non-existent.

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Hemanth Yamijala
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development