Hadoop Common
  1. Hadoop Common
  2. HADOOP-4100

Scheduler.assignTasks should not be dealing with cleanupTask

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-3150 introduced the notion of a cleanupTask. The implementation of the scheduling of the cleanupTask is such that the schedulers have to be aware of the cleanupTask. It would be very nice if schedulers didn't have to be aware of this special task. There is a discussion thread - https://issues.apache.org/jira/browse/HADOOP-3150?focusedCommentId=12628635#action_12628635

      1. patch-4100.txt
        6 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Hide
          Hemanth Yamijala added a comment -

          I am assuming that since this JIRA is filed, further discussions on the approach and implementation of this issue will happen here.

          In the discussion thread mentioned in the description, it is suggested that the new cleanupTask introduced b HADOOP-3150 is scheduled from the framework directly, as opposed to having it scheduled from schedulers. In general, this seems the right approach. So, +1.

          One question I have (and possibly this was indirectly brought up by Matei as well), should the running of the cleanupTask count as slots used by the Job - as this could impact fairness and capacity limits ? From what I've seen, it appears that each Job has atmost 1 cleanup task - correct ? In that case, it might not be that critical to count these for quotas etc. Comments ?

          Show
          Hemanth Yamijala added a comment - I am assuming that since this JIRA is filed, further discussions on the approach and implementation of this issue will happen here. In the discussion thread mentioned in the description, it is suggested that the new cleanupTask introduced b HADOOP-3150 is scheduled from the framework directly, as opposed to having it scheduled from schedulers. In general, this seems the right approach. So, +1. One question I have (and possibly this was indirectly brought up by Matei as well), should the running of the cleanupTask count as slots used by the Job - as this could impact fairness and capacity limits ? From what I've seen, it appears that each Job has atmost 1 cleanup task - correct ? In that case, it might not be that critical to count these for quotas etc. Comments ?
          Hide
          Devaraj Das added a comment -

          Hemanth, that's right. Every job will have at most one cleanup task (multiple attempts might happen due to failures just like a regular task). I don't think it is critical to keep track of this from the point of view of fairness/quotas, etc.

          Show
          Devaraj Das added a comment - Hemanth, that's right. Every job will have at most one cleanup task (multiple attempts might happen due to failures just like a regular task). I don't think it is critical to keep track of this from the point of view of fairness/quotas, etc.
          Hide
          Amareshwari Sriramadasu added a comment -

          Here is a patch which removes the dependency to assign a cleanup task from scheduler. Now, the JobTracker gets cleanup task for the jobs if any and launches them. If there are no cleanup tasks to be launched, it calls TaskScheduler.assignTasks().

          Show
          Amareshwari Sriramadasu added a comment - Here is a patch which removes the dependency to assign a cleanup task from scheduler. Now, the JobTracker gets cleanup task for the jobs if any and launches them. If there are no cleanup tasks to be launched, it calls TaskScheduler.assignTasks().
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12389658/patch-4100.txt
          against trunk revision 693048.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3206/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3206/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3206/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3206/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12389658/patch-4100.txt against trunk revision 693048. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3206/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3206/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3206/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3206/console This message is automatically generated.
          Hide
          Matei Zaharia added a comment -

          Looks good to me.

          Show
          Matei Zaharia added a comment - Looks good to me.
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Amareshwari!

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Amareshwari!
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #600 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/600/ )

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Devaraj Das
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development