Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1974

FairScheduler can preempt the same task many times

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.21.1, 0.22.0, 0.23.0
    • Fix Version/s: 0.22.0
    • Component/s: contrib/fair-share
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In FairScheduler.preemptTasks(), tasks are collected from JobInProgress.runningMapCache.
      But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, node and cluster).
      This makes FairScheduler preempt the same task many times.

        Activity

        Scott Chen created issue -
        Scott Chen made changes -
        Field Original Value New Value
        Affects Version/s 0.20.1 [ 12314047 ]
        Component/s contrib/fair-share [ 12312901 ]
        Scott Chen made changes -
        Fix Version/s 0.22.0 [ 12314184 ]
        Affects Version/s 0.21.0 [ 12314045 ]
        Affects Version/s 0.22.0 [ 12314184 ]
        Affects Version/s 0.20.1 [ 12314047 ]
        Scott Chen made changes -
        Attachment MAPREDUCE-1974.txt [ 12450748 ]
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Matei Zaharia added a comment -

        The patch looks good to me. Is there any way to test this? (Did you at least try to print the list before and see that a task is in there multiple times?)

        Show
        Matei Zaharia added a comment - The patch looks good to me. Is there any way to test this? (Did you at least try to print the list before and see that a task is in there multiple times?)
        Hide
        Scott Chen added a comment -

        Hi Matei,
        I see multiple same preemption in the log. I think I should add a test here to verify that.

        Scott

        Show
        Scott Chen added a comment - Hi Matei, I see multiple same preemption in the log. I think I should add a test here to verify that. Scott
        Scott Chen made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Matei Zaharia added a comment -

        Did you guys fix this one at Facebook? It would be nice to put it into trunk and 0.21.1 too.

        Show
        Matei Zaharia added a comment - Did you guys fix this one at Facebook? It would be nice to put it into trunk and 0.21.1 too.
        Hide
        Scott Chen added a comment -

        Hey Matei,

        Sorry for not responding on this for a while.
        We have put this in production for a long while and it fixes the problem.
        I think it is OK to commit this.

        Show
        Scott Chen added a comment - Hey Matei, Sorry for not responding on this for a while. We have put this in production for a long while and it fixes the problem. I think it is OK to commit this.
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.21.1 [ 12315272 ]
        Affects Version/s 0.23.0 [ 12315570 ]
        Fix Version/s 0.21.1 [ 12315272 ]
        Fix Version/s 0.23.0 [ 12315570 ]
        Nigel Daley made changes -
        Fix Version/s 0.22.0 [ 12314184 ]
        Fix Version/s 0.21.1 [ 12315272 ]
        Fix Version/s 0.23.0 [ 12315570 ]
        Hide
        Joydeep Sen Sarma added a comment -

        +1. this seems like a no-brainer!

        Show
        Joydeep Sen Sarma added a comment - +1. this seems like a no-brainer!
        Hide
        Scott Chen added a comment -
        
             [exec]
             [exec] BUILD SUCCESSFUL
             [exec] Total time: 2 minutes 15 seconds
             [exec]
             [exec]
             [exec]
             [exec]
             [exec] +1 overall.
             [exec]
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec]
             [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
             [exec]
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec]
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec]
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
             [exec]
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
             [exec]
             [exec]     +1 system test framework.  The patch passed system test framework compile.
             [exec]
             [exec]
             [exec]
             [exec]
             [exec] ======================================================================
             [exec] ======================================================================
             [exec]     Finished build.
             [exec] ======================================================================
             [exec] ======================================================================
        
        Show
        Scott Chen added a comment - [exec] [exec] BUILD SUCCESSFUL [exec] Total time: 2 minutes 15 seconds [exec] [exec] [exec] [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ======================================================================
        Hide
        Scott Chen added a comment -

        I posted the result on the wrong patch. It should not have a +1 on test included.
        Will run this again.

        Show
        Scott Chen added a comment - I posted the result on the wrong patch. It should not have a +1 on test included. Will run this again.
        Hide
        Scott Chen added a comment -

        test-patch result

        [exec]
        [exec]
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
        [exec] Please justify why no new tests are needed for this patch.
        [exec] Also please list what manual steps were performed to verify this patch.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] +1 system test framework. The patch passed system test framework compile.
        [exec]
        [exec]
        [exec]
        [exec]
        [exec] ======================================================================
        [exec] ======================================================================
        [exec] Finished build.
        [exec] ======================================================================
        [exec] ======================================================================

        Show
        Scott Chen added a comment - test-patch result [exec] [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ======================================================================
        Hide
        Priyo Mustafi added a comment -

        I have tested this as well and all looks good

        Show
        Priyo Mustafi added a comment - I have tested this as well and all looks good
        Hide
        Konstantin Shvachko added a comment -

        I just committed this. Thank you Scott.

        Show
        Konstantin Shvachko added a comment - I just committed this. Thank you Scott.
        Konstantin Shvachko made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.22.0 [ 12314184 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #603 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/603/)
        MAPREDUCE-1974. Fix multiple preemtions of the same task in FairScheduler. Contributed by Scott Chen.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #603 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/603/ ) MAPREDUCE-1974 . Fix multiple preemtions of the same task in FairScheduler. Contributed by Scott Chen.
        Hide
        Scott Chen added a comment -

        Thanks for the help, guys

        Show
        Scott Chen added a comment - Thanks for the help, guys
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
        Konstantin Shvachko made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        2h 22m 1 Scott Chen 26/Aug/10 22:59
        Open Open Patch Available Patch Available
        124d 16h 11m 2 Scott Chen 29/Nov/10 18:09
        Patch Available Patch Available Resolved Resolved
        67d 4h 48m 1 Konstantin Shvachko 04/Feb/11 22:57
        Resolved Resolved Closed Closed
        310d 7h 22m 1 Konstantin Shvachko 12/Dec/11 06:19

          People

          • Assignee:
            Scott Chen
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development