Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2705

tasks localized and launched serially by TaskLauncher - causing other tasks to be delayed

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.20.205.0, 0.23.0
    • Component/s: tasktracker
    • Labels:
      None

      Description

      The current TaskLauncher serially launches new tasks one at a time. During the launch it does the localization and then starts the map/reduce task. This can cause any other tasks to be blocked waiting for the current task to be localized and started. In some instances we have seen a task that has a large file to localize (1.2MB) block another task for about 40 minutes. This particular task being blocked was a cleanup task which caused the job to be delayed finishing for the 40 minutes.

      1. MAPREDUCE-2705-branch20.patch
        10 kB
        Thomas Graves
      2. MAPREDUCE-2705-trunk.patch
        9 kB
        Thomas Graves

        Activity

        Thomas Graves created issue -
        Hide
        Thomas Graves added a comment -

        Note 1.2MB should be 1.2GB.

        Show
        Thomas Graves added a comment - Note 1.2MB should be 1.2GB.
        Thomas Graves made changes -
        Field Original Value New Value
        Fix Version/s 0.20.205.0 [ 12316391 ]
        Fix Version/s 0.23.0 [ 12315570 ]
        Affects Version/s 0.23.0 [ 12315570 ]
        Hide
        Thomas Graves added a comment -

        patch for branch-0.20-security

        Show
        Thomas Graves added a comment - patch for branch-0.20-security
        Thomas Graves made changes -
        Attachment MAPREDUCE-2705-branch20.patch [ 12487205 ]
        Hide
        Thomas Graves added a comment -

        branch 20 security test-patch results:

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
        [exec]
        [exec]
        [exec]

        Show
        Thomas Graves added a comment - branch 20 security test-patch results: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] [exec]
        Hide
        Thomas Graves added a comment -

        The solution implemented is to have it start a new thread to do the actual localization and task launching for that particular task, that way it doesn't block other tasks behind it if the localization of one task is taking a long time.

        Show
        Thomas Graves added a comment - The solution implemented is to have it start a new thread to do the actual localization and task launching for that particular task, that way it doesn't block other tasks behind it if the localization of one task is taking a long time.
        Hide
        Thomas Graves added a comment -

        patch for trunk

        Show
        Thomas Graves added a comment - patch for trunk
        Thomas Graves made changes -
        Attachment MAPREDUCE-2705-trunk.patch [ 12487212 ]
        Thomas Graves made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Mahadev konar made changes -
        Affects Version/s 0.23.0 [ 12315570 ]
        Thomas Graves made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Thomas Graves made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Aaron T. Myers added a comment -

        Hey Tom, simply changing the JIRA status to be PA -> Open -> PA won't retrigger Hudson. You either need to upload another attachment (could be the same contents) or get someone with Hudson access to retrigger the build for you. I just did the latter. You can watch the build here: https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/489/

        Show
        Aaron T. Myers added a comment - Hey Tom, simply changing the JIRA status to be PA -> Open -> PA won't retrigger Hudson. You either need to upload another attachment (could be the same contents) or get someone with Hudson access to retrigger the build for you. I just did the latter. You can watch the build here: https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/489/
        Hide
        Thomas Graves added a comment -

        Hey Aaron, thanks for the info and kicking the build.

        Show
        Thomas Graves added a comment - Hey Aaron, thanks for the info and kicking the build.
        Hide
        Devaraj Das added a comment -

        +1

        Show
        Devaraj Das added a comment - +1
        Hide
        Devaraj Das added a comment -

        Committed the patch in branch-0.20-security. Thomas, there are a bunch of test failures in the build for trunk. Can you please confirm these are harmless.

        Show
        Devaraj Das added a comment - Committed the patch in branch-0.20-security. Thomas, there are a bunch of test failures in the build for trunk. Can you please confirm these are harmless.
        Hide
        Thomas Graves added a comment -

        when I first submitted the patch the tests on trunk were already broken. For instance see https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/472/ which is for a different jira. Let me rerun now just to double check and I'll update you on results.

        Show
        Thomas Graves added a comment - when I first submitted the patch the tests on trunk were already broken. For instance see https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/472/ which is for a different jira. Let me rerun now just to double check and I'll update you on results.
        Hide
        Thomas Graves added a comment -

        reconfirmed those failures aren't from this patch. All tests that failed in the hudson build (https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/489) either passed when I run them or also failed on trunk without this patch.

        Thanks for the review/commit!

        Show
        Thomas Graves added a comment - reconfirmed those failures aren't from this patch. All tests that failed in the hudson build ( https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/489 ) either passed when I run them or also failed on trunk without this patch. Thanks for the review/commit!
        Hide
        Devaraj Das added a comment -

        Committed this to trunk. Thanks, Thomas.

        Show
        Devaraj Das added a comment - Committed this to trunk. Thanks, Thomas.
        Devaraj Das made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #760 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/760/)
        MAPREDUCE-2705. Permits parallel multiple task launches. Contributed by Thomas Graves.
        MAPREDUCE-2705. Permits parallel multiple task launches. Contributed by Thomas Graves.

        ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153720
        Files :

        • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestTaskLauncherThreaded.java

        ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153717
        Files :

        • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskTracker.java
        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #760 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/760/ ) MAPREDUCE-2705 . Permits parallel multiple task launches. Contributed by Thomas Graves. MAPREDUCE-2705 . Permits parallel multiple task launches. Contributed by Thomas Graves. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153720 Files : /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestTaskLauncherThreaded.java ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153717 Files : /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskTracker.java /hadoop/common/trunk/mapreduce/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #751 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/751/)
        MAPREDUCE-2705. Permits parallel multiple task launches. Contributed by Thomas Graves.
        MAPREDUCE-2705. Permits parallel multiple task launches. Contributed by Thomas Graves.

        ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153720
        Files :

        • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestTaskLauncherThreaded.java

        ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153717
        Files :

        • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskTracker.java
        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #751 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/751/ ) MAPREDUCE-2705 . Permits parallel multiple task launches. Contributed by Thomas Graves. MAPREDUCE-2705 . Permits parallel multiple task launches. Contributed by Thomas Graves. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153720 Files : /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestTaskLauncherThreaded.java ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153717 Files : /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskTracker.java /hadoop/common/trunk/mapreduce/CHANGES.txt
        Hide
        Matt Foley added a comment -

        Closed upon release of 0.20.205.0

        Show
        Matt Foley added a comment - Closed upon release of 0.20.205.0
        Matt Foley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Thomas Graves
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development