Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2787

MR-279: Performance improvement in running Uber MapTasks

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      The runUberMapTasks() in org.apache.hadoop.mapred.UberTask obtains the local fileSystem and local job configuration for every task attempt. This will have a negative performance impact.

        Activity

        Hide
        Ahmed Radwan added a comment -

        The attached patch fixes the described issue by only creating the FileSystem and Configuration once for all task attempts.

        All mapreduce unit tests ran successfully.

        Show
        Ahmed Radwan added a comment - The attached patch fixes the described issue by only creating the FileSystem and Configuration once for all task attempts. All mapreduce unit tests ran successfully.
        Hide
        Arun C Murthy added a comment -

        Ahmed - UberTask isn't used anymore in MR-279. The AM uses a different LocalContainerLauncher and LocalContainerAllocator in-lieu of UberTask. Please do take a look at the newer impl. Thanks.

        Should we close this as won't fix?

        Show
        Arun C Murthy added a comment - Ahmed - UberTask isn't used anymore in MR-279. The AM uses a different LocalContainerLauncher and LocalContainerAllocator in-lieu of UberTask. Please do take a look at the newer impl. Thanks. Should we close this as won't fix?
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Just to give more clarity to what Arun said: even though there are patches on MR-279 branch which implement the uber-task feature for the classic runtime (JT/TT), those patches are not going to be ported to trunk when we merge MR-279 to trunk. OTOH, the uber-task feature for yarn+MR runtime is implemented via LocalContainerAllocator and LocalContainerLauncher which is what you should look at.

        Should we close this as won't fix?

        +1. Ahmed, please close this once you are convinced. Thanks!

        Show
        Vinod Kumar Vavilapalli added a comment - Just to give more clarity to what Arun said: even though there are patches on MR-279 branch which implement the uber-task feature for the classic runtime (JT/TT), those patches are not going to be ported to trunk when we merge MR-279 to trunk. OTOH, the uber-task feature for yarn+MR runtime is implemented via LocalContainerAllocator and LocalContainerLauncher which is what you should look at. Should we close this as won't fix? +1. Ahmed, please close this once you are convinced. Thanks!
        Hide
        Ahmed Radwan added a comment -

        Thanks Arun and Vinod for the clarification. I am closing the ticket.

        Show
        Ahmed Radwan added a comment - Thanks Arun and Vinod for the clarification. I am closing the ticket.

          People

          • Assignee:
            Ahmed Radwan
            Reporter:
            Ahmed Radwan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development