Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      Users are often unaware of localization cost that their jobs incur. To measure effectiveness of localization caches it is necessary to expose the overhead in the form of user-visible metrics. The purpose of this JIRA is to compliment YARN-1529. While YARN-1529 attempts to provide a cluster-wide view to cluster admins, this JIRA focuses on exposing the localization overhead on per-job basis to the job owner/user.

      1. MAPREDUCE-5696.v02.patch
        12 kB
        Gera Shegalov
      2. MAPREDUCE-5696.v01.patch
        20 kB
        Gera Shegalov
      3. LocalizationCounters.png
        139 kB
        Gera Shegalov

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12620348/MAPREDUCE-5696.v01.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4282//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12620348/MAPREDUCE-5696.v01.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4282//console This message is automatically generated.
          Hide
          Hitesh Shah added a comment -

          The introduction of localization counters in the env is akin to introducing a new API in YARN. Could you split this jira out into 2. One in YARN for the YARN changes where the new API/interface is introduced and this jira could be leveraged for the MR specific changes.

          Show
          Hitesh Shah added a comment - The introduction of localization counters in the env is akin to introducing a new API in YARN. Could you split this jira out into 2. One in YARN for the YARN changes where the new API/interface is introduced and this jira could be leveraged for the MR specific changes.
          Hide
          Gera Shegalov added a comment -

          Hitesh Shah, I think we can move these few YARN changes to YARN-1529 instead of creating a new JIRA. I'll upload new patch versions.

          Show
          Gera Shegalov added a comment - Hitesh Shah , I think we can move these few YARN changes to YARN-1529 instead of creating a new JIRA. I'll upload new patch versions.
          Hide
          Gera Shegalov added a comment -

          v02 patch without YARN-changes.

          Show
          Gera Shegalov added a comment - v02 patch without YARN-changes.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12621231/MAPREDUCE-5696.v02.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4299//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12621231/MAPREDUCE-5696.v02.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4299//console This message is automatically generated.

            People

            • Assignee:
              Gera Shegalov
              Reporter:
              Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development