Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6925

CLONE - Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: applicationmaster, client, task
    • Labels:
      None
    • Target Version/s:

      Description

      Currently, counter limits "mapreduce.job.counters.*" handled by org.apache.hadoop.mapreduce.counters.Limits are initialized asymmetrically: on the client side, and on the AM, job.xml is ignored whereas it's taken into account in YarnChild.

      It would be good to make the Limits job-configurable, such that max counters/groups is only increased when needed. With the current Limits implementation relying on static constants, it's going to be challenging for tools that submit jobs concurrently without resorting to class loading isolation.

      The patch that I am uploading is not perfect but demonstrates the issue.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              jira.shegalov Gera Shegalov
              Reporter:
              jira.shegalov Gera Shegalov

              Dates

              • Created:
                Updated:

                Issue deployment