Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5875

Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently, counter limits "mapreduce.job.counters.*" handled by org.apache.hadoop.mapreduce.counters.Limits are initialized asymmetrically: on the client side, and on the AM, job.xml is ignored whereas it's taken into account in YarnChild.

      It would be good to make the Limits job-configurable, such that max counters/groups is only increased when needed. With the current Limits implementation relying on static constants, it's going to be challenging for tools that submit jobs concurrently without resorting to class loading isolation.

      The patch that I am uploading is not perfect but demonstrates the issue.

        Attachments

        1. MAPREDUCE-5875.v01.patch
          7 kB
          Gera Shegalov
        2. MAPREDUCE-5875.v02.patch
          11 kB
          Gera Shegalov
        3. MAPREDUCE-5875.v03.patch
          17 kB
          Gera Shegalov
        4. MAPREDUCE-5875.v04.patch
          18 kB
          Gera Shegalov
        5. MAPREDUCE-5875.v05.patch
          18 kB
          Gera Shegalov
        6. MAPREDUCE-5875.v06.patch
          20 kB
          Gera Shegalov
        7. MAPREDUCE-5875.v07.patch
          21 kB
          Gera Shegalov
        8. MAPREDUCE-5875.v08.patch
          25 kB
          Gera Shegalov
        9. MAPREDUCE-5875.v09.patch
          17 kB
          Gera Shegalov

          Issue Links

            Activity

              People

              • Assignee:
                jira.shegalov Gera Shegalov
                Reporter:
                jira.shegalov Gera Shegalov
              • Votes:
                0 Vote for this issue
                Watchers:
                20 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: