Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5785

Derive heap size or mapreduce.*.memory.mb automatically

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: mr-am, task
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change

      Description

      Currently users have to set 2 memory-related configs per Job / per task type. One first chooses some container size map reduce.*.memory.mb and then a corresponding maximum Java heap size Xmx < map reduce.*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be

      • allocating big containers whereas the JVM will only use the default -Xmx200m.
      • allocating small containers that will OOM because Xmx is too high.

      With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user.

      1. mr-5785-9.patch
        23 kB
        Karthik Kambatla
      2. mr-5785-8.patch
        23 kB
        Karthik Kambatla
      3. mr-5785-7.patch
        23 kB
        Karthik Kambatla
      4. mr-5785-6.patch
        24 kB
        Karthik Kambatla
      5. mr-5785-5.patch
        24 kB
        Karthik Kambatla
      6. mr-5785-4.patch
        25 kB
        Karthik Kambatla
      7. MAPREDUCE-5785.v03.patch
        25 kB
        Gera Shegalov
      8. MAPREDUCE-5785.v02.patch
        26 kB
        Gera Shegalov
      9. MAPREDUCE-5785.v01.patch
        10 kB
        Gera Shegalov

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Gera Shegalov
              Reporter:
              Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development