Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5785

Derive heap size or mapreduce.*.memory.mb automatically

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: mr-am, task
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change

      Description

      Currently users have to set 2 memory-related configs per Job / per task type. One first chooses some container size map reduce.*.memory.mb and then a corresponding maximum Java heap size Xmx < map reduce.*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be

      • allocating big containers whereas the JVM will only use the default -Xmx200m.
      • allocating small containers that will OOM because Xmx is too high.

      With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user.

      1. MAPREDUCE-5785.v01.patch
        10 kB
        Gera Shegalov
      2. MAPREDUCE-5785.v02.patch
        26 kB
        Gera Shegalov
      3. MAPREDUCE-5785.v03.patch
        25 kB
        Gera Shegalov
      4. mr-5785-4.patch
        25 kB
        Karthik Kambatla
      5. mr-5785-5.patch
        24 kB
        Karthik Kambatla
      6. mr-5785-6.patch
        24 kB
        Karthik Kambatla
      7. mr-5785-7.patch
        23 kB
        Karthik Kambatla
      8. mr-5785-8.patch
        23 kB
        Karthik Kambatla
      9. mr-5785-9.patch
        23 kB
        Karthik Kambatla

        Issue Links

          Activity

          Gera Shegalov created issue -
          Gera Shegalov made changes -
          Field Original Value New Value
          Description Currently users have to set 2 memory-related configs for per Job / per task type. One fist choses some container size mapreduce.*.memory.mb and then a corresponding Xmx < mapreduce.*.memory.mb to make sure that the JVM with the user code heap, and its native memory do not exceed this limit. If one forgets to tune Xmx, MR-AM might be allocating big containers whereas the JVM will only use the default -Xmx200m.

          With this JIRA, we propose to set Xmx automatically base on an empirical ratio that can be adjusted. Xmx is not changed if provided by the user.
          Currently users have to set 2 memory-related configs for per Job / per task type. One fist choses some container size mapreduce.*.memory.mb and then a corresponding Xmx < mapreduce.*.memory.mb to make sure that the JVM with the user code heap, and its native memory do not exceed this limit. If one forgets to tune Xmx, MR-AM might be allocating big containers whereas the JVM will only use the default -Xmx200m.

          With this JIRA, we propose to set Xmx automatically base on an empirical ratio that can be adjusted. Xmx is not changed automaically if provided by the user.
          Gera Shegalov made changes -
          Remote Link This issue links to "MAPREDUCE-5785 (Web Link)" [ 14532 ]
          Gera Shegalov made changes -
          Attachment MAPREDUCE-5785.v01.patch [ 12633538 ]
          Gera Shegalov made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Gera Shegalov made changes -
          Description Currently users have to set 2 memory-related configs for per Job / per task type. One fist choses some container size mapreduce.*.memory.mb and then a corresponding Xmx < mapreduce.*.memory.mb to make sure that the JVM with the user code heap, and its native memory do not exceed this limit. If one forgets to tune Xmx, MR-AM might be allocating big containers whereas the JVM will only use the default -Xmx200m.

          With this JIRA, we propose to set Xmx automatically base on an empirical ratio that can be adjusted. Xmx is not changed automaically if provided by the user.
          Currently users have to set 2 memory-related configs per Job / per task type. One first chooses some container size mapreduce.*.memory.mb and then a corresponding maximum Java heap size Xmx < mapreduce.*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be
          - allocating big containers whereas the JVM will only use the default -Xmx200m.
          - allocating small containers that will OOM because Xmx is too high.

          With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user.
          Gera Shegalov made changes -
          Description Currently users have to set 2 memory-related configs per Job / per task type. One first chooses some container size mapreduce.*.memory.mb and then a corresponding maximum Java heap size Xmx < mapreduce.*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be
          - allocating big containers whereas the JVM will only use the default -Xmx200m.
          - allocating small containers that will OOM because Xmx is too high.

          With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user.
          Currently users have to set 2 memory-related configs per Job / per task type. One first chooses some container size map reduce.\*.memory.mb and then a corresponding maximum Java heap size Xmx < map reduce.\*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be
          - allocating big containers whereas the JVM will only use the default -Xmx200m.
          - allocating small containers that will OOM because Xmx is too high.

          With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user.
          Gera Shegalov made changes -
          Attachment MAPREDUCE-5785.v02.patch [ 12636218 ]
          Gera Shegalov made changes -
          Summary Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb Derive task attempt JVM max heap size and io.sort.mb automatically from mapreduce.*.memory.mb
          Gera Shegalov made changes -
          Hadoop Flags Incompatible change [ 10342 ]
          Gera Shegalov made changes -
          Attachment MAPREDUCE-5785.v03.patch [ 12644318 ]
          Gera Shegalov made changes -
          Link This issue is related to MAPREDUCE-5892 [ MAPREDUCE-5892 ]
          Rohini Palaniswamy made changes -
          Link This issue relates to TEZ-699 [ TEZ-699 ]
          Karthik Kambatla made changes -
          Attachment mr-5785-4.patch [ 12682773 ]
          Karthik Kambatla made changes -
          Attachment mr-5785-5.patch [ 12682912 ]
          Karthik Kambatla made changes -
          Summary Derive task attempt JVM max heap size and io.sort.mb automatically from mapreduce.*.memory.mb Derive heap size or mapreduce.*.memory.mb automatically
          Karthik Kambatla made changes -
          Attachment mr-5785-6.patch [ 12682982 ]
          Sandy Ryza made changes -
          Target Version/s 2.4.0 [ 12326141 ] 3.0.0 [ 12320355 ]
          Karthik Kambatla made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Incompatible change [ 10342 ] Incompatible change,Reviewed [ 10342, 10343 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Resolution Fixed [ 1 ]
          Gera Shegalov made changes -
          Remote Link This issue links to "MAPREDUCE-5785 (Web Link)" [ 14532 ]
          Karthik Kambatla made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Karthik Kambatla made changes -
          Attachment mr-5785-7.patch [ 12690903 ]
          Karthik Kambatla made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Hadoop Flags Incompatible change,Reviewed [ 10342, 10343 ] Incompatible change [ 10342 ]
          Karthik Kambatla made changes -
          Assignee Gera Shegalov [ jira.shegalov ] Karthik Kambatla [ kasha ]
          Karthik Kambatla made changes -
          Attachment mr-5785-8.patch [ 12693106 ]
          Karthik Kambatla made changes -
          Attachment mr-5785-9.patch [ 12693467 ]
          Gera Shegalov made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Assignee Karthik Kambatla [ kasha ] Gera Shegalov [ jira.shegalov ]
          Resolution Fixed [ 1 ]
          Gera Shegalov made changes -
          Link This issue breaks MAPREDUCE-6223 [ MAPREDUCE-6223 ]
          Masatake Iwasaki made changes -
          Link This issue relates to MAPREDUCE-6234 [ MAPREDUCE-6234 ]
          Hao Xia made changes -
          Link This issue relates to MAPREDUCE-6343 [ MAPREDUCE-6343 ]

            People

            • Assignee:
              Gera Shegalov
              Reporter:
              Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development