Details

    • Type: Improvement
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2, nodemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Resource limits are now expressed and enforced in terms of physical memory, rather than virtual memory. The virtual memory limit is set as a configurable multiple of the physical limit. The NodeManager's memory usage is now configured in units of MB rather than GB.
      Show
      Resource limits are now expressed and enforced in terms of physical memory, rather than virtual memory. The virtual memory limit is set as a configurable multiple of the physical limit. The NodeManager's memory usage is now configured in units of MB rather than GB.

      Description

      Currently, the memory resources requested for a container limit the amount of virtual memory used by the container. On my test clusters, at least, Java processes take up nearly twice as much vmem as pmem - a Java process running with -Xmx500m uses 935m of vmem and only about 560m of pmem.

      This will force admins to either under-utilize available physical memory, or oversubscribe it by configuring the available resources on a TT to be larger than the true amount of physical RAM.

      Instead, I would propose that the resource limit apply to pmem, and allow the admin to configure a "vmem overcommit ratio" which sets the vmem limit as a function of pmem limit.

        Attachments

        1. mr-3205.txt
          36 kB
          Todd Lipcon
        2. mr-3205.txt
          30 kB
          Todd Lipcon
        3. mr-3205.txt
          29 kB
          Todd Lipcon
        4. mr-3205.txt
          29 kB
          Todd Lipcon
        5. mr-3205.txt
          21 kB
          Todd Lipcon
        6. mr-3205.txt
          20 kB
          Todd Lipcon

          Activity

            People

            • Assignee:
              tlipcon Todd Lipcon
              Reporter:
              tlipcon Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: