Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2, nodemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Resource limits are now expressed and enforced in terms of physical memory, rather than virtual memory. The virtual memory limit is set as a configurable multiple of the physical limit. The NodeManager's memory usage is now configured in units of MB rather than GB.
      Show
      Resource limits are now expressed and enforced in terms of physical memory, rather than virtual memory. The virtual memory limit is set as a configurable multiple of the physical limit. The NodeManager's memory usage is now configured in units of MB rather than GB.

      Description

      Currently, the memory resources requested for a container limit the amount of virtual memory used by the container. On my test clusters, at least, Java processes take up nearly twice as much vmem as pmem - a Java process running with -Xmx500m uses 935m of vmem and only about 560m of pmem.

      This will force admins to either under-utilize available physical memory, or oversubscribe it by configuring the available resources on a TT to be larger than the true amount of physical RAM.

      Instead, I would propose that the resource limit apply to pmem, and allow the admin to configure a "vmem overcommit ratio" which sets the vmem limit as a function of pmem limit.

      1. mr-3205.txt
        36 kB
        Todd Lipcon
      2. mr-3205.txt
        30 kB
        Todd Lipcon
      3. mr-3205.txt
        29 kB
        Todd Lipcon
      4. mr-3205.txt
        29 kB
        Todd Lipcon
      5. mr-3205.txt
        21 kB
        Todd Lipcon
      6. mr-3205.txt
        20 kB
        Todd Lipcon

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development