Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4768

getAvailablePhysicalMemorySize can be inaccurate on linux

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2, 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Environment:

      Linux

    • Target Version/s:

      Description

      Algorithm currently uses "MemFree" + "Inactive" from /proc/meminfo

      "Inactive" may not be a very good indication of how much memory can be readily freed because it contains both:

      • Pages mapped with MAP_SHARED|MAP_ANONYMOUS (regardless of whether they're being actively accessed or not. Unclear to me why this is the case...)
      • Pages mapped MAP_PRIVATE|MAP_ANONYMOUS that have not been accessed recently

      Both of these types of pages probably shouldn't be considered "Available".

      "Inactive(file)" would seem more accurate but it's not available in all kernel versions. To keep things simple, maybe just use "Inactive(file)" if available, otherwise fallback to "Inactive".

        Attachments

        1. YARN-4768.patch
          9 kB
          Nathan Roberts

          Issue Links

            Activity

              People

              • Assignee:
                nroberts Nathan Roberts
                Reporter:
                nroberts Nathan Roberts
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: