Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-566

Add a misconfiguration diagnostic or highlight mismatched resource.mb vs -Xmx

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      It is possible to misconfigure a pure-java MR application by setting mapreduce resource size to 1536Mb, while leaving the -Xmx value at the default 512Mb.

      The AM UI can track over-all memory usage of an app either via Runtime.maxMemory() or via the process watcher in YARN to report the actual memory used by the application.

      MR does not fail even when the memory -Xmx is set lower than the YARN allocations, but it spills & GCs instead to keep it with in the memory sizes.

      This is sub-optimal and would make sense for YARN to highlight in the AM UI or JHS UI for tuning/debugging purposes.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              gopalv Gopal Vijayaraghavan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: