Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5768

Spark UI Shows incorrect memory under Yarn

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 1.2.1
    • Fix Version/s: 1.4.1, 1.5.0
    • Component/s: Web UI
    • Labels:
      None
    • Environment:

      Centos 6

    • Target Version/s:

      Description

      I am running Spark on Yarn with 2 executors. The executors are running on separate physical machines.

      I have spark.executor.memory set to '40g'. This is because I want to have 40g of memory used on each machine. I have one executor per machine.

      When I run my application I see from 'top' that both my executors are using the full 40g of memory I allocated to them.

      The 'Executors' tab in the Spark UI shows something different. It shows the memory used as a total of 20GB per executor e.g. x / 20.3GB. This makes it look like I only have 20GB available per executor when really I have 40GB available.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rekhajoshm Rekha Joshi
                Reporter:
                alrocks47 Al M
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: