[SPARK-5768] Spark UI Shows incorrect memory under Yarn

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.2.1
• Fix Version/s: 1.4.1, 1.5.0
• Component/s: Web UI

Environment

Centos 6

• Target Version/s: 1.5.0
• Assignee: Rekha Joshi
• Reporter: Al M
• Created: 12/Feb/15 10:37
• Updated: 02/Feb/16 03:38
• Resolved: 25/Jun/15 11:47

Description

I am running Spark on Yarn with 2 executors. The executors are running on separate physical machines.

I have spark.executor.memory set to '40g'. This is because I want to have 40g of memory used on each machine. I have one executor per machine.

When I run my application I see from 'top' that both my executors are using the full 40g of memory I allocated to them.

The 'Executors' tab in the Spark UI shows something different. It shows the memory used as a total of 20GB per executor e.g. x / 20.3GB. This makes it look like I only have 20GB available per executor when really I have 40GB available.

Issue Links

• is duplicated by: SPARK-4388 Add tooltips to explain maxMemory / usedMemory columns in executor UI
• links to: [Github] Pull Request #6972 (rekhajoshm)