[SPARK-20150] Add permsize statistics for worker memory which may be very useful for the memory usage assessment

• Type: Wish
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.0.2
• Component/s: Web UI
• Assignee: Unassigned
• Reporter: Jinhua Fu
• Created: 30/Mar/17 02:25
• Updated: 30/Mar/17 09:29
• Resolved: 30/Mar/17 09:29

Description

It seems worker memory only be assigned to executor heap which is usually not enough for estimating the whole clauster memory usage,especially when memory becomes a bottleneck of the clauster.In many case,we found a executor's real memory usage was much larger than its heap size which make me have to check for every application's real memory expenditure.

This can be improved by adding a mechanism for Non-Heap(permsize) statistics,only shown for extra memory usage which has no effect on the current worker memory allocation and statistics.The permsize can be obtained easily from executor java options.