[SPARK-28577] Ensure executorMemoryHead requested value not less than MEMORY_OFFHEAP_SIZE when MEMORY_OFFHEAP_ENABLED is true

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core, YARN
• Labels: release-notes
• Docs Text: On YARN, The off heap memory size is now separately included for the user in the container size it requests from YARN. Previously you had to add that into the overhead memory you requested, that is no longer needed.
• Assignee: Yang Jie
• Reporter: Yang Jie
• Created: 31/Jul/19 06:13
• Updated: 14/Jul/21 07:25
• Resolved: 04/Sep/19 14:01

Description

If MEMORY_OFFHEAP_ENABLED is true, we should ensure executorOverheadMemory not less than MEMORY_OFFHEAP_SIZE, otherwise the memory resource requested for executor may be not enough.

 

 

If MEMORY_OFFHEAP_ENABLED is true, add MEMORY_OFFHEAP_SIZE to resource requested for executor to ensure instance has enough memory to use.

Issue Links

• links to: GitHub Pull Request #25309