[SPARK-50421] Executor memory config is incorrect when multiple stage level resource profiles works

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.3, 3.2.4, 3.3.4, 3.4.4, 3.5.3
• Fix Version/s: 4.0.0, 3.5.4
• Component/s: Scheduler, YARN
• Labels: pull-request-available
• Flags: Important
• Assignee: Terry Wang
• Reporter: Terry Wang
• Created: 26/Nov/24 03:35
• Updated: 5 days ago 06:13
• Resolved: 05/Dec/24 16:28

Description

https://issues.apache.org/jira/browse/SPARK-27495 has imported a great feature to help adjust stage's level resource profile(abbv rp), we are observing some config is not as expected when multiple resource profile works.
When one stage assigned rp2 different with default rp, the `UnifiedMemoryManager` will still get old conf as the default rp, which will cause maxOffHeapMemory is still the old value!

protected[this] val maxOffHeapMemory = conf.get(MEMORY_OFFHEAP_SIZE)

Issue Links

• Blocked: SPARK-50419 Executor memory config is incorrect when multiple stage level resource profiles works
• Blocked: SPARK-50420 Executor memory config is incorrect when multiple stage level resource profiles works
• links to: GitHub Pull Request #48963
• links to: GitHub Pull Request #49090