[SPARK-46947] Executor memory related initialization should happen after custom DriverPlugin

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Apache Spark
• Reporter: Chao Sun
• Created: 01/Feb/24 23:15
• Updated: 05/Jun/24 07:33
• Resolved: 26/Feb/24 18:17

Description

Today, almost all executor memory related settings, including UnifiedMemoryManager, ResourceProfileManager, etc, are initialized before a custom Driver plugin is initialized. This means it is impossible to overwrite memory related configurations for Spark executors.

This JIRA proposes to change the initialization for the above.

Issue Links

• links to: GitHub Pull Request #45052