[SPARK-38194] Make memory overhead factor configurable

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.3.0, 3.4.0
• Component/s: Kubernetes, Mesos, YARN
• Assignee: Adam Binford
• Reporter: Adam Binford
• Created: 13/Feb/22 14:03
• Updated: 02/Jun/22 13:47
• Resolved: 16/Mar/22 15:56

Description

Currently if the memory overhead is not provided for a Yarn job, it defaults to 10% of the respective driver/executor memory. This 10% is hard-coded and the only way to increase memory overhead is to set the exact memory overhead. We have seen more than 10% memory being used, and it would be helpful to be able to set the default overhead factor so that the overhead doesn't need to be pre-calculated for any driver/executor memory size. 

Issue Links

• causes: SPARK-39363 fix spark.kubernetes.memoryOverheadFactor deprecation warning
• links to: [Github] Pull Request #35504 (Kimahriman)
• links to: [Github] Pull Request #35900 (tgravescs)
• links to: [Github] Pull Request #35901 (Kimahriman)
• links to: [Github] Pull Request #35901 (Kimahriman)
• links to: [Github] Pull Request #35912 (Kimahriman)
• links to: [Github] Pull Request #35934 (Kimahriman)