[SPARK-39363] fix spark.kubernetes.memoryOverheadFactor deprecation warning

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: Kubernetes
• Assignee: Unassigned
• Reporter: Thomas Graves
• Created: 02/Jun/22 13:45
• Updated: 04/Jun/22 13:06

Description

see https://github.com/apache/spark/pull/36744 for details.

 

It looks like we deprecated spark.kubernetes.memoryOverheadFactor, but it has a default value which leads it to printing warnings all the time.

{{}}

22/06/01 23:53:49 WARN SparkConf: The configuration key 'spark.kubernetes.memoryOverheadFactor' has been deprecated as of Spark 3.3.0 and may be removed in the future. Please use spark.driver.memoryOverheadFactor and spark.executor.memoryOverheadFactor

{{}}

{{We should remove the default value if possible. It should only be used as fallback but we should be able to use the default from }}spark.driver.memoryOverheadFactor.

 

Issue Links

• is caused by: SPARK-38194 Make memory overhead factor configurable
• links to: [Github] Pull Request #36767 (Kimahriman)
• links to: [Github] Pull Request #36767 (Kimahriman)