[SPARK-25021] Add spark.executor.pyspark.memory support to Kubernetes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: Kubernetes, Spark Core
• Assignee: Ilan Filonenko
• Reporter: Ryan Blue
• Created: 03/Aug/18 18:35
• Updated: 17/May/20 18:24
• Resolved: 09/Sep/18 05:20

Description

SPARK-25004 adds spark.executor.pyspark.memory to control the memory allocation for PySpark and updates YARN to add this memory to its container requests. Kubernetes should do something similar to account for the python memory allocation.

Issue Links

• depends upon: SPARK-25004 Add spark.executor.pyspark.memory config to set resource.RLIMIT_AS
• links to: [Github] Pull Request #22298 (ifilonenko)
• links to: [Github] Pull Request #22376 (ifilonenko)