[SPARK-33406] k8s pyspark shell doesn't honor pyspark memory setting

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Kubernetes
• Assignee: Unassigned
• Reporter: Thomas Graves
• Created: 10/Nov/20 00:43
• Updated: 10/Nov/20 00:43

Description

When running the interactive pyspark shell in client mode on k8s, it doesn't honor thespark.executor.pyspark.memory memory setting when requesting containers from k8s. 

If you run it in cluster mode and specify a python script then it does work.