[SPARK-33374] [CORE] Remove unnecessary python path from spark home

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.0.1
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Zhongwei Zhu
• Created: 06/Nov/20 19:24
• Updated: 10/Nov/20 16:47
• Resolved: 10/Nov/20 16:46

Description

Currently, spark-submit will upload pyspark.zip and py4j-0.10.9-src.zip into staging folder, and both files will be added into PYTHONPATH. So it's unnecessary to add duplicate files in current spark home folder on local machine.
Output of `sys.path` as below:

'D:\\data\\yarnnm\\local\\usercache\\z\\appcache\\application_1603546638930_150736\\container_e1148_1603546638930_150736_01_000002
pyspark.zip',
'D:\\data\\yarnnm\\local\\usercache\\z\\appcache\\application_1603546638930_150736\\container_e1148_1603546638930_150736_01_000002
py4j-0.10.7-src.zip',
'D:\\data\\spark.latest\\python\\lib
pyspark.zip',
'D:\\data\\spark.latest\\python\\lib
py4j-0.10.7-src.zip',

Issue Links

• links to: [Github] Pull Request #30281 (warrenzhu25)