[SPARK-24384] spark-submit --py-files with .py files doesn't work in client mode before context initialization

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0, 2.4.0
• Fix Version/s: 2.3.1, 2.4.0
• Component/s: PySpark, Spark Submit
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 24/May/18 17:34
• Updated: 12/Dec/22 17:51
• Resolved: 30/May/18 17:51

Description

In case the given Python file is .py file (zip file seems fine), seems the python path is dynamically added after the context is got initialized.

with this pyFile:

$ cat /home/spark/tmp.py
def testtest():
    return 1

This works:

$ cat app.py
import pyspark
pyspark.sql.SparkSession.builder.getOrCreate()
import tmp
print("************************%s" % tmp.testtest())

$ ./bin/spark-submit --master yarn --deploy-mode client --py-files /home/spark/tmp.py app.py
...
************************1

but this doesn't:

$ cat app.py
import pyspark
import tmp
pyspark.sql.SparkSession.builder.getOrCreate()
print("************************%s" % tmp.testtest())

$ ./bin/spark-submit --master yarn --deploy-mode client --py-files /home/spark/tmp.py app.py
Traceback (most recent call last):
  File "/home/spark/spark/app.py", line 2, in <module>
    import tmp
ImportError: No module named tmp

See https://issues.apache.org/jira/browse/SPARK-21945?focusedCommentId=16488486&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16488486

Issue Links

• links to: [Github] Pull Request #21426 (HyukjinKwon)