Details
-
Improvement
-
Status: In Progress
-
Minor
-
Resolution: Unresolved
-
3.0.1
-
None
-
None
Description
This config provides similar function as `spark.yarn.archive`, but instead passing location of pyspark.zip and py4j.zip. It has below benefits:
1. If I have different version of `spark.yarn.archive` from current machine, then there is no way to provide matched pyspark version. This happens in version upgrade.
2. This can save effort of uploading pyspark.zip and py4j.zip every time.