[SPARK-24113] --archives hdfs://some/path.zip#newname renaming no longer works

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.3.0
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Peter Parente
• Created: 27/Apr/18 18:25
• Updated: 17/May/20 18:14
• Resolved: 27/Apr/18 18:30

Description

In spark < 2.3.0, using #NAME as part of an archive name results in a symlink of NAME in executor yarn containers pointing to the extracted archive. In spark 2.3.0, the #NAME is no longer honored and the symlink is named after basename of the archive file instead.

For instance:

org.apache.spark.deploy.SparkSubmit --master yarn --deploy-mode client --conf spark.executor.memory=8G --conf spark.driver.memory=4g --conf spark.driver.maxResultSize=2g --conf spark.sql.catalogImplementation=hive --conf spark.executorEnv.PYSPARK_PYTHON=./CONDA/my-custom-env/bin/python --conf spark.driver.extraClassPath=./resources/conf --conf spark.executor.cores=5 --conf spark.dynamicAllocation.maxExecutors=10 --conf spark.sql.shuffle.partitions=2000 --conf spark.dynamicAllocation.cachedExecutorIdleTimeout=30m --conf spark.shuffle.service.enabled=True --conf spark.executor.instances=1 --conf spark.yarn.queue=notebook --conf spark.dynamicAllocation.enabled=True --conf spark.driver.extraJavaOptions=-Dmpi.conf.dir=./resources/conf --jars ./resources/PlatformToolkit.jar --keytab /home/p-pparente/.keytab --principal [e-mail] --archives hdfs:///some-path/my-custom-env.zip#CONDA --executor-memory 8G --executor-cores 5 pyspark-shell

results in the following in executors containers in Spark 2.2.1 (which is correct)

lrwxrwxrwx 1 parente yarn   65 Apr 27 11:44 CONDA -> /mnt/disk1/yarn/local/filecache/6013/my-custom-env.zip

and results in the following in executor containers in Spark 2.3.0 (which appears to be a regression)

lrwxrwxrwx 1 parente yarn 65 Apr 27 11:51 my-custom-env.zip -> /mnt/disk4/yarn/local/filecache/1272/my-custom-env.zip

Issue Links

• duplicates: SPARK-23729 Glob resolution breaks remote naming of files/archives