[SPARK-23729] Glob resolution breaks remote naming of files/archives

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.1, 2.4.0
• Component/s: Spark Submit
• Labels: regression
• Assignee: Mihaly Toth
• Reporter: Mihaly Toth
• Created: 18/Mar/18 05:59
• Updated: 27/Apr/18 18:30
• Resolved: 22/Mar/18 00:14

Description

Given one uses spark-submit with either of the --archives or the --files parameters, in case the file name actually contains glob patterns, the rename part (...#nameAs) of the filename will eventually be ignored.

Thinking over the resolution cases, if the resolution results in multiple files, it does not make sense to send all of them under the same remote name. So this should then result in an error.

Issue Links

• is duplicated by: SPARK-24113 --archives hdfs://some/path.zip#newname renaming no longer works
• links to: [Github] Pull Request #20853 (misutoth)