[SPARK-39090] AddJar or AddFile call on a duplicate file should not result in failure

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.1.0
• Component/s: Spark Submit

Language

Python scala

• Assignee: Unassigned
• Reporter: Praneet Sharma
• Created: 02/May/22 22:09
• Updated: 02/May/22 22:15
• Resolved: 02/May/22 22:15

Description

Currently, when addFile or addJar is used to upload a file or a jar which was already uploaded, it should not result in IOException. This is because the spark-submit behavior is to ignore adding duplicate jars/files and logging appropriate warning message. The same should be done for addFile and addJar APIs in Livy.