[SPARK-34506] ADD JAR with ivy coordinates should be compatible with Hive transitive behavior

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Spark Core
• Assignee: Shardul Mahadik
• Reporter: Shardul Mahadik
• Created: 23/Feb/21 09:34
• Updated: 01/Mar/21 00:18
• Resolved: 01/Mar/21 00:18

Description

SPARK-33084 added the ability to use ivy coordinates with `SparkContext.addJar`. PR #29966 claims to mimic Hive behavior although I found a few cases where it doesn't

1) The default value of the transitive parameter is false, both in case of parameter not being specified in coordinate or parameter value being invalid. The Hive behavior is transitive is true if not specified in the coordinate and false for invalid values. Also, regardless of Hive, I think a default of true for the transitive parameter also matches ivy's own defaults.

2) The parameter value for transitive parameter is regarded as case-sensitive based on the understanding that Hive behavior is case-sensitive. However, this is not correct, Hive treats the parameter value case-insensitively.

Issue Links

• relates to: SPARK-34472 SparkContext.addJar with an ivy path fails in cluster mode with a custom ivySettings file
• links to: [Github] Pull Request #31623 (shardulm94)