[SPARK-27205] spark-shell with packages option fails to load transitive dependencies even ivy successfully pulls jars

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 19/Mar/19 14:09
• Updated: 25/Mar/19 00:08
• Resolved: 20/Mar/19 22:56

Description

I found this bug while testing my patch regarding Spark SQL Kafka module - I tend to open spark-shell and link kafka module via `–packages`.

When we run

./bin/spark-shell --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0

we should be able to import "org.apache.kafka" in spark-shell, but it doesn't work for current master branch.

There's not enough evidence as well as I have no idea what's happening here even with `–verbose` option, so I had to spend couple of hours dealing with git bisect.

Turned out the commit introducing the bug was SPARK-26977 (81dd21fda99da48ed76adb739a07d1dabf1ffb51).

 

Issue Links

• links to: GitHub Pull Request #24147