[SPARK-9885] IsolatedClientLoader ignores shared prefixes and barrier prefixes when "spark.sql.hive.metastore.jars" is set to "maven"

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.1, 1.5.0
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Sprint: Spark 1.5 doc/QA sprint
• Assignee: Yin Huai
• Reporter: Cheng Lian
• Created: 12/Aug/15 17:06
• Updated: 13/Aug/15 07:10
• Resolved: 13/Aug/15 07:10

Description

I have a local Hive 0.13.1 metastore backed by MySQL. MySQL JDBC connector is put under $JAVA_HOME/jre/lib/ext. With the following spark-defaults.conf

spark.sql.hive.metastore.version 0.13.1
spark.sql.hive.metastore.jars maven

Spark shell fails to create HiveContext because MySQL JDBC driver couldn't be properly loaded.

This is probably because IsolatedClientLoader ignores shared prefixes and barrier prefixes when spark.sql.hive.metastore.jars is set to maven. See this line.

Issue Links

• links to: [Github] Pull Request #8158 (yhuai)