[SPARK-8586] SQL add jar command does not work well with Scala REPL

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.4.0, 2.1.0
• Component/s: SQL
• Assignee: Yin Huai
• Reporter: Yin Huai
• Created: 24/Jun/15 04:39
• Updated: 13/Apr/17 14:59
• Resolved: 24/Mar/17 11:46

Description

Seems SparkIMain always resets the context class loader in loadAndRunReq. So, SerDe added through add jar command may not be loaded in the context class loader when we lookup the table.
For example, the following code will fail when we try to show the table.

hive.sql("add jar sql/hive/src/test/resources/hive-hcatalog-core-0.13.1.jar")
hive.sql("drop table if exists jsonTable")
hive.sql("CREATE TABLE jsonTable(key int, val string) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'")
hive.createDataFrame((1 to 100).map(i => (i, s"str$i"))).toDF("key", "val").insertInto("jsonTable")
hive.table("jsonTable").show

Issue Links

• duplicates: SPARK-11421 Add the ability to add a jar to the current class loader
• is related to: SPARK-15634 SQL repl is bricked if a function is registered with a non-existent jar