[SPARK-19878] Add hive configuration when initialize hive serde in InsertIntoHiveTable.scala

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0, 1.6.0, 2.0.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: SQL
• Labels: patch

Environment

Centos 6.5: Hadoop 2.6.0, Spark 1.5.0, Hive 1.1.0

• Flags: Patch
• Assignee: Vinod KC
• Reporter: jianjin qin
• Created: 09/Mar/17 02:55
• Updated: 22/Nov/17 17:23
• Resolved: 22/Nov/17 17:23

Description

When case class InsertIntoHiveTable intializes a serde it explicitly passes null for the Configuration in Spark 1.5.0:

https://github.com/apache/spark/blob/v1.5.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L58

While in Spark 2.0.0, the HiveWriterContainer intializes a serde it also just passes null for the Configuration:

https://github.com/apache/spark/blob/v2.0.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveWriterContainers.scala#L161

When we implement a hive serde, we want to use the hive configuration to get some static and dynamic settings, but we can not do it !

So this patch add the configuration when initialize hive serde.

Issue Links

• relates to: SPARK-19580 Support for avro.schema.url while writing to hive table
• relates to: SPARK-17920 HiveWriterContainer passes null configuration to serde.initialize, causing NullPointerException in AvroSerde when using avro.schema.url
• links to: [Github] Pull Request #19779 (vinodkc)
• links to: [Github] Pull Request #19795 (vinodkc)