[SPARK-24096] create table as select not using hive.default.fileformat

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.3.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: StephenZou
• Created: 26/Apr/18 07:25
• Updated: 26/Apr/18 16:09
• Resolved: 26/Apr/18 16:08

Description

In my spark conf directory, hive-site.xml have an item indicating orc is the default file format.

<property>
<name>hive.default.fileformat</name>
<value>orc</value>
</property>

 

But when I use "create table as select ..." to create a table, the output format is plain text. 

It works only I use "set hive.default.fileformat=orc"

 

Then I walked through the spark code and found in 

sparkSqlParser:visitCreateHiveTable(), 

val defaultStorage = HiveSerDe.getDefaultStorage(conf)  the conf is SQLConf,

that explains the above observation, 

"set hive.default.fileformat=orc" is put into conf map, hive-site.xml is not. 

 

It's quite misleading, How to unify the settings?

 

 

Issue Links

• duplicates: SPARK-18397 cannot create table by using the hive default fileformat