[SPARK-24758] Create table wants to use /user/hive/warehouse in clean clone

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Bruce Robbins
• Created: 07/Jul/18 21:19
• Updated: 14/Mar/21 03:05
• Resolved: 14/Mar/21 03:05

Description

Got a clean clone of repository:

• git clone https://github.com/apache/spark.git spark_clean
• cd spark_clean/
• ./build/sbt -Phive -Phive-thriftserver clean package

Ran spark-sql and tried to create a table:

• ./bin/spark-sql
• create table testit as select 1 a, 2 b;

Got error:

18/07/07 13:33:20 WARN HiveMetaStore: Location: file:/user/hive/warehouse/testit specified for non-external table:testit
18/07/07 13:33:20 INFO FileUtils: Creating directory if it doesn't exist: file:/user/hive/warehouse/testit
Error in query: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:file:/user/hive/warehouse/testit is not a directory or unable to create one);

To get things working, it seems you need to do something with dataframewriter (after removing metastore_db, if it exists):

scala> spark.range(0,1).write.saveAsTable("fred")
spark.range(0,1).write.saveAsTable("fred")
18/07/07 14:08:08 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
18/07/07 14:08:08 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
scala> 

After that, create table statements work:

spark-sql> create table testit as select 1 a, 2 b;
create table testit as select 1 a, 2 b;
18/07/07 14:14:40 WARN HiveMetaStore: Location: file:<work-copy-dir>/spark-warehouse/testit specified for non-external table:testit
18/07/07 14:14:41 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Time taken: 3.387 seconds
spark-sql> show tables;
show tables;
default fred    false
default testit  false
Time taken: 0.07 seconds, Fetched 2 row(s)
spark-sql>