[SPARK-19367] Hive metastore temporary configuration doesn't specify default filesystem

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 1.6.3, 2.0.2, 2.1.0, 2.2.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Jacek Lewandowski
• Created: 26/Jan/17 12:41
• Updated: 21/May/19 04:12
• Resolved: 21/May/19 04:12

Description

When a temporary configuration is obtained in HiveUtils, it specify the warehouse location on the local file system. This is fine, but if the default file system is specified different than local, it may fail due to the lack of required configuration options.

What I propose here is to set default file system to file:/// in addition to setting warehouse location to local file system.

Why - because regardless of what file system is accessed by Hive, it does initialisation with the default file system and if it requires some configuration, it will fail.

Issue Links

• relates to: SPARK-17810 Default spark.sql.warehouse.dir is relative to local FS but can resolve as HDFS path