[SPARK-18439] spark 2.0.1 fails in windows when using file:/// scratchdir in hive-site.xml

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.1
• Component/s: Windows
• Labels: windows

Environment

Windows

• Assignee: Unassigned
• Reporter: Javier Luraschi
• Created: 14/Nov/16 23:40
• Updated: 15/Nov/16 15:18
• Resolved: 15/Nov/16 15:18

Description

Derby-based Hive metastore using WINUTILS.exe and hive.exec.scratchdir, hive.exec.local.scratchdir and hive.metastore.warehouse.dir with "C:\" paths works fine in Spark 2.0.0.

However, Spark 2.0.1 throws a "No FileSystem for scheme: C" at "at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)"

Changing changing hive-site.xml to use a "file:///c:\" path instead fixes this issue but throws a "Unable to create log directory file:///C:\" at "org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)" error instead.

One note: This last exception I've seen associated when users don't run WINUTILS.exe to set the permissions. Therefore, it makes me think that it is possible that WINUTILS.exe might not support file:/// paths

Issue Links

• duplicates: SPARK-15899 file scheme should be used correctly