[SPARK-10066] Can't create HiveContext with spark-shell or spark-sql on snapshot

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.5.0
• Component/s: Spark Shell, SQL

Environment

Centos 6.6

• Assignee: Unassigned Assign to me
• Reporter: Robert Beauchemin
• Created: 17/Aug/15 21:16
• Updated: 08/Feb/16 16:45
• Resolved: 13/Jan/16 09:23

Description

Built the 1.5.0-preview-20150812 with the following:

./make-distribution.sh -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -Psparkr -DskipTests

Starting spark-shell or spark-sql returns the following error:
java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx------
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:612) .... [elided]
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)

It's trying to create a new HiveContext. Running pySpark or sparkR works and creates a HiveContext successfully. SqlContext can be created successfully with any shell.

I've tried changing permissions on that HDFS directory (even as far as making it world-writable) without success. Tried changing SPARK_USER and also running spark-shell as different users without success.

This works on same machine on 1.4.1 and on earlier pre-release versions of Spark 1.5.0 (same make-distribution parms) sucessfully. Just trying the snapshot...

Issue Links

• duplicates: SPARK-10528 spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.
• relates to: SPARK-10528 spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.