[SPARK-12120] Improve exception message when failing to initialize HiveContext in PySpark

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.6.1, 2.0.0
• Component/s: PySpark
• Assignee: Jeff Zhang
• Reporter: Jeff Zhang
• Created: 03/Dec/15 08:50
• Updated: 24/Jan/16 20:31
• Resolved: 24/Jan/16 20:31

Description

I get the following exception message when failing to initialize HiveContext. This is hard to figure out why HiveContext failed to initialize. Actually I build spark with hive profile enabled. The reason the HiveContext failed is due to I didn't start hdfs service. And actually I can see the full stacktrace in spark-shell. And I also can see the full stack trace in python2. The issue only exists in python2.x

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jzhang/github/spark/python/pyspark/sql/context.py", line 430, in createDataFrame
    jdf = self._ssql_ctx.applySchemaToPythonRDD(jrdd.rdd(), schema.json())
  File "/Users/jzhang/github/spark/python/pyspark/sql/context.py", line 691, in _ssql_ctx
    "build/sbt assembly", e)
Exception: ("You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt assembly", Py4JJavaError(u'An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.\n', JavaObject id=o34))

Issue Links

• links to: [Github] Pull Request #10126 (zjffdu)