[SPARK-16013] Add option to disable HiveContext in spark-shell/pyspark

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.6.1
• Component/s: PySpark, Spark Shell
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Jeff Zhang
• Created: 17/Jun/16 07:06
• Updated: 21/May/19 04:32
• Resolved: 21/May/19 04:32

Description

In spark2.0, we can disable hivecontext by setting spark.sql.catalogImplementation, but in spark 1.6 it seems there's no option to turn off hivecontext. This bring several issues I can see.

• If user spark multiple spark-shell on the same machine but don't specify hive-site.xml will cause embedded derby conflict.
• For any spark downstream project, if they want to reuse the code in spark-shell, then they have to reply on whether hive profile is turned on to decide whether HiveContext is created. This doesn't make sense to me.

Although spark 2.0 don't have such issue, considering most of people are still use spark 1.x, this feature would benefit users I think. I can create PR if this feature is reasonable for users.