[HIVE-7436] Load Spark configuration into Hive driver [Spark Branch]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Spark
• Assignee: Chengxiang Li
• Reporter: Chengxiang Li
• Created: 17/Jul/14 12:32
• Updated: 29/May/15 02:31
• Resolved: 22/Jul/14 18:43

Description

load Spark configuration into Hive driver, there are 3 ways to setup spark configurations:

1. Java property.
2. Configure properties in spark configuration file(spark-defaults.conf).
3. Hive configuration file(hive-site.xml).

The below configuration has more priority, and would overwrite previous configuration with the same property name.

Please refer to http://spark.apache.org/docs/latest/configuration.html for all configurable properties of spark, and you can configure spark configuration in Hive through following ways:

1. Configure through spark configuration file.
   • Create spark-defaults.conf, and place it in the /etc/spark/conf configuration directory. configure properties in spark-defaults.conf in java properties format.
   • Create the $SPARK_CONF_DIR environment variable and set it to the location of spark-defaults.conf.
     export SPARK_CONF_DIR=/etc/spark/conf
   • Add $SAPRK_CONF_DIR to the $HADOOP_CLASSPATH environment variable.
     export HADOOP_CLASSPATH=$SPARK_CONF_DIR:$HADOOP_CLASSPATH
2. Configure through hive configuration file.
   • edit hive-site.xml in hive conf directory, configure properties in spark-defaults.conf in xml format.

Hive driver default spark properties:

name	default value	description
spark.master	local	Spark master url.
spark.app.name	Hive on Spark	Default Spark application name.

NO PRECOMMIT TESTS. This is for spark-branch only.