Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7436

Load Spark configuration into Hive driver [Spark Branch]

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.1.0
    • Spark
    • None

    Description

      load Spark configuration into Hive driver, there are 3 ways to setup spark configurations:

      1. Java property.
      2. Configure properties in spark configuration file(spark-defaults.conf).
      3. Hive configuration file(hive-site.xml).

      The below configuration has more priority, and would overwrite previous configuration with the same property name.

      Please refer to http://spark.apache.org/docs/latest/configuration.html for all configurable properties of spark, and you can configure spark configuration in Hive through following ways:

      1. Configure through spark configuration file.
        • Create spark-defaults.conf, and place it in the /etc/spark/conf configuration directory. configure properties in spark-defaults.conf in java properties format.
        • Create the $SPARK_CONF_DIR environment variable and set it to the location of spark-defaults.conf.
          export SPARK_CONF_DIR=/etc/spark/conf
        • Add $SAPRK_CONF_DIR to the $HADOOP_CLASSPATH environment variable.
          export HADOOP_CLASSPATH=$SPARK_CONF_DIR:$HADOOP_CLASSPATH
      2. Configure through hive configuration file.
        • edit hive-site.xml in hive conf directory, configure properties in spark-defaults.conf in xml format.

      Hive driver default spark properties:

      name default value description
      spark.master local Spark master url.
      spark.app.name Hive on Spark Default Spark application name.

      NO PRECOMMIT TESTS. This is for spark-branch only.

      Attachments

        1. HIVE-7436-Spark.3.patch
          6 kB
          Chengxiang Li
        2. HIVE-7436-Spark.2.patch
          5 kB
          Chengxiang Li
        3. HIVE-7436-Spark.1.patch
          4 kB
          Chengxiang Li

        Activity

          People

            chengxiang li Chengxiang Li
            chengxiang li Chengxiang Li
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: