Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7436

Load Spark configuration into Hive driver [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:
      None

      Description

      load Spark configuration into Hive driver, there are 3 ways to setup spark configurations:

      1. Java property.
      2. Configure properties in spark configuration file(spark-defaults.conf).
      3. Hive configuration file(hive-site.xml).

      The below configuration has more priority, and would overwrite previous configuration with the same property name.

      Please refer to http://spark.apache.org/docs/latest/configuration.html for all configurable properties of spark, and you can configure spark configuration in Hive through following ways:

      1. Configure through spark configuration file.
        • Create spark-defaults.conf, and place it in the /etc/spark/conf configuration directory. configure properties in spark-defaults.conf in java properties format.
        • Create the $SPARK_CONF_DIR environment variable and set it to the location of spark-defaults.conf.
          export SPARK_CONF_DIR=/etc/spark/conf
        • Add $SAPRK_CONF_DIR to the $HADOOP_CLASSPATH environment variable.
          export HADOOP_CLASSPATH=$SPARK_CONF_DIR:$HADOOP_CLASSPATH
      2. Configure through hive configuration file.
        • edit hive-site.xml in hive conf directory, configure properties in spark-defaults.conf in xml format.

      Hive driver default spark properties:

      name default value description
      spark.master local Spark master url.
      spark.app.name Hive on Spark Default Spark application name.

      NO PRECOMMIT TESTS. This is for spark-branch only.

        Attachments

        1. HIVE-7436-Spark.1.patch
          4 kB
          Chengxiang Li
        2. HIVE-7436-Spark.2.patch
          5 kB
          Chengxiang Li
        3. HIVE-7436-Spark.3.patch
          6 kB
          Chengxiang Li

          Activity

            People

            • Assignee:
              chengxiang li Chengxiang Li
              Reporter:
              chengxiang li Chengxiang Li
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: