Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-9449

Push YARN configuration to Spark while deply Spark on YARN [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:
      None

      Description

      We only push Spark configuration and RSC configuration to Spark while launch Spark cluster now, for Spark on YARN mode, Spark need extra YARN configuration to launch Spark cluster. Besides this, to support dynamically configuration setting for RSC configuration/YARN configuration, we need to recreate SparkSession while RSC configuration/YARN configuration update as well, as they may influence the Spark cluster deployment as well.

        Attachments

        1. HIVE-9449.1-spark.patch
          5 kB
          Xuefu Zhang
        2. HIVE-9449.1-spark.patch
          5 kB
          Chengxiang Li
        3. HIVE-9449.2-spark.patch
          5 kB
          Chengxiang Li
        4. HiveonSparkconfiguration.pdf
          93 kB
          Chengxiang Li

          Issue Links

            Activity

              People

              • Assignee:
                chengxiang li Chengxiang Li
                Reporter:
                chengxiang li Chengxiang Li
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: