Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-11363

Prewarm Hive on Spark containers [Spark Branch]

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: spark-branch, 1.3.0, 2.0.0
    • Component/s: Spark
    • Labels:
      None

      Description

      When Hive job is launched by Oozie, a Hive session is created and job script is executed. Session is closed when Hive job is completed. Thus, Hive session is not shared among Hive jobs either in an Oozie workflow or across workflows. Since the parallelism of a Hive job executed on Spark is impacted by the available executors, such Hive jobs will suffer the executor ramp-up overhead. The idea here is to wait a bit so that enough executors can come up before a job can be executed.

        Attachments

        1. HIVE-11363.5-spark.patch
          7 kB
          Xuefu Zhang
        2. HIVE-11363.4-spark.patch
          6 kB
          Xuefu Zhang
        3. HIVE-11363.3-spark.patch
          5 kB
          Xuefu Zhang
        4. HIVE-11363.2-spark.patch
          0.3 kB
          Xuefu Zhang
        5. HIVE-11363.1-spark.patch
          14 kB
          Xuefu Zhang

          Issue Links

            Activity

              People

              • Assignee:
                xuefuz Xuefu Zhang
                Reporter:
                xuefuz Xuefu Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: