Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-11276

Optimization around job submission and adding jars [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 1.1.0
    • Fix Version/s: None
    • Component/s: Spark
    • Labels:
      None

      Description

      It seems that Hive on Spark has some room for performance improvement on job submission. Specifically, we are calling refreshLocalResources() for every job submission despite there is are no changes in the jar list. Since Hive on Spark is reusing the containers in the whole user session, we might be able to optimize that.

      We do need to take into consideration the case of dynamic allocation, in which new executors might be added.

      This task is some R&D in this area.

        Attachments

          Activity

            People

            • Assignee:
              chengxiang li Chengxiang Li
              Reporter:
              xuefuz Xuefu Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: