Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7503

Support Hive's multi-table insert query with Spark [Spark Branch]

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.1.0
    • Spark

    Description

      For Hive's multi insert query (https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML), there may be an MR job for each insert. When we achieve this with Spark, it would be nice if all the inserts can happen concurrently.

      It seems that this functionality isn't available in Spark. To make things worse, the source of the insert may be re-computed unless it's staged. Even with this, the inserts will happen sequentially, making the performance suffer.

      This task is to find out what takes in Spark to enable this without requiring staging the source and sequential insertion. If this has to be solved in Hive, find out an optimum way to do this.

      Attachments

        1. HIVE-7503.1-spark.patch
          24 kB
          Chao Sun
        2. HIVE-7503.2-spark.patch
          26 kB
          Chao Sun
        3. HIVE-7503.3-spark.patch
          25 kB
          Chao Sun
        4. HIVE-7503.4-spark.patch
          27 kB
          Chao Sun
        5. HIVE-7503.5-spark.patch
          27 kB
          Chao Sun
        6. HIVE-7503.6-spark.patch
          34 kB
          Chao Sun
        7. HIVE-7503.7-spark.patch
          33 kB
          Chao Sun
        8. HIVE-7503.8-spark.patch
          61 kB
          Chao Sun
        9. HIVE-7503.9-spark.patch
          61 kB
          Chao Sun

        Issue Links

          is blocked by

          New Feature - A new feature of the product, which has yet to be developed. SPARK-2688 Need a way to run multiple data pipeline concurrently

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue HIVE-7842 Enable qtest load_dyn_part1.q [Spark Branch]

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-8233 multi-table insertion doesn't work with ForwardOperator [Spark Branch]

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-8208 Multi-table insertion optimization #1: don't always break operator tree. [Spark Branch]

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-8215 Multi-table insertion optimization #3: use 1+1 tasks instead of 1+N tasks [Spark Branch]

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-8209 Multi-table insertion optimization #2: use separate context [Spark Branch]

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Test - A new unit, integration or system test. HIVE-8207 Add .q tests for multi-table insertion [Spark Branch]

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HIVE-7731 Incorrect result returned when a map work has multiple downstream reduce works [Spark Branch]

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HIVE-8438 Clean up code introduced by HIVE-7503 and such [Spark Plan]

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-8219 Multi-Insert optimization, don't sink the source into a file [Spark Branch]

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-8220 Refactor multi-insert code such that plan splitting and task generation are modular and reusable [Spark Branch]

          • Major - Major loss of function.
          • Resolved
          requires

          Sub-task - The sub-task of the issue HIVE-7525 Research to find out if it's possible to submit Spark jobs concurrently using shared SparkContext [Spark Branch]

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link RB Link

          Activity

            People

              csun Chao Sun
              xuefuz Xuefu Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: