Hive
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7503

Support Hive's multi-table insert query with Spark [Spark Branch]

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: Spark
    • Labels:

      Description

      For Hive's multi insert query (https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML), there may be an MR job for each insert. When we achieve this with Spark, it would be nice if all the inserts can happen concurrently.

      It seems that this functionality isn't available in Spark. To make things worse, the source of the insert may be re-computed unless it's staged. Even with this, the inserts will happen sequentially, making the performance suffer.

      This task is to find out what takes in Spark to enable this without requiring staging the source and sequential insertion. If this has to be solved in Hive, find out an optimum way to do this.

      1. HIVE-7503.1-spark.patch
        24 kB
        Chao
      2. HIVE-7503.2-spark.patch
        26 kB
        Chao
      3. HIVE-7503.3-spark.patch
        25 kB
        Chao
      4. HIVE-7503.4-spark.patch
        27 kB
        Chao
      5. HIVE-7503.5-spark.patch
        27 kB
        Chao
      6. HIVE-7503.6-spark.patch
        34 kB
        Chao
      7. HIVE-7503.7-spark.patch
        33 kB
        Chao
      8. HIVE-7503.8-spark.patch
        61 kB
        Chao
      9. HIVE-7503.9-spark.patch
        61 kB
        Chao

        Issue Links

          Activity

          Xuefu Zhang created issue -
          Xuefu Zhang made changes -
          Field Original Value New Value
          Link This issue is blocked by SPARK-2688 [ SPARK-2688 ]
          Xuefu Zhang made changes -
          Link This issue requires HIVE-7525 [ HIVE-7525 ]
          Xuefu Zhang made changes -
          Assignee Xuefu Zhang [ xuefuz ]
          Xuefu Zhang made changes -
          Assignee Xuefu Zhang [ xuefuz ]
          Chao made changes -
          Assignee Chao [ csun ]
          Chao made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Chao made changes -
          Link This issue relates to HIVE-7731 [ HIVE-7731 ]
          Brock Noland made changes -
          Summary Support Hive's multi-table insert query with Spark Support Hive's multi-table insert query with Spark [Spark Branch]
          Chao made changes -
          Attachment HIVE-7503.1-spark.patch [ 12663810 ]
          Xuefu Zhang made changes -
          Link This issue is depended upon by HIVE-7842 [ HIVE-7842 ]
          Xuefu Zhang made changes -
          Labels spark-m1
          Chao made changes -
          Attachment HIVE-7503.2-spark.patch [ 12666824 ]
          Chao made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Chao made changes -
          Remote Link This issue links to "RB Link (Web Link)" [ 18213 ]
          Chao made changes -
          Attachment HIVE-7503.3-spark.patch [ 12666885 ]
          Chao made changes -
          Attachment HIVE-7503.4-spark.patch [ 12666955 ]
          Chao made changes -
          Attachment HIVE-7503.5-spark.patch [ 12667195 ]
          Chao made changes -
          Attachment HIVE-7503.6-spark.patch [ 12669756 ]
          Chao made changes -
          Attachment HIVE-7503.7-spark.patch [ 12670148 ]
          Chao made changes -
          Attachment HIVE-7503.8-spark.patch [ 12670162 ]
          Chao made changes -
          Attachment HIVE-7503.9-spark.patch [ 12670163 ]
          Xuefu Zhang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s spark-branch [ 12327352 ]
          Resolution Fixed [ 1 ]
          Chao made changes -
          Link This issue is depended upon by HIVE-8207 [ HIVE-8207 ]
          Chao made changes -
          Link This issue is depended upon by HIVE-8208 [ HIVE-8208 ]
          Chao made changes -
          Link This issue is depended upon by HIVE-8209 [ HIVE-8209 ]
          Chao made changes -
          Link This issue is depended upon by HIVE-8215 [ HIVE-8215 ]
          Xuefu Zhang made changes -
          Link This issue relates to HIVE-8219 [ HIVE-8219 ]
          Xuefu Zhang made changes -
          Link This issue relates to HIVE-8220 [ HIVE-8220 ]
          Chao made changes -
          Link This issue is depended upon by HIVE-8233 [ HIVE-8233 ]
          Xuefu Zhang made changes -
          Link This issue relates to HIVE-8438 [ HIVE-8438 ]

            People

            • Assignee:
              Chao
              Reporter:
              Xuefu Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development