Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7870

Insert overwrite table query does not generate correct task plan [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:

      Description

      Insert overwrite table query does not generate correct task plan when hive.optimize.union.remove and hive.merge.sparkfiles properties are ON.

      set hive.optimize.union.remove=true
      set hive.merge.sparkfiles=true
      
      insert overwrite table outputTbl1
      SELECT * FROM
      (
      select key, 1 as values from inputTbl1
      union all
      select * FROM (
        SELECT key, count(1) as values from inputTbl1 group by key
        UNION ALL
        SELECT key, 2 as values from inputTbl1
      ) a
      )b;
      select * from outputTbl1 order by key, values;
      

      query result

      1	1
      1	2
      2	1
      2	2
      3	1
      3	2
      7	1
      7	2
      8	2
      8	2
      8	2
      

      expected result:

      1	1
      1	1
      1	2
      2	1
      2	1
      2	2
      3	1
      3	1
      3	2
      7	1
      7	1
      7	2
      8	1
      8	1
      8	2
      8	2
      8	2
      

      Move work is not working properly and some data are missing during move.

        Attachments

        1. HIVE-7870.1-spark.patch
          6 kB
          Chao Sun
        2. HIVE-7870.2-spark.patch
          320 kB
          Na Yang
        3. HIVE-7870.3-spark.patch
          608 kB
          Na Yang
        4. HIVE-7870.4-spark.patch
          52 kB
          Na Yang
        5. HIVE-7870.5-spark.patch
          52 kB
          Na Yang

          Issue Links

            Activity

              People

              • Assignee:
                nyang Na Yang
                Reporter:
                nyang Na Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: