Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7810

Insert overwrite table query has strange behavior when set hive.optimize.union.remove=true [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Spark

      Description

      Insert overwrite table query has strange behavior when
      set hive.optimize.union.remove=true
      set hive.mapred.supports.subdirectories=true;
      set hive.merge.mapfiles=true;
      set hive.merge.mapredfiles=true;

      We expect the following two sets of queries return the same set of data result, but they do not.
      1)

      insert overwrite table outputTbl1
      SELECT * FROM
      (
      select key, 1 as values from inputTbl1
      union all
      select * FROM (
        SELECT key, count(1) as values from inputTbl1 group by key
        UNION ALL
        SELECT key, 2 as values from inputTbl1
      ) a
      )b;
      select * from outputTbl1 order by key, values;
      

      Below is the query result:

      1	1
      1	2
      2	1
      2	2
      3	1
      3	2
      7	1
      7	2
      8	2
      8	2
      8	2
      

      2)

      SELECT * FROM
      (
      select key, 1 as values from inputTbl1
      union all
      select * FROM (
        SELECT key, count(1) as values from inputTbl1 group by key
        UNION ALL
        SELECT key, 2 as values from inputTbl1
      ) a
      )b order by key, values;
      

      Below is the query result:

      1	1
      1	1
      1	2
      2	1
      2	1
      2	2
      3	1
      3	1
      3	2
      7	1
      7	1
      7	2
      8	1
      8	1
      8	2
      8	2
      8	2
      

      Some data is missing in the first set of query result.

        Attachments

        1. HIVE-7810.1-spark.patch
          32 kB
          Na Yang

          Issue Links

            Activity

              People

              • Assignee:
                nyang Na Yang
                Reporter:
                nyang Na Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: