Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7767

hive.optimize.union.remove does not work properly [Spark Branch]

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersConvert to IssueMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.1.0
    • None
    • None

    Description

      Turing on the hive.optimize.union.remove property generates wrong union all result.

      For Example:

      create table inputTbl1(key string, val string) stored as textfile;
      load data local inpath '../../data/files/T1.txt' into table inputTbl1;
      SELECT *
      FROM (
        SELECT key, count(1) as values from inputTbl1 group by key
        UNION ALL
        SELECT key, count(1) as values from inputTbl1 group by key
      ) a;  
      

      when the hive.optimize.union.remove is turned on, the query result is like:

      1	1
      2	1
      3	1
      7	1
      8	2
      

      when the hive.optimize.union.remove is turned off, the query result is like:

      7	1
      2	1
      8	2
      3	1
      1	1
      7	1
      2	1
      8	2
      3	1
      1	1
      

      The expected query result is:

      7	1
      2	1
      8	2
      3	1
      1	1
      7	1
      2	1
      8	2
      3	1
      1	1
      

      Attachments

        1. HIVE-7767.1-spark.patch
          267 kB
          Na Yang
        2. HIVE-7767.2-spark.patch
          278 kB
          Brock Noland
        3. HIVE-7767.2-spark.patch
          278 kB
          Na Yang
        4. HIVE-7767.3-spark.patch
          246 kB
          Na Yang

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            nyang Na Yang Assign to me
            nyang Na Yang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment