[HIVE-7810] Insert overwrite table query has strange behavior when set hive.optimize.union.remove=true [Spark Branch]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Spark
• Labels: TODOC-SPARK TODOC15
• Assignee: Na Yang
• Reporter: Na Yang
• Created: 20/Aug/14 21:11
• Updated: 30/May/15 21:44
• Resolved: 26/Aug/14 01:07

Description

Insert overwrite table query has strange behavior when
set hive.optimize.union.remove=true
set hive.mapred.supports.subdirectories=true;
set hive.merge.mapfiles=true;
set hive.merge.mapredfiles=true;

We expect the following two sets of queries return the same set of data result, but they do not.
1)

insert overwrite table outputTbl1
SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
  SELECT key, count(1) as values from inputTbl1 group by key
  UNION ALL
  SELECT key, 2 as values from inputTbl1
) a
)b;
select * from outputTbl1 order by key, values;

Below is the query result:

1	1
1	2
2	1
2	2
3	1
3	2
7	1
7	2
8	2
8	2
8	2

2)

SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
  SELECT key, count(1) as values from inputTbl1 group by key
  UNION ALL
  SELECT key, 2 as values from inputTbl1
) a
)b order by key, values;

Below is the query result:

1	1
1	1
1	2
2	1
2	1
2	2
3	1
3	1
3	2
7	1
7	1
7	2
8	1
8	1
8	2
8	2
8	2

Some data is missing in the first set of query result.

Issue Links

• is part of: HIVE-7292 Hive on Spark
• relates to: HIVE-7870 Insert overwrite table query does not generate correct task plan [Spark Branch]
• relates to: HIVE-8043 Support merging small files [Spark Branch]