[HIVE-7870] Insert overwrite table query does not generate correct task plan [Spark Branch]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Spark
• Labels: Spark-M1
• Assignee: Na Yang
• Reporter: Na Yang
• Created: 25/Aug/14 16:30
• Updated: 29/May/15 02:27
• Resolved: 16/Sep/14 14:35

Description

Insert overwrite table query does not generate correct task plan when hive.optimize.union.remove and hive.merge.sparkfiles properties are ON.

set hive.optimize.union.remove=true
set hive.merge.sparkfiles=true

insert overwrite table outputTbl1
SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
  SELECT key, count(1) as values from inputTbl1 group by key
  UNION ALL
  SELECT key, 2 as values from inputTbl1
) a
)b;
select * from outputTbl1 order by key, values;

query result

1	1
1	2
2	1
2	2
3	1
3	2
7	1
7	2
8	2
8	2
8	2

expected result:

1	1
1	1
1	2
2	1
2	1
2	2
3	1
3	1
3	2
7	1
7	1
7	2
8	1
8	1
8	2
8	2
8	2

Move work is not working properly and some data are missing during move.

Issue Links

• is part of: HIVE-7292 Hive on Spark
• is related to: HIVE-7810 Insert overwrite table query has strange behavior when set hive.optimize.union.remove=true [Spark Branch]