[HIVE-15899] Make CTAS with acid target table and insert into acid_tbl select ... union all ... work

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Transactions
• Target Version/s: 3.0.0
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 13/Feb/17 22:38
• Updated: 22/May/18 23:59
• Resolved: 20/Sep/17 17:48

Description

Consider:

create table T stored as ORC TBLPROPERTIES('transactional'='true') as
      select a, b from A where a <= 5 union all select a, b from B where a >= 5

and

create table T (a int, b int) stored as ORC  TBLPROPERTIES ('transactional'='false';
insert into T(a,b) select a, b from T where a between 1 and 3 group by a, b union all select a, b from A where a between 5 and 7 union all select a, b from B where a >= 9

On Tez, there is an optimization that removes Union All operator writes the data into
subdirectories of T (in this case T is unpartitioned).

This also happens on MR but requires

hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_OPTIMIZE_UNION_REMOVE, true);
hiveConf.setVar(HiveConf.ConfVars.HIVEFETCHTASKCONVERSION, "none");

Need to ensure that when target table is Acid, we generate unique ROW__IDs
When target is not acid, that we can convert it to Acid via Alter Table even when data layout includes subdirectories.

Issue Links

• is blocked by: HIVE-17204 support un-bucketed tables in acid
• is related to: HIVE-18021 Insert overwrite on acid table with Union All optimizations
• is related to: HIVE-16177 non Acid to acid conversion doesn't handle _copy_N files
• links to: Review Board