[HIVE-16177] non Acid to acid conversion doesn't handle _copy_N files

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 2.4.0, 3.0.0
• Component/s: Transactions
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 10/Mar/17 20:51
• Updated: 22/May/18 23:59
• Resolved: 12/Jul/17 17:37

Description

create table T(a int, b int) clustered by (a)  into 2 buckets stored as orc TBLPROPERTIES('transactional'='false')
insert into T(a,b) values(1,2)
insert into T(a,b) values(1,3)
alter table T SET TBLPROPERTIES ('transactional'='true')

//we should now have bucket files 000001_0 and 000001_0_copy_1

but OrcRawRecordMerger.OriginalReaderPair.next() doesn't know that there can be copy_N files and numbers rows in each bucket from 0 thus generating duplicate IDs

select ROW__ID, INPUT__FILE__NAME, a, b from T

produces

{"transactionid":0,"bucketid":1,"rowid":0},file:/Users/ekoifman/dev/hiverwgit/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands.../warehouse/nonacidorctbl/000001_0,1,2
{"transactionid\":0,"bucketid":1,"rowid":0},file:/Users/ekoifman/dev/hiverwgit/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands.../warehouse/nonacidorctbl/000001_0_copy_1,1,3

owen.omalley, do you have any thoughts on a good way to handle this?

attached patch has a few changes to make Acid even recognize copy_N but this is just a pre-requisite. The new UT demonstrates the issue.

Futhermore,

alter table T compact 'major'
select ROW__ID, INPUT__FILE__NAME, a, b from T order by b

produces

{"transactionid":0,"bucketid":1,"rowid":0}	file:/Users/ekoifman/dev/hiverwgit/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands....warehouse/nonacidorctbl/base_-9223372036854775808/bucket_00001	1	2

HIVE-16177.04.patch has TestTxnCommands.testNonAcidToAcidConversion0() demonstrating this

This is because compactor doesn't handle copy_N files either (skips them)

Issue Links

• blocks: HIVE-17069 Refactor OrcRawRecrodMerger.ReaderPair
• is related to: HIVE-17526 Disable conversion to ACID if table has _copy_N files on branch-1
• is related to: HIVE-16732 Transactional tables should block LOAD DATA
• relates to: HIVE-15899 Make CTAS with acid target table and insert into acid_tbl select ... union all ... work
• relates to: HIVE-12724 ACID: Major compaction fails to include the original bucket files into MR job
• relates to: HIVE-13961 ACID: Major compaction fails to include the original bucket files if there's no delta directory
• relates to: HIVE-14366 Conversion of a Non-ACID table to an ACID table produces non-unique primary keys
• relates to: HIVE-11525 Bucket pruning
• requires: HIVE-16964 _orc_acid_version file is missing
• links to: Review Board

Sub-Tasks

• Make sure Alter Table to set transaction=true acquires X lock: Closed
• Converting bucketed non-acid table to acid should perform validation: Closed