[HIVE-16722] Converting bucketed non-acid table to acid should perform validation

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 3.0.0
• Component/s: Transactions
• Target Version/s: 3.0.0
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 19/May/17 20:58
• Updated: 01/Oct/19 22:07
• Resolved: 26/Oct/17 00:10

Description

Converting a non acid table to acid only performs metadata validation (in TransactionalValidationListener).
The data read code path only understands certain directory layouts and file names and ignores (generally) files that don't match the expected format.

In Hive, directory layout and bucket file naming (especially older releases) is poorly enforced.

Need to add a validation step on

alter table T SET TBLPROPERTIES ('transactional'='true')

to
scan the file system and report any possible data loss scenarios.

Currently Acid understands bucket files name like "00000_0" and (with HIVE-16177) 00000_0_copy1" etc at the root of the partition.

Issue Links

• relates to: HIVE-16688 Make sure Alter Table to set transaction=true acquires X lock