[HIVE-14035] Enable predicate pushdown to delta files created by ACID Transactions

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: Transactions
• Labels: TODOC2.2
• Target Version/s: 2.2.0
• Assignee: Saket Saurabh
• Reporter: Saket Saurabh
• Created: 16/Jun/16 18:12
• Updated: 19/Feb/18 23:51
• Resolved: 12/Aug/16 23:42

Description

In current Hive version, delta files created by ACID transactions do not allow predicate pushdown if they contain any update/delete events. This is done to preserve correctness when following a multi-version approach during event collapsing, where an update event overwrites an existing insert event.
This JIRA proposes to split an update event into a combination of a delete event followed by a new insert event, that can enable predicate push down to all delta files without breaking correctness. To support backward compatibility for this feature, this JIRA also proposes to add some sort of versioning to ACID that can allow different versions of ACID transactions to co-exist together.

Issue Links

• is related to: HIVE-14878 integrate MM tables into ACID: add separate ACID type
• is related to: HIVE-14448 Queries with predicate fail when ETL split strategy is chosen for ACID tables
• is related to: HIVE-16832 duplicate ROW__ID possible in multi insert into transactional table
• is related to: SPARK-15348 Hive ACID
• is related to: HIVE-17458 VectorizedOrcAcidRowBatchReader doesn't handle 'original' files
• is required by: HIVE-14667 Eliminate OrcRecordUpdate.OPERATION
• is required by: HIVE-14607 ORC split generation failed with exception: java.lang.ArrayIndexOutOfBoundsException: 1
• is required by: HIVE-12631 LLAP IO: support ORC ACID tables
• is required by: HIVE-14516 OrcInputFormat.SplitGenerator.callInternal() can be optimized
• is required by: HIVE-14233 Improve vectorization for ACID by eliminating row-by-row stitching
• is required by: HIVE-17204 support un-bucketed tables in acid
• is required by: HIVE-17089 make acid 2.0 the default
• is required by: HIVE-14199 Enable Bucket Pruning for ACID tables
• is required by: HIVE-15988 LLAP: cache ORC ACID delete deltas
• is required by: HIVE-14534 modify tables in tests in HIVE-14479 to use transactional_properties=default
• relates to: HIVE-16669 Fine tune Compaction to take advantage of Acid 2.0
• relates to: HIVE-16830 delete delta buckets don't have maxLength set
• relates to: HIVE-14514 OrcRecordUpdater should clone writerOptions when creating delete event writers
• links to: ReviewBoard # 49766