[HIVE-14535] add insert-only ACID tables to Hive

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Transactions
• Labels: TODOC3.0
• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 13/Aug/16 02:09
• Updated: 17/Nov/22 08:54
• Resolved: 30/Oct/18 20:15

Description

Design doc:
https://docs.google.com/document/d/1b3t1RywfyRb73-cdvkEzJUyOiekWwkMHdiQ-42zCllY

Feel free to comment.

Update: we ended up going with sequence number based implementation

Update #2: this feature has been partially merged with ACID; the new table type is insert_only ACID, and the difference from the regular ACID is that it only supports inserts on one hand; and that it has no restrictions on file format, table type (bucketing), and much fewer restrictions on other operations (export/import, list bucketing, etc.)
Currently some features that used to work when it was separated are not integrated properly; integration of these features is the remaining work in this JIRA

Issue Links

• duplicates: HIVE-20725 Simultaneous dynamic inserts can result in partition files lost
• is related to: HIVE-16085 get path-to-alias, path-to-work etc recursively in Hive
• is related to: HIVE-17780 MM table followup list

Sub-Tasks

• consider optimizing Utilities::handleMmTableFinalPath: Open
• Bucketed/Sorted tables - SMB join: Open
• Compaction for MM runs Cleaner - needs test once IOW is supported: Open
• JavaUtils.extractTxnId() etc: Patch Available
• CompactorMR.run() should update compaction_queue table for MM: Open
• Miscellaneous List: Open
• collapse union all produced directories into delta directory name suffix for MM: Open
• MM/ACID tables: make tests that will never be compatible with acid use non-txn tables explicitly: Patch Available