[HIVE-13403] Make Streaming API not create empty buckets

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 2.3.0
• Component/s: HCatalog, Transactions
• Target Version/s: 2.2.0
• Assignee: Wei Zheng
• Reporter: Eugene Koifman
• Created: 31/Mar/16 21:11
• Updated: 27/Feb/24 22:23
• Resolved: 24/Aug/16 23:37

Description

as of HIVE-11983, when a TransactionBatch is opened in StreamingAPI, a full compliment of bucket files (AbstractRecordWriter.createRecordUpdaters()) is created on disk even though some may end up receiving no data.

It would be better to create them on demand and not clog the FS.

Tez can handle missing (empty) buckets and on MR bucket join algorithms will check if all buckets are there and bail out if not.

Issue Links

• is related to: HIVE-11719 acid insert with dynamic partitioning doesn't create empty buckets
• is related to: HIVE-17138 FileSinkOperator/Compactor doesn't create empty files for acid path
• relates to: HIVE-11983 Hive streaming API uses incorrect logic to assign buckets to incoming records