[HIVE-17206] make a version of Compactor specific to unbucketed tables

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Transactions
• Target Version/s: 3.0.0
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 29/Jul/17 00:48
• Updated: 15/Dec/18 01:01
• Resolved: 15/Dec/18 00:58

Description

current Compactor will work but is not optimized/flexible enough

The current compactor is designed to generate the number of splits equal to the number of buckets in the table. That is the degree of parallelism.

For unbucketed tables, the same is used but the "number of buckets" is derived from the files found in the deltas. For small writes, there will likely be just 1 bucket_00000 file. For large writes, the parallelism of the write determines the number of output files.

Need to make sure Compactor can control parallelism for unbucketed tables as it wishes. For example, hash partition all records (by ROW__ID?) into N disjoint sets.