[HIVE-12403] Too many delta files during Compaction - OOM Part Deux

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.3.0
• Component/s: Transactions
• Target Version/s: 1.3.0
• Assignee: Unassigned
• Reporter: Eugene Koifman
• Created: 13/Nov/15 03:17
• Updated: 07/Jul/16 00:27

Description

HIVE-11540 solution is somewhat incomplete.
Suppose hive.compactor.max.num.delta=10 and there are 1000 delta files to compact.

CompactorMR will currently run 100 jobs on 10 files each to create 100 wider delta files but then it will run 1 job which will include 100 of the wider deltas.

it's a simple fix to make sure the max.num.delta is always respected

Issue Links

• is blocked by: HIVE-11540 Too many delta files during Compaction - OOM
• is related to: HIVE-14179 Too many delta files causes select queries on the table to fail with OOM