[HIVE-12077] MSCK Repair table should fix partitions in batches

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: Hive
• Assignee: Chinna Rao Lalam
• Reporter: Ryan P
• Created: 09/Oct/15 12:37
• Updated: 26/Jul/17 03:30
• Resolved: 01/Aug/16 09:36

Description

If a user attempts to run MSCK REPAIR TABLE on a directory with a large number of untracked partitions HMS will OOME. I suspect this is because it attempts to do one large bulk load in an effort to save time. Ultimately this can lead to a collection so large in size that HMS eventually hits an Out of Memory Exception.

Instead I suggest that Hive include a configurable batch size that HMS can use to break up the load.

Issue Links

• Is contained by: HIVE-12859 MSCK Repair table gives error for higher number of partitions
• is related to: HIVE-14571 Document configuration hive.msck.repair.batch.size
• relates to: HIVE-14693 Some partitions will be left out when partition number is a multiple of the option hive.msck.repair.batch.size