[HIVE-9381] HCatalog hardcodes maximum append limit to 1000.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: HCatalog
• Release Note: Introduces new hcat jobconf parameter : hcat.append.limit for those that want to increase the hcatalog append limit above 1000.
• Assignee: Sushanth Sowmyan
• Reporter: Sushanth Sowmyan
• Created: 14/Jan/15 22:41
• Updated: 12/Feb/15 23:40
• Resolved: 16/Jan/15 19:52

Description

When HCatalog appends to a table, it tries to avoid fileclashes while moving by appending a "_a_NNN" suffix to a file if there is a clash, where the NNN is a number from 0 to 999.

This limit is hardcoded and this causes issues with some users who have tables with a large number of files. Ideally, we should push back on those users to change their usecase, or run some manner of compactions on their table (a CREATE TABLE AS SELECT from this table to another table, followed by a drop and re creating as select from the other table would suffice, for example). But, for users where those are not viable solutions, we need to help them get unstuck. One way is by making that hardcoded parameter a configuration parameter.

Issue Links

• relates to: HIVE-6332 HCatConstants Documentation needed