[HCATALOG-487] HCatalog should tolerate a user-defined amount of bad records

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5
• Assignee: Travis Crawford
• Reporter: Travis Crawford
• Created: 30/Aug/12 22:42
• Updated: 15/Feb/13 21:32
• Resolved: 31/Aug/12 15:20

Description

HCatalog tasks currently fail when deserializing corrupt records. In some cases, large data sets have a small number of corrupt records and its okay to skip them. In fact Hadoop has support for skipping bad records for exactly this reason.

However, using the Hadoop-native record skipping feature (like Hive does) is very coarse and leads to a large number of failed tasks, task scheduling overhead, and limited control over the skipping behavior.

HCatalog should have native support for skipping a user-defined amount of bad records.