[FALCON-143] Enable Late data handling for hive tables

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.3
• Fix Version/s: 0.4
• Assignee: Venkatesh Seetharam
• Reporter: Venkatesh Seetharam
• Created: 08/Oct/13 18:39
• Updated: 16/Jan/15 08:21
• Resolved: 01/Nov/13 23:49

Description

HCat nor Hive APIs expose internal stats about a given partition. The only way to get the partition size is to get the location of the partition on HDFS and then use globStatus and contentSummary APIs.

With the addition of HIVE-5317, this is going to get more complicated with deltas and minor and major compactions with no locking.

Need to work with hive to see if there will be an API or Falcon needs to understand the structure of the layout of the data on the file system.

Issue Links

• depends upon: FALCON-95 Enable embedding hive scripts directly in a process
• is depended upon by: FALCON-161 Feed evictor evicts instances not eligible when pattern has dash
• relates to: FALCON-996 Update the "Handling late input data" section in the documentation