[DATAFU-71] Create IncrementalAvroStorage UDF for incrementally processing date partitioned data

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Assignee: Matthew Hayes
• Reporter: Matthew Hayes
• Created: 04/Nov/14 16:31
• Updated: 14/Jan/20 18:42
• Resolved: 14/Jan/20 18:42

Description

Data can sometimes be stored in HDFS in a time-partitioned manner, e.g. /some/input/yyyy/mm/dd. You may want to process this data incrementally, where the output has a format like /some/output/yyyy/mm/dd. It be useful if there is a UDF that handles the incremental processing for you.