[CAMEL-4555] Support nested directories with multiple segment files in the HDFS endpoint consumer

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.1
• Fix Version/s: 2.8.3, 2.9.0
• Component/s: camel-hdfs
• Labels: hdfs
• Patch Info: Patch Available
• Estimated Complexity: Moderate
• Assignee: Claus Ibsen
• Reporter: Ben Hoyt
• Created: 17/Oct/11 18:50
• Updated: 14/Nov/11 19:27
• Resolved: 22/Oct/11 14:02

Description

A common pattern in HDFS is for multiple segment files underneath a given directory, representing the fragments of data. Lots of tools understand to automatically merge these segment files (ie hadoop fs -getmerge, pig script loaders). This patch does the same for the HDFS consumer, using a temporary local directory for the merging.

Additionally, tools like pig and oozie understand to look for a _SUCCESS file in one of these directories containing segments. This file indicates that the segments have been completely written. This patch additionally skips the directory if a _SUCCESS file is not present.