[SAMZA-263] Create SystemConsumer and SystemProducer for HDFS

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Jakob Homan
• Reporter: Jakob Homan
• Created: 05/May/14 22:04
• Updated: 31/Mar/15 15:28

Description

It would be nice to be able to read/write from HDFS, particularly for bootstrapping purposes. A few points:

• Per the discussion about leveldb this support should be separated into its own package and project (jar) for easy testing and severability.
• Similar to the Kafka RegexTopicGenerator, we can enumerate (recursively or not) the files in an HDFS directory during job startup.
• Connectivity with HCatalog would be interesting as well, but should be handled in a separate JIRA.

Sub-Tasks

• Add simple HDFS Producer system to Samza: Resolved
• Add HDFS system consumer to Samza: Resolved
• Documentation page for HDFS System Consumer feature: Resolved
• HDFS System Producer should not have Kafka dependency: Resolved
• Support LATEST path and single file as the input of HDFSSystemConsumer: Resolved