Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-263

Create SystemConsumer and SystemProducer for HDFS

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      It would be nice to be able to read/write from HDFS, particularly for bootstrapping purposes. A few points:

      • Per the discussion about leveldb this support should be separated into its own package and project (jar) for easy testing and severability.
      • Similar to the Kafka RegexTopicGenerator, we can enumerate (recursively or not) the files in an HDFS directory during job startup.
      • Connectivity with HCatalog would be interesting as well, but should be handled in a separate JIRA.

        Attachments

        There are no Sub-Tasks for this issue.

          Activity

            People

            • Assignee:
              jghoman Jakob Homan
              Reporter:
              jghoman Jakob Homan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: