[SAMZA-138] System that places specified file contents onto stream

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Labels: feature newbie patch

Environment

RHELinux 2.6.18-371.4.1.el5

• Assignee: Yan Fang
• Reporter: Jonathan Poltak Samosir
• Created: 05/Feb/14 06:27
• Updated: 23/May/14 23:21
• Resolved: 23/May/14 18:19

Description

A fairly straightforward Samza System that reads from a specified file, and places that file's contents onto a SystemStreamPartition for use as input for a StreamTask.

Roughly based off how the hello-samza example project's WikipediaSystem works (more the SystemConsumerFactory rather than SystemConsumer class).

Probably needs a bit of work, but basic functionality works as intended. Hopefully useful to some, either as a functioning system or as a base for a more robust and functionally-promising system that you wish to implement.

Some suggested improvements (not yet implemented):

• handle reading from multiple files (suggested alternative input specification- point 2)
• use of filepos for IncomingMessageEnvelope offset (more info here
• come up with a reasonable bounded queue threshold (the value of 100 was arbitrary, as I was unsure of a reasonable value here)
• better handling for the exceptions encountered (I wasn't 100% sure about some of them)