[STREAMS-207] Customize metadata fields written/read to/from HDFS/S3

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Assignee: Steve Blackmon
• Reporter: Steve Blackmon
• Created: 03/Nov/14 17:06
• Updated: 08/Jul/18 16:36
• Resolved: 13/Feb/17 23:49

Description

These persisters currently expect a 4-column format. It should be possible via configuration to omit one or more of the four principle fields (id, timestamp, metadata, document) and have the reader and writer operations succeed. The default behavior should remain write and read of all four fields.

This will allow streams to be constructed from files containing only ids, only documents, or only metadata without processor-based workarounds.