[FLUME-3084] HDFS Sink - Provide ability to roll the file beyond interval, size and count.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Kallam Reddy
• Created: 12/Apr/17 02:45
• Updated: 16/Apr/17 14:24

Description

Currently we are creating Avro containers using HDFS Sink, and we don't have a fine grained control over when can we roll a file and create new file,
since we are using avro records and containers, whenever the avro version changes we need to roll a new file (close the current file and create new file to roll over to new avro schema).

Suggested changes:
1) Make BucketWriter.java public and make required fields and mthods protected (like, shouldRotate and pass Event to shouldRotate method, so that we can determine based on the event we are parsing currently), so that we can extend that class and modify required changes.
2) In HDFSEventSink.java, make initializeBucketWriter method protected and provide getters to the private properties.