[FLUME-2128] HDFS Sink rollSize is calculated based off of uncompressed size of cumulative events.

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.4.0, 1.3.1
• Component/s: Sinks+Sources
• Labels: features
• Release Note: I made the fix to trunk
• Assignee: Theodore michael Malaska
• Reporter: Jeff Lord
• Created: 17/Jul/13 15:43
• Updated: 11/Mar/14 18:21

Description

The hdfs sink rollSize parameter is compared against uncompressed event sizes.
The net of this is that if you are using compression and expect the size of your files on HDFS to be rolled/sized based on the value set for rollSize than your files will be much smaller due to compression.

We should take into account when compression is set and roll based on the compressed size on hdfs.

Issue Links

• links to: Review