[FLUME-635] Allow the compression codec to be specified in SequenceFiles

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.4
• Assignee: Disabled imported user
• Reporter: Disabled imported user
• Created: 17/May/11 21:57
• Updated: 19/Aug/11 03:04
• Resolved: 31/May/11 21:08

Description

The dfs sink writes sequence files but doesn't honour flume.collector.dfs.compress.codec. On the other hand, the customdfs sink supports compression (via flume.collector.dfs.compress.codec), but doesn't write SequenceFiles.

This can be fixed by either changing the dfs sink to honour flume.collector.dfs.compress.codec (compression would be enabled by setting io.seqfile.compression.type), or by creating a SequenceFileOutputFormat for use with the customdfs sink.

Issue Links

• blocks: FLUME-632 Support arguments in output format configuration
• is duplicated by: FLUME-506 Sink that writes output to a sequenceFile in hdfs
• is related to: FLUME-641 Support aliases for compression codecs