[AVRO-1862] AvroOutputFormat saves compressed avrò files without respecting codec's default extension

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.8.1
• Fix Version/s: 1.9.0
• Component/s: java
• Labels: patch
• Release Note: avro.mapred.output.extension.from-codec (boolean) MapReduce Job property that allows change extension of output files from .avro to .$compressionCodec.avro
• Flags: Patch
• Assignee: Unassigned
• Reporter: Piotr Wikieł
• Created: 15/Jun/16 07:27
• Updated: 03/May/21 18:49
• Resolved: 27/Feb/20 14:19

Description

Common pattern in naming compressed files is giving them extension derived from compression codec, for example: .gz, .zip, .bz2.
AvroOutputFormat currently does not respect this convention.

I've adapted some code from Hadoop's TextOutputFormat in backward-compatible manner adding following JobConf property:

avro.mapred.output.extension.from-codec (boolean, default: false) - when set to true, extension will be changed according to above rule.

EDIT: Please take a look at first comment for an update. .gz.avro, .snappy.avro will be an extension of the file when above property will be set to true.