[CRUNCH-36] Avro Compression settings are ignored by Crunch

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.3.0
• Fix Version/s: 0.3.0
• Component/s: IO
• Assignee: Unassigned
• Reporter: Joseph Adler
• Created: 07/Aug/12 22:50
• Updated: 17/Sep/12 06:41
• Resolved: 08/Aug/12 04:49

Description

Java Avro Map/reduce jobs will compress the output if the configuration setting mapred.output.compress is true. Avro supports two codecs (deflate and snappy). Users typically select the code by calling AvroJob.setOutputCodec and specifying either org.apache.avro.file.DataFileConstants.SNAPPY_CODEC or org.apache.avro.file.DataFileConstants.DEFLATE_CODEC. Users can specify the deflate level by setting the configuration setting avro.mapred.deflate.level.

org.apache.crunch.types.avro.AvroOutuptFormat currently ignores these setting.

I'll follow up with a patch that fixes this problem.