[NIFI-3191] HDFS Processors Should Allow Choosing LZO Compression

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0, 0.7.1
• Fix Version/s: 1.3.0
• Assignee: Pierre Villard
• Reporter: Bryan Bende
• Created: 12/Dec/16 22:43
• Updated: 24/May/17 15:04
• Resolved: 24/May/17 15:04

Description

The HDFS processors have a CompressionType enum that maps the enum to a class name, currently it has the following:

case NONE: return "NONE";
case DEFAULT: return DefaultCodec.class.getName();
case BZIP: return BZip2Codec.class.getName();
case GZIP: return GzipCodec.class.getName();
case LZ4: return Lz4Codec.class.getName();
case SNAPPY: return SnappyCodec.class.getName();
case AUTOMATIC: return "Automatically Detected";

The Lzo codec class is GPL (http://code.google.com/p/hadoop-gpl-compression and https://github.com/twitter/hadoop-lzo) so it can not be shipped with NiFi or Hadoop, but if someone adds it to their Hadoop distribution there should be a way to select it from NiFi.

Seems LZO could be added to the enumeration with a mapping of:

case LZO: return "com.hadoop.compression.lzo.LzoCodec";

The libraries would still need to be linked through the LD_LIBRARY_PATH such as:

java.arg.15=-Djava.libaray.path=/path/to/your/lzocodec.so

and possibly point the additional classpath resource to the jar containing LzoCodec.java.

Issue Links

• links to: GitHub Pull Request #1802