[NIFI-3197] Unable to use Snappy Compression Codec on PutHDFS

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 1.0.0, 1.1.0
• Labels: HDFS Processor PutHDFS Snappy
• Environment: centos 7.2; NiFi 1.1.0 or NiFi 1.0.0; HDP 2.4.3.0
• Assignee: Unassigned
• Reporter: Josh Meyer
• Created: 14/Dec/16 16:24
• Updated: 14/Dec/16 22:16
• Resolved: 14/Dec/16 22:16

Description

When setting the 'Compression code' to SNAPPY on PutHDFS NiFi is unable to compress and push data to HDFS. Attached is the sample workflow that is having trouble. Below are the errors for NiFi 1.0.0 and the errors for NiFi 1.1.0. Same configuration, but the error is a bit different.

I have tried setting both LD_LIBRARY_PATH as an environment variable, and then I tried setting java.library.path in the bootstrap.conf.

NiFi 1.1.0 nifi-app.log error message:

2016-12-14 15:13:56,563 ERROR [Timer-Driven Process Thread-6] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=fdd52005-0158-1000-ac0f-2a87ed12a1e7] Failed to write to HDFS due to java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
2016-12-14 15:13:56,644 ERROR [Timer-Driven Process Thread-6] o.apache.nifi.processors.hadoop.PutHDFS
java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
        at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65) ~[hadoop-common-2.7.3.jar:na]
        at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134) ~[hadoop-common-2.7.3.jar:na]
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150) ~[hadoop-common-2.7.3.jar:na]
        at org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131) ~[hadoop-common-2.7.3.jar:na]
        at org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:100) ~[hadoop-common-2.7.3.jar:na]
        at org.apache.nifi.processors.hadoop.PutHDFS$1$1.process(PutHDFS.java:313) ~[nifi-hdfs-processors-1.1.0.jar:1.1.0]
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2082) ~[na:na]
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2053) ~[na:na]
        at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:300) ~[nifi-hdfs-processors-1.1.0.jar:1.1.0]
        at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_111-debug]
        at javax.security.auth.Subject.doAs(Subject.java:360) [na:1.8.0_111-debug]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678) [hadoop-common-2.7.3.jar:na]
        at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230) [nifi-hdfs-processors-1.1.0.jar:1.1.0]
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.jar:1.1.0]
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.jar:1.1.0]
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.jar:1.1.0]
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.jar:1.1.0]
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.jar:1.1.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111-debug]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_111-debug]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_111-debug]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_111-debug]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111-debug]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111-debug]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111-debug]

NiFi 1.0.0 nifi-bootstrap.log

2016-12-14 15:32:50,373 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 40280
2016-12-14 15:43:47,752 INFO [NiFi logging handler] org.apache.nifi.StdOut FATAL ERROR in native method: JNI received a class argument that is not a class
2016-12-14 15:43:47,758 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native Method)
2016-12-14 15:43:47,759 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:232)
2016-12-14 15:43:47,759 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:149)
2016-12-14 15:43:47,760 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:142)
2016-12-14 15:43:47,761 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.hadoop.io.compress.CompressorStream.close(CompressorStream.java:106)
2016-12-14 15:43:47,761 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.processors.hadoop.PutHDFS$1.process(PutHDFS.java:295)
2016-12-14 15:43:47,761 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1880)
2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851)
2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:276)
2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.lang.Thread.run(Thread.java:745)
2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut Current thread is 139782124459776
2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut Dumping core ...
2016-12-14 15:43:48,525 WARN [main] org.apache.nifi.bootstrap.RunNiFi Apache NiFi appears to have died. Restarting...
2016-12-14 15:43:48,545 INFO [main] org.apache.nifi.bootstrap.RunNiFi Successfully started Apache NiFi with PID 27342
2016-12-14 15:43:52,874 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 43961
2016-12-14 15:44:09,020 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: []
2016-12-14 15:44:09,053 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED
2016-12-14 15:44:09,054 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED
2016-12-14 15:44:09,054 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED
2016-12-14 15:44:09,231 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi has accepted the Shutdown Command and is shutting down now