Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-3197

Unable to use Snappy Compression Codec on PutHDFS

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 1.0.0, 1.1.0
    • Fix Version/s: None
    • Component/s: None
    • Environment:
      centos 7.2
      NiFi 1.1.0 or NiFi 1.0.0
      HDP 2.4.3.0

      Description

      When setting the 'Compression code' to SNAPPY on PutHDFS NiFi is unable to compress and push data to HDFS. Attached is the sample workflow that is having trouble. Below are the errors for NiFi 1.0.0 and the errors for NiFi 1.1.0. Same configuration, but the error is a bit different.

      I have tried setting both LD_LIBRARY_PATH as an environment variable, and then I tried setting java.library.path in the bootstrap.conf.

      NiFi 1.1.0 nifi-app.log error message:

      2016-12-14 15:13:56,563 ERROR [Timer-Driven Process Thread-6] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=fdd52005-0158-1000-ac0f-2a87ed12a1e7] Failed to write to HDFS due to java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
      2016-12-14 15:13:56,644 ERROR [Timer-Driven Process Thread-6] o.apache.nifi.processors.hadoop.PutHDFS
      java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
              at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65) ~[hadoop-common-2.7.3.jar:na]
              at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134) ~[hadoop-common-2.7.3.jar:na]
              at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150) ~[hadoop-common-2.7.3.jar:na]
              at org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131) ~[hadoop-common-2.7.3.jar:na]
              at org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:100) ~[hadoop-common-2.7.3.jar:na]
              at org.apache.nifi.processors.hadoop.PutHDFS$1$1.process(PutHDFS.java:313) ~[nifi-hdfs-processors-1.1.0.jar:1.1.0]
              at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2082) ~[na:na]
              at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2053) ~[na:na]
              at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:300) ~[nifi-hdfs-processors-1.1.0.jar:1.1.0]
              at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_111-debug]
              at javax.security.auth.Subject.doAs(Subject.java:360) [na:1.8.0_111-debug]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678) [hadoop-common-2.7.3.jar:na]
              at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230) [nifi-hdfs-processors-1.1.0.jar:1.1.0]
              at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.jar:1.1.0]
              at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.jar:1.1.0]
              at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.jar:1.1.0]
              at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.jar:1.1.0]
              at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.jar:1.1.0]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111-debug]
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_111-debug]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_111-debug]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_111-debug]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111-debug]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111-debug]
              at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111-debug]
      

      NiFi 1.0.0 nifi-bootstrap.log

      2016-12-14 15:32:50,373 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 40280
      2016-12-14 15:43:47,752 INFO [NiFi logging handler] org.apache.nifi.StdOut FATAL ERROR in native method: JNI received a class argument that is not a class
      2016-12-14 15:43:47,758 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native Method)
      2016-12-14 15:43:47,759 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:232)
      2016-12-14 15:43:47,759 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:149)
      2016-12-14 15:43:47,760 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:142)
      2016-12-14 15:43:47,761 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.hadoop.io.compress.CompressorStream.close(CompressorStream.java:106)
      2016-12-14 15:43:47,761 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.processors.hadoop.PutHDFS$1.process(PutHDFS.java:295)
      2016-12-14 15:43:47,761 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1880)
      2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851)
      2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:276)
      2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
      2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
      2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
      2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
      2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut      at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
      2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut      at java.lang.Thread.run(Thread.java:745)
      2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut Current thread is 139782124459776
      2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut Dumping core ...
      2016-12-14 15:43:48,525 WARN [main] org.apache.nifi.bootstrap.RunNiFi Apache NiFi appears to have died. Restarting...
      2016-12-14 15:43:48,545 INFO [main] org.apache.nifi.bootstrap.RunNiFi Successfully started Apache NiFi with PID 27342
      2016-12-14 15:43:52,874 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 43961
      2016-12-14 15:44:09,020 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: []
      2016-12-14 15:44:09,053 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED
      2016-12-14 15:44:09,054 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED
      2016-12-14 15:44:09,054 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED
      2016-12-14 15:44:09,231 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi has accepted the Shutdown Command and is shutting down now
      

        Attachments

        1. native-directory.png
          127 kB
          Josh Meyer
        2. processor-config.png
          189 kB
          Josh Meyer
        3. nifi-puthdfs-snappy-issue.xml
          17 kB
          Josh Meyer

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jjmeyer0 Josh Meyer
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: