Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-4944

PutHiveStreaming multiple instances with Snappy fail intermittently

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: Extensions
    • Labels:
      None

      Description

      When data coming into PutHiveStreaming is compressed with Snappy, then multiple instances of PutHiveStreaming in a flow can cause a failure, the log often shows the following:

      org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d failed to process due to org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {}

      This is due to a race condition in Snappy 1.0.5 (the version used by the Hive NAR) where two classloaders can try to define the native loader class, thus the second one would fail, giving the error above.

      The proposed solution is to guarantee that Snappy is loaded before this situation is encountered (i.e. before the InstanceClassLoaders are created).

        Attachments

          Activity

            People

            • Assignee:
              mattyb149 Matt Burgess
              Reporter:
              mattyb149 Matt Burgess

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment