Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-1529

hadoop-libraries-nar dependency issue for httpclient and httpcore

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0, 0.5.0
    • Fix Version/s: 0.6.0, 0.5.1
    • Component/s: None
    • Labels:
      None

      Description

      The following issue was reported:

      016-02-17 15:23:11,814 ERROR [Timer-Driven Process Thread-7] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=0575cb57-786c-45d9-b9ef-19c15cc01290] Failed to writ
      e to HDFS due to java.lang.NoClassDefFoundError: org/apache/http/Consts
      2016-02-17 15:23:11,816 ERROR [Timer-Driven Process Thread-7] o.apache.nifi.processors.hadoop.PutHDFS
      java.lang.NoClassDefFoundError: org/apache/http/Consts
              at org.apache.http.client.utils.URIBuilder.digestURI(URIBuilder.java:181) ~[httpclient-4.4.1.jar:4.4.1]
              at org.apache.http.client.utils.URIBuilder.<init>(URIBuilder.java:82) ~[httpclient-4.4.1.jar:4.4.1]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL(KMSClientProvider.java:368) ~[hadoop-common-2.6.2.jar:na]

      From looking into this, we have the following transitive dependencies from the hadoop libraries nar:

      • net.java.dev.jets3t:jets3t:jar:0.9.0:compile
        [INFO] | | +- org.apache.httpcomponents:httpclient:jar:4.4.1:compile
        [INFO] | | +- org.apache.httpcomponents:httpcore:jar:4.1.2:compile

      I believe the 4.4.1 of httpclient is coming from the top level pom which has a dependency management block that forces the client to 4.4.1 and doesn't for core to 4.4.1.

      If we look at httpcore 4.x branch it has the missing Consts class:
      https://github.com/apache/httpcore/tree/4.4.x/httpcore/src/main/java/org/apache/http

      and the 4.1.x branch does not:
      https://github.com/apache/httpcore/tree/4.1.x/httpcore/src/main/java/org/apache/http

      We either need to force both to 4.4.1, or force only the hadoop libraries nar to both be 4.1.2 if that is what it normally brings in.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bende Bryan Bende
                Reporter:
                bende Bryan Bende
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: