Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.5
    • Fix Version/s: 0.5
    • Component/s: build-tools, oozie
    • Labels:

      Description

      Unable to compile falcon code in Hadoop-2 profile.

      The issue is hadoop 2.4 depends on httpcomponents:httpclient 4.2.5, while hive-10 depends on httpcomponents:httpclient 4.1.3 and the two versions of httpclient aren't compatible and oozie builds with hive-10 by default and falcon builds with hive-13 by default, resulting in both 4.1.3 & 4.2.5 jars of httpclient being in classpath. Result is that the build isn't predictable.

      Errors observed during build:

      Running org.apache.falcon.cli.FalconCLIIT
      2014-09-02 09:49:34.297:WARN::Error for /falcon-webapp/api/options
      java.lang.NoSuchMethodError: org.apache.http.client.utils.URLEncodedUtils.parse(Ljava/lang/String;Ljava/nio/charset/Charset;)Ljava/util/List;
      	at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.getUserName(PseudoAuthenticationHandler.java:124)
      	at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:160)
      	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:357)
      	at org.apache.falcon.security.BasicAuthFilter.doFilter(BasicAuthFilter.java:221)
      	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      
      Running org.apache.falcon.cli.FalconCLIIT
      2014-09-02 16:45:21.367:WARN::Error for /oozie/versions
      java.lang.NoSuchMethodError: org.apache.http.client.utils.URLEncodedUtils.parse(Ljava/lang/String;Ljava/nio/charset/Charset;)Ljava/util/List;
      	at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.getUserName(PseudoAuthenticationHandler.java:124)
      	at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:160)
      	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:357)
      	at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
      

      The issue is apparent in the snippet below when hadoop-webapp is being built the oozie.war is shown to have both versions of hive besides both version of httpcomponents.

      main:
      fileset: Setup scanner in dir /projects/incubator-falcon/hadoop-webapp/target/oozie with patternSet{ includes: [] excludes: [] }
            [jar] Building jar: /projects/incubator-falcon/hadoop-webapp/target/oozie.war
            [jar] adding directory META-INF/
            [jar] adding entry META-INF/MANIFEST.MF
            [jar] adding directory META-INF/maven/
            [jar] adding directory META-INF/maven/org.apache.oozie/
            [jar] adding directory META-INF/maven/org.apache.oozie/oozie-webapp/
            [jar] adding directory WEB-INF/
            [jar] adding directory WEB-INF/classes/
            [jar] adding directory WEB-INF/lib/
      ...
            [jar] adding entry WEB-INF/lib/hive-ant-0.13.0.jar
            [jar] adding entry WEB-INF/lib/hive-builtins-0.10.0.jar
            [jar] adding entry WEB-INF/lib/hive-cli-0.10.0.jar
            [jar] adding entry WEB-INF/lib/hive-cli-0.13.0.jar
            [jar] adding entry WEB-INF/lib/hive-common-0.10.0.jar
            [jar] adding entry WEB-INF/lib/hive-common-0.13.0.jar
            [jar] adding entry WEB-INF/lib/hive-exec-0.10.0.jar
            [jar] adding entry WEB-INF/lib/hive-exec-0.13.0.jar
            [jar] adding entry WEB-INF/lib/hive-hcatalog-core-0.13.0.jar
            [jar] adding entry WEB-INF/lib/hive-metastore-0.10.0.jar
            [jar] adding entry WEB-INF/lib/hive-metastore-0.13.0.jar
            [jar] adding entry WEB-INF/lib/hive-pdk-0.10.0.jar
            [jar] adding entry WEB-INF/lib/hive-serde-0.10.0.jar
            [jar] adding entry WEB-INF/lib/hive-serde-0.13.0.jar
            [jar] adding entry WEB-INF/lib/hive-service-0.10.0.jar
            [jar] adding entry WEB-INF/lib/hive-service-0.13.0.jar
      ...
            [jar] adding entry WEB-INF/lib/httpclient-4.1.3.jar
            [jar] adding entry WEB-INF/lib/httpcore-4.1.3.jar
      ...
      

      The fix is to exclude httpcomponents from hive-metastore, hive-webhcat and oozie-webapp dependencies besides bumping up the version of hive by default when oozie is built.

        Attachments

        1. FALCON-691.patch
          2 kB
          Srikanth Sundarrajan

          Activity

            People

            • Assignee:
              sriksun Srikanth Sundarrajan
              Reporter:
              sriksun Srikanth Sundarrajan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: