Flume
  1. Flume
  2. FLUME-2184

flume-ng-morphline-solr-sink Build failing due to incorrect hadoop-common dependency declaration

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.4.0
    • Fix Version/s: v1.5.0
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Release Note:
      Added exclusion for hadoop common jar dependency in flume-ng-morphline-solr-sink

      Description

      flume-ng-morphline-solr-sink build fails due to incorrect dependency declaration.

      Downloaded the code from flume git repo.

      Trying to build it

      mvn clean install -DskipTests

      It gives me this error.

      The project expects 1.0.1 version of jar which was never there at maven central.

      http://search.maven.org/#search|gav|1|g%3A%22org.apache.hadoop%22%20AND%20a%3A%22hadoop-common%22

      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.1:process (default) on project flume-ng-morphline-solr-sink: Failed to resolve dependencies for one or more projects in the reactor. Reason: Missing:
      [ERROR] ----------
      [ERROR] 1) org.apache.hadoop:hadoop-common:jar:1.0.1
      [ERROR]
      [ERROR] Try downloading the file manually from the project website.

      [ERROR] Path to dependency:
      [ERROR] 1) org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.5.0-SNAPSHOT
      [ERROR] 2) com.cloudera.cdk:cdk-morphlines-all:pom:0.6.0
      [ERROR] 3) com.cloudera.cdk:cdk-morphlines-solr-core:jar:0.6.0
      [ERROR] 4) org.apache.solr:solr-core:jar:4.4.0
      [ERROR] 5) org.apache.hadoop:hadoop-common:jar:1.0.1

      Details here

      http://mail-archives.apache.org/mod_mbox/flume-dev/201309.mbox/%3CCAJ-d8Xep4LcoSE0Yo%3D1w17CewQFzDU%2B5KQDa3DGZDT-oQ3XHYg%40mail.gmail.com%3E

      Attaching patch to fix it.

        Activity

        Hide
        Jagat Singh added a comment -

        Patch to fix dependency for hadoop common

        Show
        Jagat Singh added a comment - Patch to fix dependency for hadoop common
        Hide
        Jagat Singh added a comment -

        Added exclusion for hadoop common jar dependency in flume-ng-morphline-solr-sink

        Show
        Jagat Singh added a comment - Added exclusion for hadoop common jar dependency in flume-ng-morphline-solr-sink
        Hide
        wolfgang hoschek added a comment -

        Looks like the underlying problem is in the top level flume/pom.xml, which currently reads:

        <profile>
          <id>hadoop-1.0</id>
            ...
            <hadoop.version>1.0.1</hadoop.version>
            ...
        </profile>
        
        ...
        
        <profile>
          <id>hadoop-2</id>
            ...
            <hadoop.version>2.0.0-alpha</hadoop.version>
            ...
        </profile>
        
        ...
                  <dependency>
                    <groupId>org.apache.hadoop</groupId>
                    <artifactId>hadoop-client</artifactId>
                    <version>${hadoop.version}</version>
                  </dependency>
        
        

        Also note that the following works fine without your patch: mvn -Dhadoop.profile=2 clean test -DskipTests

        Show
        wolfgang hoschek added a comment - Looks like the underlying problem is in the top level flume/pom.xml, which currently reads: <profile> <id>hadoop-1.0</id> ... <hadoop.version>1.0.1</hadoop.version> ... </profile> ... <profile> <id>hadoop-2</id> ... <hadoop.version>2.0.0-alpha</hadoop.version> ... </profile> ... <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>${hadoop.version}</version> </dependency> Also note that the following works fine without your patch: mvn -Dhadoop.profile=2 clean test -DskipTests
        Hide
        Israel Ekpo added a comment -

        The hadoop-client dependency [1] is fine because 1.0.1 exists for hadoop-client

        It is because version 1.0.1 is not available for hadoop-common [2], that is why we are having the issue

        The dependency tree needs to be adjusted so that the right artifacts are specified, if there are necessary or excluded if the module does not require them.

        [1] http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client
        [2] http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common

        Show
        Israel Ekpo added a comment - The hadoop-client dependency [1] is fine because 1.0.1 exists for hadoop-client It is because version 1.0.1 is not available for hadoop-common [2] , that is why we are having the issue The dependency tree needs to be adjusted so that the right artifacts are specified, if there are necessary or excluded if the module does not require them. [1] http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client [2] http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common
        Hide
        wolfgang hoschek added a comment -

        Ok, thanks!
        +1 to the patch

        Show
        wolfgang hoschek added a comment - Ok, thanks! +1 to the patch
        Hide
        Hari Shreedharan added a comment -

        This looks good. Running tests and committing.

        Show
        Hari Shreedharan added a comment - This looks good. Running tests and committing.
        Hide
        Hari Shreedharan added a comment -

        Committed, rev: 6e461f1. Thanks Jagat!

        Show
        Hari Shreedharan added a comment - Committed, rev: 6e461f1. Thanks Jagat!
        Hide
        Hudson added a comment -

        FAILURE: Integrated in flume-trunk #493 (See https://builds.apache.org/job/flume-trunk/493/)
        FLUME-2184. flume-ng-morphline-solr-sink build failing due to incorrect hadoop-common dependency declaration (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=6e461f13c25a2a2e6c50c1dd06b38b600036ee9f)

        • flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml
        Show
        Hudson added a comment - FAILURE: Integrated in flume-trunk #493 (See https://builds.apache.org/job/flume-trunk/493/ ) FLUME-2184 . flume-ng-morphline-solr-sink build failing due to incorrect hadoop-common dependency declaration (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=6e461f13c25a2a2e6c50c1dd06b38b600036ee9f ) flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml

          People

          • Assignee:
            Jagat Singh
            Reporter:
            Jagat Singh
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development