Flume
  1. Flume
  2. FLUME-2184

flume-ng-morphline-solr-sink Build failing due to incorrect hadoop-common dependency declaration

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.4.0
    • Fix Version/s: v1.5.0
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Release Note:
      Added exclusion for hadoop common jar dependency in flume-ng-morphline-solr-sink

      Description

      flume-ng-morphline-solr-sink build fails due to incorrect dependency declaration.

      Downloaded the code from flume git repo.

      Trying to build it

      mvn clean install -DskipTests

      It gives me this error.

      The project expects 1.0.1 version of jar which was never there at maven central.

      http://search.maven.org/#search|gav|1|g%3A%22org.apache.hadoop%22%20AND%20a%3A%22hadoop-common%22

      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.1:process (default) on project flume-ng-morphline-solr-sink: Failed to resolve dependencies for one or more projects in the reactor. Reason: Missing:
      [ERROR] ----------
      [ERROR] 1) org.apache.hadoop:hadoop-common:jar:1.0.1
      [ERROR]
      [ERROR] Try downloading the file manually from the project website.

      [ERROR] Path to dependency:
      [ERROR] 1) org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.5.0-SNAPSHOT
      [ERROR] 2) com.cloudera.cdk:cdk-morphlines-all:pom:0.6.0
      [ERROR] 3) com.cloudera.cdk:cdk-morphlines-solr-core:jar:0.6.0
      [ERROR] 4) org.apache.solr:solr-core:jar:4.4.0
      [ERROR] 5) org.apache.hadoop:hadoop-common:jar:1.0.1

      Details here

      http://mail-archives.apache.org/mod_mbox/flume-dev/201309.mbox/%3CCAJ-d8Xep4LcoSE0Yo%3D1w17CewQFzDU%2B5KQDa3DGZDT-oQ3XHYg%40mail.gmail.com%3E

      Attaching patch to fix it.

        Activity

        Jagat Singh created issue -
        Hide
        Jagat Singh added a comment -

        Patch to fix dependency for hadoop common

        Show
        Jagat Singh added a comment - Patch to fix dependency for hadoop common
        Jagat Singh made changes -
        Field Original Value New Value
        Attachment FLUME-2184-0.patch [ 12602031 ]
        Hide
        Jagat Singh added a comment -

        Added exclusion for hadoop common jar dependency in flume-ng-morphline-solr-sink

        Show
        Jagat Singh added a comment - Added exclusion for hadoop common jar dependency in flume-ng-morphline-solr-sink
        Jagat Singh made changes -
        Release Note Added exclusion for hadoop common jar dependency in flume-ng-morphline-solr-sink
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        wolfgang hoschek added a comment -

        Looks like the underlying problem is in the top level flume/pom.xml, which currently reads:

        <profile>
          <id>hadoop-1.0</id>
            ...
            <hadoop.version>1.0.1</hadoop.version>
            ...
        </profile>
        
        ...
        
        <profile>
          <id>hadoop-2</id>
            ...
            <hadoop.version>2.0.0-alpha</hadoop.version>
            ...
        </profile>
        
        ...
                  <dependency>
                    <groupId>org.apache.hadoop</groupId>
                    <artifactId>hadoop-client</artifactId>
                    <version>${hadoop.version}</version>
                  </dependency>
        
        

        Also note that the following works fine without your patch: mvn -Dhadoop.profile=2 clean test -DskipTests

        Show
        wolfgang hoschek added a comment - Looks like the underlying problem is in the top level flume/pom.xml, which currently reads: <profile> <id>hadoop-1.0</id> ... <hadoop.version>1.0.1</hadoop.version> ... </profile> ... <profile> <id>hadoop-2</id> ... <hadoop.version>2.0.0-alpha</hadoop.version> ... </profile> ... <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>${hadoop.version}</version> </dependency> Also note that the following works fine without your patch: mvn -Dhadoop.profile=2 clean test -DskipTests
        Hide
        Israel Ekpo added a comment -

        The hadoop-client dependency [1] is fine because 1.0.1 exists for hadoop-client

        It is because version 1.0.1 is not available for hadoop-common [2], that is why we are having the issue

        The dependency tree needs to be adjusted so that the right artifacts are specified, if there are necessary or excluded if the module does not require them.

        [1] http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client
        [2] http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common

        Show
        Israel Ekpo added a comment - The hadoop-client dependency [1] is fine because 1.0.1 exists for hadoop-client It is because version 1.0.1 is not available for hadoop-common [2] , that is why we are having the issue The dependency tree needs to be adjusted so that the right artifacts are specified, if there are necessary or excluded if the module does not require them. [1] http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client [2] http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common
        Hide
        wolfgang hoschek added a comment -

        Ok, thanks!
        +1 to the patch

        Show
        wolfgang hoschek added a comment - Ok, thanks! +1 to the patch
        Hide
        Hari Shreedharan added a comment -

        This looks good. Running tests and committing.

        Show
        Hari Shreedharan added a comment - This looks good. Running tests and committing.
        Hari Shreedharan made changes -
        Assignee Jagat Singh [ jagatsingh ]
        Hide
        Hari Shreedharan added a comment -

        Committed, rev: 6e461f1. Thanks Jagat!

        Show
        Hari Shreedharan added a comment - Committed, rev: 6e461f1. Thanks Jagat!
        Hari Shreedharan made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        FAILURE: Integrated in flume-trunk #493 (See https://builds.apache.org/job/flume-trunk/493/)
        FLUME-2184. flume-ng-morphline-solr-sink build failing due to incorrect hadoop-common dependency declaration (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=6e461f13c25a2a2e6c50c1dd06b38b600036ee9f)

        • flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml
        Show
        Hudson added a comment - FAILURE: Integrated in flume-trunk #493 (See https://builds.apache.org/job/flume-trunk/493/ ) FLUME-2184 . flume-ng-morphline-solr-sink build failing due to incorrect hadoop-common dependency declaration (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=6e461f13c25a2a2e6c50c1dd06b38b600036ee9f ) flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml
        Hari Shreedharan made changes -
        Affects Version/s v1.3.1 [ 12323879 ]
        Affects Version/s v1.5.0 [ 12324642 ]
        Fix Version/s v1.5.0 [ 12324642 ]
        Hari Shreedharan made changes -
        Affects Version/s v1.4.0 [ 12323372 ]
        Affects Version/s v1.3.1 [ 12323879 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        47m 18s 1 Jagat Singh 08/Sep/13 04:35
        Patch Available Patch Available Resolved Resolved
        3d 19h 46m 1 Hari Shreedharan 12/Sep/13 00:22

          People

          • Assignee:
            Jagat Singh
            Reporter:
            Jagat Singh
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development