HBase
  1. HBase
  2. HBASE-8488

HBase transitive dependencies not being pulled in when building apps like Flume which depend on HBase

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.95.0
    • Fix Version/s: 0.98.0, 0.95.2
    • Component/s: build
    • Labels:
      None

      Description

      Here is a snippet of the errors seen when building against Hbase....

      [WARNING] Invalid POM for org.apache.hbase:hbase-common:jar:0.97.0-SNAPSHOT, transitive dependencies (if any) will not be available, enable debug logging for more details: Some problems were encountered while processing the POMs:
      [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for org.apache.hbase:${compat.module}:jar with value '${compat.module}' does not match a valid id pattern. @ org.apache.hbase:hbase:0.97.0-SNAPSHOT, /Users/rnaik/.m2/repository/org/apache/hbase/hbase/0.97.0-SNAPSHOT/hbase-0.97.0-SNAPSHOT.pom, line 982, column 21
      [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for org.apache.hbase:${compat.module}:test-jar with value '${compat.module}' does not match a valid id pattern. @ org.apache.hbase:hbase:0.97.0-SNAPSHOT, /Users/rnaik/.m2/repository/org/apache/hbase/hbase/0.97.0-SNAPSHOT/hbase-0.97.0-SNAPSHOT.pom, line 987, column 21
      

        Issue Links

          Activity

          Roshan Naik created issue -
          Hide
          Roshan Naik added a comment -

          The problem initially appears to be indicate that $

          {compat.module}

          property is not defined .. although it actually is (inside the hadoop 1/2 profiles in parent pom). But these are unfortunately not visible when building apps that depend on hbase.

          Show
          Roshan Naik added a comment - The problem initially appears to be indicate that $ {compat.module} property is not defined .. although it actually is (inside the hadoop 1/2 profiles in parent pom). But these are unfortunately not visible when building apps that depend on hbase.
          Hide
          stack added a comment -

          If you define it on the command line when building, will that work?

          How are you refering to hbase in your pom? Are you trying to build against the SNAPSHOTS for hadoop1 and hadoop2?

          Thanks.

          Show
          stack added a comment - If you define it on the command line when building, will that work? How are you refering to hbase in your pom? Are you trying to build against the SNAPSHOTS for hadoop1 and hadoop2? Thanks.
          Hide
          Enis Soztutar added a comment -

          I think the problem kicks in because when you refer the installed pom as a dependency, then no profile kicks in, and hence compat.module and the runtime dependency cannot be resolved. We did some testing with roshan to add hadoop1|2-compat modules as normal modules (not as defined inside profiles), but did not do change the hbase-server's dependency on compat as a runtime dependency.

          Show
          Enis Soztutar added a comment - I think the problem kicks in because when you refer the installed pom as a dependency, then no profile kicks in, and hence compat.module and the runtime dependency cannot be resolved. We did some testing with roshan to add hadoop1|2-compat modules as normal modules (not as defined inside profiles), but did not do change the hbase-server's dependency on compat as a runtime dependency.
          Hide
          Enis Soztutar added a comment -

          Turns out the way we do compat modules, we cannot do depend on the hadoop1|2 compat only on runtime unless we use reflection, or add hadoop dependency to hadoop-compat. I am leaning more and more towards having two different poms for hadoop versions.

          Show
          Enis Soztutar added a comment - Turns out the way we do compat modules, we cannot do depend on the hadoop1|2 compat only on runtime unless we use reflection, or add hadoop dependency to hadoop-compat. I am leaning more and more towards having two different poms for hadoop versions.
          Hide
          Roshan Naik added a comment -

          i built a hadoop 2 version of hbase locally (0.97.0-SNAPSHOT) and pointed the flume pom to hbase as follows:

               <dependency>
                  <groupId>org.apache.hbase</groupId>
                  <artifactId>hbase-common</artifactId>
                  <version>${hbaseversion}</version>
                </dependency>
          
                <dependency>
                  <groupId>org.apache.hbase</groupId>
                  <artifactId>hbase-common</artifactId>
                  <version>${hbaseversion}</version>
                  <classifier>tests</classifier>
                  <scope>test</scope>
                </dependency>
          
                <dependency>
                  <groupId>org.apache.hbase</groupId>
                  <artifactId>hbase-client</artifactId>
                  <version>${hbaseversion}</version>
                </dependency>
          
                <dependency>
                  <groupId>org.apache.hbase</groupId>
                  <artifactId>hbase-server</artifactId>
                  <version>${hbaseversion}</version>
                </dependency>
          
                <dependency>
                  <groupId>org.apache.hbase</groupId>
                  <artifactId>hbase-server</artifactId>
                  <version>${hbaseversion}</version>
                  <classifier>tests</classifier>
                  <scope>test</scope>
                </dependency>
          
          

          Then built hbase as follows:

          mvn install -Dhadoop.profile=2.0 -Dhadoop-two.version=2.0.4-alpha -DskipTests
          

          I built flume as follows:

          mvn -Dhadoop-two.version=2.0.4-alpha -Dhbaseversion=0.97.0-SNAPSHOT -Dhadoop.profile=2 clean package -X -DskipTests

          Show
          Roshan Naik added a comment - i built a hadoop 2 version of hbase locally (0.97.0-SNAPSHOT) and pointed the flume pom to hbase as follows: <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-common</artifactId> <version>${hbaseversion}</version> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-common</artifactId> <version>${hbaseversion}</version> <classifier>tests</classifier> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-client</artifactId> <version>${hbaseversion}</version> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-server</artifactId> <version>${hbaseversion}</version> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-server</artifactId> <version>${hbaseversion}</version> <classifier>tests</classifier> <scope>test</scope> </dependency> Then built hbase as follows: mvn install -Dhadoop.profile=2.0 -Dhadoop-two.version=2.0.4-alpha -DskipTests I built flume as follows: mvn -Dhadoop-two.version=2.0.4-alpha -Dhbaseversion=0.97.0-SNAPSHOT -Dhadoop.profile=2 clean package -X -DskipTests
          Hide
          stack added a comment -
          Show
          stack added a comment - Roshan Naik Your workaround works for you? Do these snapshots help at all? We are trying to figure out how to publish our artifacts. Will this work for you? See: https://repository.apache.org/content/repositories/snapshots/org/apache/hbase/hbase-client/0.95.0-hadoop1-SNAPSHOT/ https://repository.apache.org/content/repositories/snapshots/org/apache/hbase/hbase-client/0.95.0-hadoop2-SNAPSHOT/ etc.
          Hide
          Roshan Naik added a comment -

          taking a look. will revert back soon.

          Show
          Roshan Naik added a comment - taking a look. will revert back soon.
          Hide
          Roshan Naik added a comment -

          The issue with transitive dependencies not getting pulled remains..

          [WARNING] The POM for org.apache.hbase:hbase-common:jar:tests:0.95.0-hadoop2-20130524.222913-2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
          [WARNING] The POM for org.apache.hbase:hbase-client:jar:0.95.0-hadoop2-20130524.222944-2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
          [WARNING] The POM for org.apache.hbase:hbase-server:jar:0.95.0-hadoop2-20130524.223047-2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
          [WARNING] The POM for org.apache.hbase:hbase-server:jar:tests:0.95.0-hadoop2-20130524.223047-2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details

          Show
          Roshan Naik added a comment - The issue with transitive dependencies not getting pulled remains.. [WARNING] The POM for org.apache.hbase:hbase-common:jar:tests:0.95.0-hadoop2-20130524.222913-2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details [WARNING] The POM for org.apache.hbase:hbase-client:jar:0.95.0-hadoop2-20130524.222944-2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details [WARNING] The POM for org.apache.hbase:hbase-server:jar:0.95.0-hadoop2-20130524.223047-2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details [WARNING] The POM for org.apache.hbase:hbase-server:jar:tests:0.95.0-hadoop2-20130524.223047-2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
          Hide
          Roshan Naik added a comment -

          mvn -X output

          [WARNING] The POM for org.apache.hbase:hbase-common:jar:tests:0.95.0-hadoop2-20130524.222913-2 is invalid, transitive dependencies (if any) will not be available: 2 problems were encountered while building the effective model for org.apache.hbase:hbase-common:0.95.0-hadoop2-SNAPSHOT
          [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for org.apache.hbase:$

          {compat.module}:jar with value '${compat.module}

          ' does not match a valid id pattern. @
          [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for org.apache.hbase:$

          {compat.module}:test-jar with value '${compat.module}

          ' does not match a valid id pattern. @

          Show
          Roshan Naik added a comment - mvn -X output [WARNING] The POM for org.apache.hbase:hbase-common:jar:tests:0.95.0-hadoop2-20130524.222913-2 is invalid, transitive dependencies (if any) will not be available: 2 problems were encountered while building the effective model for org.apache.hbase:hbase-common:0.95.0-hadoop2-SNAPSHOT [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for org.apache.hbase:$ {compat.module}:jar with value '${compat.module} ' does not match a valid id pattern. @ [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for org.apache.hbase:$ {compat.module}:test-jar with value '${compat.module} ' does not match a valid id pattern. @
          Enis Soztutar made changes -
          Field Original Value New Value
          Fix Version/s 0.98.0 [ 12323143 ]
          Fix Version/s 0.95.2 [ 12320040 ]
          Enis Soztutar made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Hide
          Enis Soztutar added a comment -

          Marking this critical for 0.95.2. We should fix this before 0.96 to unblock downstream projects.

          Show
          Enis Soztutar added a comment - Marking this critical for 0.95.2. We should fix this before 0.96 to unblock downstream projects.
          Enis Soztutar made changes -
          Link This issue relates to HBASE-6929 [ HBASE-6929 ]
          Enis Soztutar made changes -
          Link This issue relates to HBASE-8224 [ HBASE-8224 ]
          Enis Soztutar made changes -
          Priority Critical [ 2 ] Blocker [ 1 ]
          stack made changes -
          Assignee stack [ stack ]
          Hide
          stack added a comment -

          My little downstream project. Let me redo and check it in somewhere better named. Useful reproducing the downstreamer's experience.

          Show
          stack added a comment - My little downstream project. Let me redo and check it in somewhere better named. Useful reproducing the downstreamer's experience.
          stack made changes -
          Attachment client.tgz [ 12594876 ]
          Hide
          stack added a comment -

          Here is pointer to downstream project up on github https://github.com/saintstack/hbase-downstreamer

          Show
          stack added a comment - Here is pointer to downstream project up on github https://github.com/saintstack/hbase-downstreamer
          Hide
          stack added a comment -

          Marking as duplicate HBASE-8224. HBASE-8224 fixes the original issue where we had a variable in our poms that was not being interpolated.

          Show
          stack added a comment - Marking as duplicate HBASE-8224 . HBASE-8224 fixes the original issue where we had a variable in our poms that was not being interpolated.
          stack made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          stack made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          90d 18h 23m 1 stack 01/Aug/13 22:03
          Resolved Resolved Closed Closed
          52d 22h 19m 1 stack 23/Sep/13 20:22

            People

            • Assignee:
              stack
              Reporter:
              Roshan Naik
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development