Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21434

[hbase-connectors] Cleanup of kafka dependencies; clarify hadoop version

Details

    • Cleaned up kafka submodule dependencies. Added used dependencies to pom and removed the unused. Depends explicitly on hadoop2. No messing w/ hadoop3 versions.

    Description

      Running mvn dependency:analyze against hbase-connectors turned up a bunch of unused dependencies as wall as dependencies we rely on but don't mention in pom.

      I tried to do hadoop2/hadoop3 profiles in kafka module but figured makes no sense since it only references hadoop-common just to get at Configuration; made hadoop2 dependency explicit (Tried to do hadoop2/hadoop3 in top-level pom but won't be relevant always and its messy at the top-level around the hbase-hadoop2-compat module.... For another time).

      Attachments

        Issue Links

          Activity

            githubbot ASF GitHub Bot added a comment -

            saintstack opened a new pull request #6: HBASE-21434 [hbase-connectors] Cleanup of kafka dependencies; clarify…
            URL: https://github.com/apache/hbase-connectors/pull/6

            … hadoop version

            ----------------------------------------------------------------
            This is an automated message from the Apache Git Service.
            To respond to the message, please log on GitHub and use the
            URL above to go to the specific comment.

            For queries about this service, please contact Infrastructure at:
            users@infra.apache.org

            githubbot ASF GitHub Bot added a comment - saintstack opened a new pull request #6: HBASE-21434 [hbase-connectors] Cleanup of kafka dependencies; clarify… URL: https://github.com/apache/hbase-connectors/pull/6 … hadoop version ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org
            githubbot ASF GitHub Bot added a comment -

            saintstack closed pull request #6: HBASE-21434 [hbase-connectors] Cleanup of kafka dependencies; clarify…
            URL: https://github.com/apache/hbase-connectors/pull/6

            This is a PR merged from a forked repository.
            As GitHub hides the original diff on merge, it is displayed below for
            the sake of provenance:

            As this is a foreign pull request (from a fork), the diff is supplied
            below (as it won't show otherwise due to GitHub magic):

            diff --git a/kafka/hbase-kafka-proxy/pom.xml b/kafka/hbase-kafka-proxy/pom.xml
            index 66061f7..10ca760 100755
            — a/kafka/hbase-kafka-proxy/pom.xml
            +++ b/kafka/hbase-kafka-proxy/pom.xml
            @@ -34,9 +34,7 @@
            <properties>
            <collections.version>4.1</collections.version>
            <commons-lang3.version>3.6</commons-lang3.version>

            • <commons-io.version>2.5</commons-io.version>
              <kafka-clients.version>2.0.0</kafka-clients.version>
            • <commons-io.version>2.5</commons-io.version>
              </properties>
              <build>
              <plugins>
              @@ -73,24 +71,82 @@
              </build>
              <dependencies>
              <dependency>
            • <groupId>org.apache.avro</groupId>
            • <artifactId>avro</artifactId>
              + <groupId>org.slf4j</groupId>
              + <artifactId>slf4j-api</artifactId>
              </dependency>
              <dependency>
            • <groupId>org.apache.hbase.connectors.kafka</groupId>
            • <artifactId>hbase-kafka-model</artifactId>
              + <groupId>org.apache.yetus</groupId>
              + <artifactId>audience-annotations</artifactId>
              </dependency>
              <dependency>
            • <groupId>org.apache.hbase</groupId>
            • <artifactId>hbase-common</artifactId>
            • <type>test-jar</type>
            • <scope>test</scope>
              + <groupId>junit</groupId>
              + <artifactId>junit</artifactId>
              </dependency>
              <dependency>
            • <groupId>org.apache.hbase</groupId>
            • <artifactId>hbase-annotations</artifactId>
            • <type>test-jar</type>
            • <scope>test</scope>
              + <!--Relying on hadoop2. Not messing with hadoop2/hadoop3 stuff
              + in here. Its just a proxy. Just needs hadoop-common for
              + Configuration stuff and some basics -->
              + <groupId>org.apache.hadoop</groupId>
              + <artifactId>hadoop-common</artifactId>
              + <version>${hadoop-two.version}</version>
              + <exclusions>
              + <exclusion>
              + <groupId>commons-beanutils</groupId>
              + <artifactId>commons-beanutils</artifactId>
              + </exclusion>
              + <exclusion>
              + <groupId>javax.servlet.jsp</groupId>
              + <artifactId>jsp-api</artifactId>
              + </exclusion>
              + <exclusion>
              + <groupId>javax.servlet</groupId>
              + <artifactId>servlet-api</artifactId>
              + </exclusion>
              + <exclusion>
              + <groupId>stax</groupId>
              + <artifactId>stax-api</artifactId>
              + </exclusion>
              + <exclusion>
              + <groupId>io.netty</groupId>
              + <artifactId>netty</artifactId>
              + </exclusion>
              + <exclusion>
              + <groupId>com.google.code.findbugs</groupId>
              + <artifactId>jsr305</artifactId>
              + </exclusion>
              + <exclusion>
              + <groupId>junit</groupId>
              + <artifactId>junit</artifactId>
              + </exclusion>
              + <exclusion>
              + <groupId>org.codehaus.jackson</groupId>
              + <artifactId>jackson-core-asl</artifactId>
              + </exclusion>
              + <exclusion>
              + <groupId>org.codehaus.jackson</groupId>
              + <artifactId>jackson-mapper-asl</artifactId>
              + </exclusion>
              + <exclusion>
              + <groupId>org.apache.curator</groupId>
              + <artifactId>curator-client</artifactId>
              + </exclusion>
              + <exclusion>
              + <groupId>org.apache.curator</groupId>
              + <artifactId>curator-framework</artifactId>
              + </exclusion>
              + </exclusions>
              + </dependency>
              + <dependency>
              + <groupId>org.apache.hbase.thirdparty</groupId>
              + <artifactId>hbase-shaded-miscellaneous</artifactId>
              + </dependency>
              + <dependency>
              + <groupId>org.apache.avro</groupId>
              + <artifactId>avro</artifactId>
              + </dependency>
              + <dependency>
              + <groupId>org.apache.hbase.connectors.kafka</groupId>
              + <artifactId>hbase-kafka-model</artifactId>
              </dependency>
              <dependency>
              <groupId>org.apache.hbase</groupId>
              @@ -100,8 +156,10 @@
              <dependency>
              <groupId>org.apache.hbase</groupId>
              <artifactId>hbase-common</artifactId>
            • <type>test-jar</type>
            • <scope>test</scope>
              + </dependency>
              + <dependency>
              + <groupId>org.apache.hbase</groupId>
              + <artifactId>hbase-client</artifactId>
              </dependency>
              <dependency>
              <groupId>org.apache.kafka</groupId>
              @@ -114,18 +172,7 @@
              <artifactId>commons-lang3</artifactId>
              <version>${commons-lang3.version}</version>
              </dependency>
            • <dependency>
            • <groupId>org.apache.commons</groupId>
            • <artifactId>commons-collections4</artifactId>
            • <version>${collections.version}</version>
            • </dependency>
            • <dependency>
            • <groupId>commons-io</groupId>
            • <artifactId>commons-io</artifactId>
            • <version>${commons-io.version}</version>
            • </dependency>
              </dependencies>
              -
              <profiles>
              <!-- Needs to make the profile in apache parent pom -->
              <profile>
              diff --git a/pom.xml b/pom.xml
              index 0a6b39c..940681a 100755
                • a/pom.xml
                  +++ b/pom.xml
                  @@ -124,15 +124,45 @@
                  <maven.compiler.version>3.6.1</maven.compiler.version>
                  <exec.maven.version>1.6.0</exec.maven.version>
                  <audience-annotations.version>0.5.0</audience-annotations.version>
                  + <junit.version>4.12</junit.version>
                  + <hbase-thirdparty.version>2.1.0</hbase-thirdparty.version>
                  + <hadoop-two.version>2.7.7</hadoop-two.version>
                  + <hadoop-three.version>3.0.3</hadoop-three.version>
                  + <hadoop.version>${hadoop-two.version}</hadoop.version>
                  + <compat.module>hbase-hadoop2-compat</compat.module>
                  + <slf4j.version>1.7.25</slf4j.version>
                  </properties>
                  <dependencyManagement>
                  <dependencies>
                  + <dependency>
                  + <groupId>org.slf4j</groupId>
                  + <artifactId>slf4j-log4j12</artifactId>
                  + <version>${slf4j.version}</version>
                  + </dependency>
                  + <dependency>
                  + <groupId>org.slf4j</groupId>
                  + <artifactId>slf4j-api</artifactId>
                  + <version>${slf4j.version}</version>
                  + </dependency>
                  + <dependency>
                  + <groupId>org.apache.yetus</groupId>
                  + <artifactId>audience-annotations</artifactId>
                  + <version>${audience-annotations.version}</version>
                  + </dependency>
                  + <dependency>
                  + <groupId>junit</groupId>
                  + <artifactId>junit</artifactId>
                  + <version>${junit.version}</version>
                  + </dependency>
                  + <dependency>
                  + <groupId>org.apache.hbase.thirdparty</groupId>
                  + <artifactId>hbase-shaded-miscellaneous</artifactId>
                  + <version>${hbase-thirdparty.version}</version>
                  + </dependency>
                  <dependency>
                  <groupId>org.apache.hbase</groupId>
            • <artifactId>hbase-annotations</artifactId>
              + <artifactId>hbase-common</artifactId>
              <version>${hbase.version}</version>
            • <type>test-jar</type>
            • <scope>test</scope>
              </dependency>
              <dependency>
              <groupId>org.apache.hbase</groupId>
              @@ -141,12 +171,22 @@
              <type>test-jar</type>
              <scope>test</scope>
              </dependency>
              + <dependency>
              + <groupId>org.apache.hbase</groupId>
              + <artifactId>hbase-hadoop2-compat</artifactId>
              + <version>${hbase.version}</version>
              + </dependency>
              <dependency>
              <groupId>org.apache.hbase</groupId>
              <artifactId>hbase-server</artifactId>
              <version>${hbase.version}</version>
              <scope>provided</scope>
              </dependency>
              + <dependency>
              + <groupId>org.apache.hbase</groupId>
              + <artifactId>hbase-client</artifactId>
              + <version>${hbase.version}</version>
              + </dependency>
              </dependencies>
              </dependencyManagement>
              <build>

            ----------------------------------------------------------------
            This is an automated message from the Apache Git Service.
            To respond to the message, please log on GitHub and use the
            URL above to go to the specific comment.

            For queries about this service, please contact Infrastructure at:
            users@infra.apache.org

            githubbot ASF GitHub Bot added a comment - saintstack closed pull request #6: HBASE-21434 [hbase-connectors] Cleanup of kafka dependencies; clarify… URL: https://github.com/apache/hbase-connectors/pull/6 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/kafka/hbase-kafka-proxy/pom.xml b/kafka/hbase-kafka-proxy/pom.xml index 66061f7..10ca760 100755 — a/kafka/hbase-kafka-proxy/pom.xml +++ b/kafka/hbase-kafka-proxy/pom.xml @@ -34,9 +34,7 @@ <properties> <collections.version>4.1</collections.version> <commons-lang3.version>3.6</commons-lang3.version> <commons-io.version>2.5</commons-io.version> <kafka-clients.version>2.0.0</kafka-clients.version> <commons-io.version>2.5</commons-io.version> </properties> <build> <plugins> @@ -73,24 +71,82 @@ </build> <dependencies> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro</artifactId> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> </dependency> <dependency> <groupId>org.apache.hbase.connectors.kafka</groupId> <artifactId>hbase-kafka-model</artifactId> + <groupId>org.apache.yetus</groupId> + <artifactId>audience-annotations</artifactId> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-common</artifactId> <type>test-jar</type> <scope>test</scope> + <groupId>junit</groupId> + <artifactId>junit</artifactId> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-annotations</artifactId> <type>test-jar</type> <scope>test</scope> + <!--Relying on hadoop2. Not messing with hadoop2/hadoop3 stuff + in here. Its just a proxy. Just needs hadoop-common for + Configuration stuff and some basics --> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + <version>${hadoop-two.version}</version> + <exclusions> + <exclusion> + <groupId>commons-beanutils</groupId> + <artifactId>commons-beanutils</artifactId> + </exclusion> + <exclusion> + <groupId>javax.servlet.jsp</groupId> + <artifactId>jsp-api</artifactId> + </exclusion> + <exclusion> + <groupId>javax.servlet</groupId> + <artifactId>servlet-api</artifactId> + </exclusion> + <exclusion> + <groupId>stax</groupId> + <artifactId>stax-api</artifactId> + </exclusion> + <exclusion> + <groupId>io.netty</groupId> + <artifactId>netty</artifactId> + </exclusion> + <exclusion> + <groupId>com.google.code.findbugs</groupId> + <artifactId>jsr305</artifactId> + </exclusion> + <exclusion> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + </exclusion> + <exclusion> + <groupId>org.codehaus.jackson</groupId> + <artifactId>jackson-core-asl</artifactId> + </exclusion> + <exclusion> + <groupId>org.codehaus.jackson</groupId> + <artifactId>jackson-mapper-asl</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.curator</groupId> + <artifactId>curator-client</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.curator</groupId> + <artifactId>curator-framework</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.hbase.thirdparty</groupId> + <artifactId>hbase-shaded-miscellaneous</artifactId> + </dependency> + <dependency> + <groupId>org.apache.avro</groupId> + <artifactId>avro</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase.connectors.kafka</groupId> + <artifactId>hbase-kafka-model</artifactId> </dependency> <dependency> <groupId>org.apache.hbase</groupId> @@ -100,8 +156,10 @@ <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-common</artifactId> <type>test-jar</type> <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-client</artifactId> </dependency> <dependency> <groupId>org.apache.kafka</groupId> @@ -114,18 +172,7 @@ <artifactId>commons-lang3</artifactId> <version>${commons-lang3.version}</version> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-collections4</artifactId> <version>${collections.version}</version> </dependency> <dependency> <groupId>commons-io</groupId> <artifactId>commons-io</artifactId> <version>${commons-io.version}</version> </dependency> </dependencies> - <profiles> <!-- Needs to make the profile in apache parent pom --> <profile> diff --git a/pom.xml b/pom.xml index 0a6b39c..940681a 100755 a/pom.xml +++ b/pom.xml @@ -124,15 +124,45 @@ <maven.compiler.version>3.6.1</maven.compiler.version> <exec.maven.version>1.6.0</exec.maven.version> <audience-annotations.version>0.5.0</audience-annotations.version> + <junit.version>4.12</junit.version> + <hbase-thirdparty.version>2.1.0</hbase-thirdparty.version> + <hadoop-two.version>2.7.7</hadoop-two.version> + <hadoop-three.version>3.0.3</hadoop-three.version> + <hadoop.version>${hadoop-two.version}</hadoop.version> + <compat.module>hbase-hadoop2-compat</compat.module> + <slf4j.version>1.7.25</slf4j.version> </properties> <dependencyManagement> <dependencies> + <dependency> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + <version>${slf4j.version}</version> + </dependency> + <dependency> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + <version>${slf4j.version}</version> + </dependency> + <dependency> + <groupId>org.apache.yetus</groupId> + <artifactId>audience-annotations</artifactId> + <version>${audience-annotations.version}</version> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <version>${junit.version}</version> + </dependency> + <dependency> + <groupId>org.apache.hbase.thirdparty</groupId> + <artifactId>hbase-shaded-miscellaneous</artifactId> + <version>${hbase-thirdparty.version}</version> + </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-annotations</artifactId> + <artifactId>hbase-common</artifactId> <version>${hbase.version}</version> <type>test-jar</type> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.hbase</groupId> @@ -141,12 +171,22 @@ <type>test-jar</type> <scope>test</scope> </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop2-compat</artifactId> + <version>${hbase.version}</version> + </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-server</artifactId> <version>${hbase.version}</version> <scope>provided</scope> </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-client</artifactId> + <version>${hbase.version}</version> + </dependency> </dependencies> </dependencyManagement> <build> ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org
            stack Michael Stack added a comment -

            Merged.

            stack Michael Stack added a comment - Merged.

            People

              stack Michael Stack
              stack Michael Stack
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: