Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1289

Update component versions in the pom.xml (including hadoop)

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: general, tests
    • Labels:
      None

      Description

      Because we are switching to protobuf 2.5 (BIGTOP-1241) as used by hadoop 2.1 and later, we need to upgrade top level pom file to specify compatible hadoop version, including all hadoop components.

      So in top level `bigtop/pom.xml` file, change:

      • /project/properties/hadoop.version = 2.3.0
      • update /project/properties/ {hbase,pig, ... }

        accordingly

      1. 0001-BIGTOP-1289.-Bump-component-versions-in-root-pom-fil.patch
        2 kB
        Martin Bukatovic
      2. 0001-BIGTOP-1289-update-component-versions-in-root-pom-fi.patch
        1 kB
        Martin Bukatovic
      3. BIGTOP-1289.patch
        4 kB
        Dasha Boudnik
      4. BIGTOP-1289.patch
        4 kB
        Dasha Boudnik
      5. BIGTOP-1289.patch
        4 kB
        Dasha Boudnik

        Issue Links

          Activity

          Hide
          mbukatov Martin Bukatovic added a comment - - edited

          Updating versions of: hadoop, hbase, pig, solr, spark (versions matches bigtop.mk file and related bigtop JIRAs slated for 0.8.0 release)

          Version was not changed for sqoop, zookeeper and giraph.

          Is there a plan to upgrade versions of other components as well?

          Show
          mbukatov Martin Bukatovic added a comment - - edited Updating versions of: hadoop, hbase, pig, solr, spark (versions matches bigtop.mk file and related bigtop JIRAs slated for 0.8.0 release) Version was not changed for sqoop, zookeeper and giraph. Is there a plan to upgrade versions of other components as well?
          Hide
          apurtell Andrew Purtell added a comment - - edited

          Is there a plan to upgrade versions of other components as well?

          We tried ZooKeeper 3.4.6 for HBase (in a 0.98 release) but this release has a change that breaks their CLI (ZOOKEEPER-1897), so recommend staying with 3.4.5 until 3.4.7 is released.

          Show
          apurtell Andrew Purtell added a comment - - edited Is there a plan to upgrade versions of other components as well? We tried ZooKeeper 3.4.6 for HBase (in a 0.98 release) but this release has a change that breaks their CLI ( ZOOKEEPER-1897 ), so recommend staying with 3.4.5 until 3.4.7 is released.
          Hide
          cos Konstantin Boudnik added a comment -

          Martin, Hive is getting bumped up to 0.13 (per BIGTOP-1110) - Giridharan Kesavan volunteered to help with the effort!

          Show
          cos Konstantin Boudnik added a comment - Martin, Hive is getting bumped up to 0.13 (per BIGTOP-1110 ) - Giridharan Kesavan volunteered to help with the effort!
          Hide
          cos Konstantin Boudnik added a comment -

          I've been thinking if we have to bring groovy version in line of what we are packaging for Bigtop stack?

          Show
          cos Konstantin Boudnik added a comment - I've been thinking if we have to bring groovy version in line of what we are packaging for Bigtop stack?
          Hide
          cos Konstantin Boudnik added a comment -

          I have just stepped on the issue that simply updating hbase version won't help the cause: we all need to reflect the fact that HBase artifact have split into a few after crossing the event horizon. E.g. something like this would need to happen to hbase smoke tests:

          --- a/bigtop-tests/test-artifacts/hbase/pom.xml
          +++ b/bigtop-tests/test-artifacts/hbase/pom.xml
          @@ -41,11 +41,15 @@
               </dependency>
               <dependency>
                 <groupId>org.apache.hbase</groupId>
          -      <artifactId>hbase</artifactId>
          +      <artifactId>hbase-client</artifactId>
               </dependency>
               <dependency>
                 <groupId>org.apache.hbase</groupId>
          -      <artifactId>hbase</artifactId>
          +      <artifactId>hbase-common</artifactId>
          +    </dependency>
          +    <dependency>
          +      <groupId>org.apache.hbase</groupId>
          +      <artifactId>hbase-common</artifactId>
                 <type>test-jar</type>
               </dependency>
          

          and to the top level pom file:

                 <dependency>
                   <groupId>org.apache.hbase</groupId>
          -        <artifactId>hbase</artifactId>
          -        <version>${hbase.version}</version>
          +        <artifactId>hbase-client</artifactId>
          +        <version>${hbase.version}-hadoop2</version>
                 </dependency>
                 <dependency>
                   <groupId>org.apache.hbase</groupId>
          -        <artifactId>hbase</artifactId>
          -        <version>${hbase.version}</version>
          +        <artifactId>hbase-common</artifactId>
          +        <version>${hbase.version}-hadoop2</version>
          +      </dependency>
          +      <dependency>
          +        <groupId>org.apache.hbase</groupId>
          +        <artifactId>hbase-common</artifactId>
          +        <version>${hbase.version}-hadoop2</version>
                   <type>test-jar</type>
          

          ... and perhaps more.
          I think we need a separate issue for that.

          Show
          cos Konstantin Boudnik added a comment - I have just stepped on the issue that simply updating hbase version won't help the cause: we all need to reflect the fact that HBase artifact have split into a few after crossing the event horizon. E.g. something like this would need to happen to hbase smoke tests: --- a/bigtop-tests/test-artifacts/hbase/pom.xml +++ b/bigtop-tests/test-artifacts/hbase/pom.xml @@ -41,11 +41,15 @@ </dependency> <dependency> <groupId>org.apache.hbase</groupId> - <artifactId>hbase</artifactId> + <artifactId>hbase-client</artifactId> </dependency> <dependency> <groupId>org.apache.hbase</groupId> - <artifactId>hbase</artifactId> + <artifactId>hbase-common</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-common</artifactId> <type>test-jar</type> </dependency> and to the top level pom file: <dependency> <groupId>org.apache.hbase</groupId> - <artifactId>hbase</artifactId> - <version>${hbase.version}</version> + <artifactId>hbase-client</artifactId> + <version>${hbase.version}-hadoop2</version> </dependency> <dependency> <groupId>org.apache.hbase</groupId> - <artifactId>hbase</artifactId> - <version>${hbase.version}</version> + <artifactId>hbase-common</artifactId> + <version>${hbase.version}-hadoop2</version> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-common</artifactId> + <version>${hbase.version}-hadoop2</version> <type>test-jar</type> ... and perhaps more. I think we need a separate issue for that.
          Hide
          cos Konstantin Boudnik added a comment -

          Martin, with BIGTOP-1302 in place - could you please take a look and make sure that the patch is still good? If so - I will commit it quickly. Thanks!

          Show
          cos Konstantin Boudnik added a comment - Martin, with BIGTOP-1302 in place - could you please take a look and make sure that the patch is still good? If so - I will commit it quickly. Thanks!
          Hide
          mbukatov Martin Bukatovic added a comment -

          Additional updates:

          No update needed for:

          • hive is not mentioned in the main pom file (bumped to 0.13 in BIGTOP-1110)

          Issues with:

          • HBase, maven wasn't able to get version 0.98.2 from repo.maven.apache.org
          • bumping groovy makes hadoopsmokes compilation fail, maven wasn't able to get
            groovy-eclipse-batch:jar:2.2.1-01 from repo.maven.apache.org

          see:

          [ERROR] Failed to execute goal on project hbase-smoke: Could not resolve dependencies for project org.apache.bigtop.itest:hbase-smoke:jar:0.8.0-SNAPSHOT: The following artifac
          ts could not be resolved: org.apache.hbase:hbase-common:jar:0.98.2, org.apache.hbase:hbase-common:jar:tests:0.98.2, org.apache.hbase:hbase-server:jar:0.98.2, org.apache.hbase:
          hbase-server:jar:tests:0.98.2: Could not find artifact org.apache.hbase:hbase-common:jar:0.98.2 in central (http://repo.maven.apache.org/maven2) -> [Help 1]
          
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project hadoop-smoke: Execution default-compile of goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile failed: Plugin org.apache.maven.plugins:maven-compiler-plugin:2.3.2 or one of its dependencies could not be resolved: Failure to find org.codehaus.groovy:groovy-eclipse-batch:jar:2.2.1-01 in http://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
          

          What is the correct way to resolve those issues? The specified versions follows
          the plan for 0.8.0 release.

          Show
          mbukatov Martin Bukatovic added a comment - Additional updates: groovy bumped to 2.2.1 (following bigtop.mk file, see BIGTOP-1097 ) spark bumped to 0.9.1 (see BIGTOP-1284 ) No update needed for: hive is not mentioned in the main pom file (bumped to 0.13 in BIGTOP-1110 ) Issues with: HBase, maven wasn't able to get version 0.98.2 from repo.maven.apache.org bumping groovy makes hadoopsmokes compilation fail, maven wasn't able to get groovy-eclipse-batch:jar:2.2.1-01 from repo.maven.apache.org see: [ERROR] Failed to execute goal on project hbase-smoke: Could not resolve dependencies for project org.apache.bigtop.itest:hbase-smoke:jar:0.8.0-SNAPSHOT: The following artifac ts could not be resolved: org.apache.hbase:hbase-common:jar:0.98.2, org.apache.hbase:hbase-common:jar:tests:0.98.2, org.apache.hbase:hbase-server:jar:0.98.2, org.apache.hbase: hbase-server:jar:tests:0.98.2: Could not find artifact org.apache.hbase:hbase-common:jar:0.98.2 in central (http://repo.maven.apache.org/maven2) -> [Help 1] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project hadoop-smoke: Execution default-compile of goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile failed: Plugin org.apache.maven.plugins:maven-compiler-plugin:2.3.2 or one of its dependencies could not be resolved: Failure to find org.codehaus.groovy:groovy-eclipse-batch:jar:2.2.1-01 in http://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1] What is the correct way to resolve those issues? The specified versions follows the plan for 0.8.0 release.
          Hide
          mbukatov Martin Bukatovic added a comment -

          Updated the patch, see description of issues in previous comment.

          Show
          mbukatov Martin Bukatovic added a comment - Updated the patch, see description of issues in previous comment.
          Hide
          apurtell Andrew Purtell added a comment -

          HBase since 0.96 munges Maven versions to indicate what version of Hadoop we built against.

          0.98.N-hadoop1 is built against Hadoop 1.2.1. 0.98.N-hadoop2 is build against Hadoop 2.2.0.

          Very much likely Bigtop doesn't want these artifacts from Maven central. We must build HBase to the local Maven cache with the exact version of Hadoop also desired, e.g. mvn -Dhadoop-two.version=2.3.0 ...

          Show
          apurtell Andrew Purtell added a comment - HBase since 0.96 munges Maven versions to indicate what version of Hadoop we built against. 0.98.N-hadoop1 is built against Hadoop 1.2.1. 0.98.N-hadoop2 is build against Hadoop 2.2.0. Very much likely Bigtop doesn't want these artifacts from Maven central. We must build HBase to the local Maven cache with the exact version of Hadoop also desired, e.g. mvn -Dhadoop-two.version=2.3.0 ...
          Hide
          cos Konstantin Boudnik added a comment - - edited

          +1 on Andrew said. During the stack we are building/installing Hadoop locally which produces version 2.3.0.
          And that's what has to be used by later ran HBase build.

          Show
          cos Konstantin Boudnik added a comment - - edited +1 on Andrew said. During the stack we are building/installing Hadoop locally which produces version 2.3.0. And that's what has to be used by later ran HBase build.
          Hide
          cos Konstantin Boudnik added a comment -

          As for groovy-eclipse-batch plugin use the latest version 2.1.8-01 as per Groovy official website here:
          http://groovy.codehaus.org/Groovy-Eclipse+compiler+plugin+for+Maven

          Show
          cos Konstantin Boudnik added a comment - As for groovy-eclipse-batch plugin use the latest version 2.1.8-01 as per Groovy official website here: http://groovy.codehaus.org/Groovy-Eclipse+compiler+plugin+for+Maven
          Hide
          dasha.boudnik Dasha Boudnik added a comment -

          groovy-eclipse-batch should be 2.8.0-01

          Show
          dasha.boudnik Dasha Boudnik added a comment - groovy-eclipse-batch should be 2.8.0-01
          Hide
          dasha.boudnik Dasha Boudnik added a comment - - edited

          Several changes in this patch:

          • groovy version changed in all relevant pom files; so is maven compiler version consequently, owing to compatibility issues
          • private method checkMetadata renamed to avoid compilation issues arising from the existence of a public class with the same name
          • other components bumped as in previous patch for this ticket
          Show
          dasha.boudnik Dasha Boudnik added a comment - - edited Several changes in this patch: groovy version changed in all relevant pom files; so is maven compiler version consequently, owing to compatibility issues private method checkMetadata renamed to avoid compilation issues arising from the existence of a public class with the same name other components bumped as in previous patch for this ticket
          Hide
          jayunit100 jay vyas added a comment -

          +1 ... thanks for updating all this stuff dasha !
          should i commit it? Or someone else is doing it?

          Show
          jayunit100 jay vyas added a comment - +1 ... thanks for updating all this stuff dasha ! should i commit it? Or someone else is doing it?
          Hide
          dasha.boudnik Dasha Boudnik added a comment -

          Nobody else is doing it, as far as I know. Might want to check with Konstantin Boudnik just to be sure...

          Show
          dasha.boudnik Dasha Boudnik added a comment - Nobody else is doing it, as far as I know. Might want to check with Konstantin Boudnik just to be sure...
          Hide
          cos Konstantin Boudnik added a comment -

          looks like the patch doesn't apply as the hadoop version is different. The rest is correct though

          Show
          cos Konstantin Boudnik added a comment - looks like the patch doesn't apply as the hadoop version is different. The rest is correct though
          Hide
          dasha.boudnik Dasha Boudnik added a comment - - edited

          Patch with fixed hadoop version.

          Show
          dasha.boudnik Dasha Boudnik added a comment - - edited Patch with fixed hadoop version.
          Hide
          dasha.boudnik Dasha Boudnik added a comment -

          Whitespaces fixed.

          Show
          dasha.boudnik Dasha Boudnik added a comment - Whitespaces fixed.
          Hide
          cos Konstantin Boudnik added a comment -

          +1 - patch applies cleanly and looks good

          Show
          cos Konstantin Boudnik added a comment - +1 - patch applies cleanly and looks good
          Hide
          cos Konstantin Boudnik added a comment -

          Committed to the master. Thanks Dasha!

          Show
          cos Konstantin Boudnik added a comment - Committed to the master. Thanks Dasha!

            People

            • Assignee:
              dasha.boudnik Dasha Boudnik
              Reporter:
              mbukatov Martin Bukatovic
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development