HBase
  1. HBase
  2. HBASE-4327

Compile HBase against hadoop 0.22

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.0
    • Fix Version/s: 0.92.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      To compile against a version of hadoop other than 0.20, set hadoop.profile. When -Dhadoop.profile=22 is specified, HBase will build against hadoop 0.22. When -Dhadoop.profile=23 is specified, HBase will build against hadoop 0.23. This is a change in behavior, previously used to be -Dhadoop23. When nothing is specified, then HBase will build against 0.20
      Show
      To compile against a version of hadoop other than 0.20, set hadoop.profile. When -Dhadoop.profile=22 is specified, HBase will build against hadoop 0.22. When -Dhadoop.profile=23 is specified, HBase will build against hadoop 0.23. This is a change in behavior, previously used to be -Dhadoop23. When nothing is specified, then HBase will build against 0.20

      Description

      Pom contains a profile for hadoop-0.20 and one for hadoop-0.23, but not one for hadoop-0.22.

      When overriding hadoop.version to 0.22, then the (compile-time) dependency on hadoop-annotations cannot be met.
      That exists on 0.23 and 0.24/trunk, but not on 0.22.

      1. HBASE-4327-Michael.patch
        5 kB
        Michael Weng
      2. HBASE-4327.patch
        5 kB
        Joep Rottinghuis
      3. HBASE-4327.patch
        6 kB
        Joep Rottinghuis
      4. HBASE-4327.patch
        6 kB
        Joep Rottinghuis

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2180 (See https://builds.apache.org/job/HBase-TRUNK/2180/)
          HBASE-4327 Compile HBase against hadoop 0.22

          stack :
          Files :

          • /hbase/trunk/CHANGES.txt
          • /hbase/trunk/pom.xml
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2180 (See https://builds.apache.org/job/HBase-TRUNK/2180/ ) HBASE-4327 Compile HBase against hadoop 0.22 stack : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/pom.xml
          Hide
          stack added a comment -

          Committed to TRUNK. I tried it w/o a profile and for 0.23.

          Todd, I updated build.apache.org so that our 0.23 build now uses this new flag instead.

          Thanks for the patch Joep. Nice on.

          Show
          stack added a comment - Committed to TRUNK. I tried it w/o a profile and for 0.23. Todd, I updated build.apache.org so that our 0.23 build now uses this new flag instead. Thanks for the patch Joep. Nice on.
          Hide
          stack added a comment -

          That should work. We have a build against 0.23 up on apache build box. I'd need to change the config. there on commit but that should be fine.

          Show
          stack added a comment - That should work. We have a build against 0.23 up on apache build box. I'd need to change the config. there on commit but that should be fine.
          Hide
          Joep Rottinghuis added a comment -

          Looks like there is a bug open in Maven to do exactly what I wanted to do:
          http://jira.codehaus.org/browse/MNG-3328

          What I can do is to change the property to hadoop.profile.

          • When nothing is specified, then HBase will build against 0.20
          • When -Dhadoop.profile=22 is specified, HBase will build against hadoop 0.22
          • When -Dhadoop.profile=23 is specified, HBase will build against hadoop 0.23. This is a change in behavior, previously used to be -Dhadoop23

          Attached a new patch with above behavior. Please let me know if this is acceptable.

          Show
          Joep Rottinghuis added a comment - Looks like there is a bug open in Maven to do exactly what I wanted to do: http://jira.codehaus.org/browse/MNG-3328 What I can do is to change the property to hadoop.profile. When nothing is specified, then HBase will build against 0.20 When -Dhadoop.profile=22 is specified, HBase will build against hadoop 0.22 When -Dhadoop.profile=23 is specified, HBase will build against hadoop 0.23. This is a change in behavior, previously used to be -Dhadoop23 Attached a new patch with above behavior. Please let me know if this is acceptable.
          Hide
          Joep Rottinghuis added a comment -

          Uhm, yes.
          Before there were two options:
          either specify nothing, or specify -Dhadoop23.
          The nothing/default option was run against 0.20 and was done when hadoop23 was not specified.
          Now we have three options.
          Let me see if we can make it something like:
          !(hadoop22 || hadoop 23)
          I tried with the specific options, but not without any.
          Will get back to you...

          Show
          Joep Rottinghuis added a comment - Uhm, yes. Before there were two options: either specify nothing, or specify -Dhadoop23. The nothing/default option was run against 0.20 and was done when hadoop23 was not specified. Now we have three options. Let me see if we can make it something like: !(hadoop22 || hadoop 23) I tried with the specific options, but not without any. Will get back to you...
          Hide
          stack added a comment -

          I applied the above patch and got this:

          [INFO] ------------------------------------------------------------------------
          [ERROR] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Compilation failure
          
          /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/Server.java:[22,29] package org.apache.hadoop.conf does not exist
          
          /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java:[29,29] package org.apache.hadoop.conf does not exist
          
          /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java:[30,27] package org.apache.hadoop.fs does not exist
          
          /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java:[31,27] package org.apache.hadoop.fs does not exist
          
          /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java:[37,27] package org.apache.hadoop.io does not exist
          
          /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java:[38,27] package org.apache.hadoop.io does not exist
          
          /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java:[66,33] cannot find symbol
          symbol: class Writable
          public class KeyValue implements Writable, HeapSize {
          ....
          

          Is it because of this change?

                 <id>hadoop-0.20</id>
                 <activation>
                   <property>
          -          <name>!hadoop23</name>
          +          <name>hadoop20</name>
                   </property>
                 </activation>
          
          Show
          stack added a comment - I applied the above patch and got this: [INFO] ------------------------------------------------------------------------ [ERROR] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Compilation failure /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/Server.java:[22,29] package org.apache.hadoop.conf does not exist /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java:[29,29] package org.apache.hadoop.conf does not exist /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java:[30,27] package org.apache.hadoop.fs does not exist /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java:[31,27] package org.apache.hadoop.fs does not exist /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java:[37,27] package org.apache.hadoop.io does not exist /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java:[38,27] package org.apache.hadoop.io does not exist /Users/stack/checkout/clean-trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java:[66,33] cannot find symbol symbol: class Writable public class KeyValue implements Writable, HeapSize { .... Is it because of this change? <id>hadoop-0.20</id> <activation> <property> - <name>!hadoop23</name> + <name>hadoop20</name> </property> </activation>
          Hide
          Joep Rottinghuis added a comment -

          Added additional comment and now that HDFS dependencies are fixed, the patch can now be applied.

          Show
          Joep Rottinghuis added a comment - Added additional comment and now that HDFS dependencies are fixed, the patch can now be applied.
          Hide
          Joep Rottinghuis added a comment -

          Turned out that HDFS was broken. I filed a bug for it: HDFS-2315.
          The newly published POM seems to have fixed the HDFS guava dependency issue
          :https://repository.apache.org/content/groups/snapshots/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-20110907.001816-381.pom

          I'll test the patch again and report back.

          Show
          Joep Rottinghuis added a comment - Turned out that HDFS was broken. I filed a bug for it: HDFS-2315 . The newly published POM seems to have fixed the HDFS guava dependency issue : https://repository.apache.org/content/groups/snapshots/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-20110907.001816-381.pom I'll test the patch again and report back.
          Hide
          Joep Rottinghuis added a comment -

          My preference is to have the hdfs pom fixed first, then I'll test the patch and confirm.

          Show
          Joep Rottinghuis added a comment - My preference is to have the hdfs pom fixed first, then I'll test the patch and confirm.
          Hide
          stack added a comment -

          Joep and Michael. You fellas are together? Should I apply Michael's last patch? It looks good to me. Do you want to make sure it works w/ whats published for hadoop 0.22 first? (Thanks for doing this work).

          Show
          stack added a comment - Joep and Michael. You fellas are together? Should I apply Michael's last patch? It looks good to me. Do you want to make sure it works w/ whats published for hadoop 0.22 first? (Thanks for doing this work).
          Hide
          Michael Weng added a comment -

          Here is my patch.

          Show
          Michael Weng added a comment - Here is my patch.
          Hide
          Joep Rottinghuis added a comment -

          Preliminary patch. Still need to resolve downstream hdfs dependency issue first. Asked Konstantin to initiate integration build on hadoop-0.22 in order to publish fixed jar+pom.
          Once in I need to check where the hadoop-test error comes from.

          Show
          Joep Rottinghuis added a comment - Preliminary patch. Still need to resolve downstream hdfs dependency issue first. Asked Konstantin to initiate integration build on hadoop-0.22 in order to publish fixed jar+pom. Once in I need to check where the hadoop-test error comes from.
          Hide
          Joep Rottinghuis added a comment -

          Problem still there on the Hadoop side:
          https://repository.apache.org/content/groups/snapshots/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-20110726.103826-380.pom

          The POM still points to org.apache.hadoop#guava and it should be com.google.guava.
          The source code in hdfs is correct, but the last published build is stale.

          Show
          Joep Rottinghuis added a comment - Problem still there on the Hadoop side: https://repository.apache.org/content/groups/snapshots/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-20110726.103826-380.pom The POM still points to org.apache.hadoop#guava and it should be com.google.guava. The source code in hdfs is correct, but the last published build is stale.
          Hide
          Joep Rottinghuis added a comment -

          Still seeing compilation error:
          [ERROR] Failed to execute goal on project hbase: Could not resolve dependencies for project org.apache.hbase:hbase:jar:0.91.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.hadoop:hadoop-test:jar:0.22.0-SNAPSHOT, org.apache.hadoop:guava:jar:r09: Could not find artifact org.apache.hadoop:hadoop-test:jar:0.22.0-SNAPSHOT in apache release (https://repository.apache.org/content/repositories/releases/) -> [Help 1]

          The guava one is related to HDFS-2189 (and HDFS-2214), which for some reason keeps rearing it's ugly head. Just wiped out ~/.m2/repository. Will download the offending POM manually to double-check.

          Show
          Joep Rottinghuis added a comment - Still seeing compilation error: [ERROR] Failed to execute goal on project hbase: Could not resolve dependencies for project org.apache.hbase:hbase:jar:0.91.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.hadoop:hadoop-test:jar:0.22.0-SNAPSHOT, org.apache.hadoop:guava:jar:r09: Could not find artifact org.apache.hadoop:hadoop-test:jar:0.22.0-SNAPSHOT in apache release ( https://repository.apache.org/content/repositories/releases/ ) -> [Help 1] The guava one is related to HDFS-2189 (and HDFS-2214 ), which for some reason keeps rearing it's ugly head. Just wiped out ~/.m2/repository. Will download the offending POM manually to double-check.
          Hide
          Joep Rottinghuis added a comment -

          Testing a patch locally, will post soon.
          Initially I thought that HBase depended on hadoop-annotations, but that is a problem with 0.23 (probably a missed dependency in those POMs).

          Show
          Joep Rottinghuis added a comment - Testing a patch locally, will post soon. Initially I thought that HBase depended on hadoop-annotations, but that is a problem with 0.23 (probably a missed dependency in those POMs).
          Hide
          stack added a comment -

          Good on you Michael. Post a patch?

          Show
          stack added a comment - Good on you Michael. Post a patch?
          Hide
          Michael Weng added a comment -

          I added a separate 0.22 profile without annotations. Build is fine. And no issue found so far while running hbase trunk on hadoop 0.22.

          Show
          Michael Weng added a comment - I added a separate 0.22 profile without annotations. Build is fine. And no issue found so far while running hbase trunk on hadoop 0.22.
          Hide
          stack added a comment -

          @Joep Any chance of seeing the compile error? How comes trunk compiles against hadoop 0.20.x w/o need of annotations? Is it because of the 0.20 profile? If we do a 0.22 profile, option 2., why we need hadoop-annotations?

          I'm wondering because I seem to remember compiling against 0.22 at one time w/o this dependency; maybe I was dreaming.

          Show
          stack added a comment - @Joep Any chance of seeing the compile error? How comes trunk compiles against hadoop 0.20.x w/o need of annotations? Is it because of the 0.20 profile? If we do a 0.22 profile, option 2., why we need hadoop-annotations? I'm wondering because I seem to remember compiling against 0.22 at one time w/o this dependency; maybe I was dreaming.
          Hide
          Joep Rottinghuis added a comment -

          I'm happy to provide a patch. I see three options:

          1) modify the POM to provide a separate property for the hadoop-annotation dependency version. That way one can depend on hadoop-0.22-SNAPSHOT (or whatever other 0.22 version) for hadoop-common, hadoop-hdfs, etc, and one could grab hadoop-annotations from 0.23 or 0.24.

          2) modify the POM to have an entirely separate 0.22 profile. Again the profile would have to point to hadoop-annotation from a 0.23 or 0.24/trunk snapshot.

          3) Not modify the pom, manually build hadoop-annotations from trunk with a modified / faked 0.22 version and deploy to local repo. This would solve my own internal problem, but is not likely to be a satisfying solution for uploading to the apache repo's (not sure if any committer is willing to upload such hadoop-annotation-0.22-SNAPSHOT.jar).

          Any suggestions?

          Show
          Joep Rottinghuis added a comment - I'm happy to provide a patch. I see three options: 1) modify the POM to provide a separate property for the hadoop-annotation dependency version. That way one can depend on hadoop-0.22-SNAPSHOT (or whatever other 0.22 version) for hadoop-common, hadoop-hdfs, etc, and one could grab hadoop-annotations from 0.23 or 0.24. 2) modify the POM to have an entirely separate 0.22 profile. Again the profile would have to point to hadoop-annotation from a 0.23 or 0.24/trunk snapshot. 3) Not modify the pom, manually build hadoop-annotations from trunk with a modified / faked 0.22 version and deploy to local repo. This would solve my own internal problem, but is not likely to be a satisfying solution for uploading to the apache repo's (not sure if any committer is willing to upload such hadoop-annotation-0.22-SNAPSHOT.jar). Any suggestions?

            People

            • Assignee:
              Joep Rottinghuis
              Reporter:
              Joep Rottinghuis
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development