HBase
  1. HBase
  2. HBASE-6929

Publish Hbase 0.94 artifacts build against hadoop-2.0

    Details

    • Type: Task Task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.94.2
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None

      Description

      Downstream projects (flume, hive, pig, etc) depends on hbase, but since the hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should also push hbase jars build with hadoop2.0 profile into maven, possibly with version string like 0.94.2-hadoop2.0.

      1. 6929.txt
        0.9 kB
        stack
      2. hbase-6929_v2.patch
        1.0 kB
        Enis Soztutar

        Issue Links

          Activity

          Hide
          Ted Yu added a comment -

          There're test failures such as TestImportExport against hadoop 2.0
          Would that affect the use of HBase on top of hadoop 2.0 ?

          Show
          Ted Yu added a comment - There're test failures such as TestImportExport against hadoop 2.0 Would that affect the use of HBase on top of hadoop 2.0 ?
          Hide
          Roshan Naik added a comment -

          Need Hbase binaries built against Hadoop 2.x for this.

          Show
          Roshan Naik added a comment - Need Hbase binaries built against Hadoop 2.x for this.
          Hide
          stack added a comment -

          Enis Soztutar Officially 0.96 supports hadoop2, not 0.94. How would we publish the different jars to mvn repo? Those built for h1 and those for h2? The artifact is hbase-X.X.X.jar? You suggest adding a hadoop1 and hadoop2 to the name? Even then, we're breaking the mvn model of an artifact per module. Maybe hbase-X.X.X.jar becomes a meta jar under which you can slot in hbase-h1 or hbase-h2... but that sounds like ugly work w/ an explosion in hbase maven modules?

          Show
          stack added a comment - Enis Soztutar Officially 0.96 supports hadoop2, not 0.94. How would we publish the different jars to mvn repo? Those built for h1 and those for h2? The artifact is hbase-X.X.X.jar? You suggest adding a hadoop1 and hadoop2 to the name? Even then, we're breaking the mvn model of an artifact per module. Maybe hbase-X.X.X.jar becomes a meta jar under which you can slot in hbase-h1 or hbase-h2... but that sounds like ugly work w/ an explosion in hbase maven modules?
          Hide
          Enis Soztutar added a comment -

          Officially 0.96 supports hadoop2, not 0.94

          I saw a bunch of fixes for hadoop-2.0, but did not check whether 0.94 it works. Let me go over the fixes that went in to see whether 0.94 can also officially support 2.0. Although not officially production-ready, 0.94 builds with hadoop-2.0, and there are some interest in downstream projects to build and run tests with h2, publishing the artifacts might be justified.

          How would we publish the different jars to mvn repo?

          We do not need to break the current setup for h1. Since the binaries are different we can publish h2 artifacts with either naming it smt like hbase-hadoop2-X.X.X.jar, or using maven classifiers.
          Agreed that this is ugly, but if we do not publish the artifacts, downstream projects cannot easily consume hbase jars build with hadoop-2.0 in their hadoop-2.0 profiles.

          Show
          Enis Soztutar added a comment - Officially 0.96 supports hadoop2, not 0.94 I saw a bunch of fixes for hadoop-2.0, but did not check whether 0.94 it works. Let me go over the fixes that went in to see whether 0.94 can also officially support 2.0. Although not officially production-ready, 0.94 builds with hadoop-2.0, and there are some interest in downstream projects to build and run tests with h2, publishing the artifacts might be justified. How would we publish the different jars to mvn repo? We do not need to break the current setup for h1. Since the binaries are different we can publish h2 artifacts with either naming it smt like hbase-hadoop2-X.X.X.jar, or using maven classifiers. Agreed that this is ugly, but if we do not publish the artifacts, downstream projects cannot easily consume hbase jars build with hadoop-2.0 in their hadoop-2.0 profiles.
          Hide
          Enis Soztutar added a comment -

          Lars Hofhansl, as the RM, do you have any feedback on this?

          Show
          Enis Soztutar added a comment - Lars Hofhansl , as the RM, do you have any feedback on this?
          Hide
          Lars Hofhansl added a comment -

          I think this is a more general discussion than just 0.94.
          The question is: Do we force folks to take the HBase "singularity" in order to use Hadoop-2? Since Hadoop-2 breaks a bunch of stuff, the answer might well be "Yes".

          That said, we have been using 0.94 with Hadoop-2 here and there (no actual testing or a lot of load) on our test cluster and everything seemed fine.

          Personally I'd like to see 0.94 tested on top of Hadoop-2.

          Show
          Lars Hofhansl added a comment - I think this is a more general discussion than just 0.94. The question is: Do we force folks to take the HBase "singularity" in order to use Hadoop-2? Since Hadoop-2 breaks a bunch of stuff, the answer might well be "Yes". That said, we have been using 0.94 with Hadoop-2 here and there (no actual testing or a lot of load) on our test cluster and everything seemed fine. Personally I'd like to see 0.94 tested on top of Hadoop-2.
          Hide
          Enis Soztutar added a comment -

          I think this is a more general discussion than just 0.94.

          Agreed. Since H2 is labeled alpha, we can just release the binary labeled with suffix hadoop-2-alpha, and document this as not production ready. I don't see any reason why we might wait instead of publishing 0.94 with H2 as long as it is clear that it is not tested enough.

          Personally I'd like to see 0.94 tested on top of Hadoop-2

          Our folks are doing some testing on that, and the tests for hive/pig/flume on hadoop-2 with hbase-0.94 are blocked by this. Getting this out will definitely help with testing.

          Show
          Enis Soztutar added a comment - I think this is a more general discussion than just 0.94. Agreed. Since H2 is labeled alpha, we can just release the binary labeled with suffix hadoop-2-alpha, and document this as not production ready. I don't see any reason why we might wait instead of publishing 0.94 with H2 as long as it is clear that it is not tested enough. Personally I'd like to see 0.94 tested on top of Hadoop-2 Our folks are doing some testing on that, and the tests for hive/pig/flume on hadoop-2 with hbase-0.94 are blocked by this. Getting this out will definitely help with testing.
          Hide
          stack added a comment -

          @Enis Seems fine by me. We'd have to backport h2 fixes. It'd be 0.94.3? Or would a snapshot do (qualified with a maven classifier?)

          Show
          stack added a comment - @Enis Seems fine by me. We'd have to backport h2 fixes. It'd be 0.94.3? Or would a snapshot do (qualified with a maven classifier?)
          Hide
          Lars Hofhansl added a comment -

          Were there many Hadoop-2 related fixes in 0.96? I would prefer not to de-stabilize the 0.94 code base at this point.

          If these are simple pom fixes and some reflection here and there I am all for it. If we have backport the Hadoop-2 module... Forget it

          Show
          Lars Hofhansl added a comment - Were there many Hadoop-2 related fixes in 0.96? I would prefer not to de-stabilize the 0.94 code base at this point. If these are simple pom fixes and some reflection here and there I am all for it. If we have backport the Hadoop-2 module... Forget it
          Hide
          stack added a comment -

          @Lars Yes. Wondering if a snapshot would do? Could do it w/o even changing the pom?

          Show
          stack added a comment - @Lars Yes. Wondering if a snapshot would do? Could do it w/o even changing the pom?
          Hide
          Enis Soztutar added a comment -

          If these are simple pom fixes and some reflection here and there I am all for it. If we have backport the Hadoop-2 module. Forget it

          Agreed. We should identify the changes, and only backport if they are not intrusive.

          Wondering if a snapshot would do?

          I think that would do for now.

          Show
          Enis Soztutar added a comment - If these are simple pom fixes and some reflection here and there I am all for it. If we have backport the Hadoop-2 module. Forget it Agreed. We should identify the changes, and only backport if they are not intrusive. Wondering if a snapshot would do? I think that would do for now.
          Hide
          Lars Hofhansl added a comment -

          What do you guys mean with "a snapshot"?

          Show
          Lars Hofhansl added a comment - What do you guys mean with "a snapshot"?
          Hide
          stack added a comment -

          Lars Hofhansl If you do mvn package, it will install the build into your local mvn repository (see under ~/.m2/repository). Notice how the suffix is -SNAPSHOT. The suggestion is to push one of these snapshots up to apache (Apache maintains an area where you can publish mvn snapshots; see https://repository.apache.org/content/groups/snapshots/org/apache/hbase/hbase/).

          Show
          stack added a comment - Lars Hofhansl If you do mvn package, it will install the build into your local mvn repository (see under ~/.m2/repository). Notice how the suffix is -SNAPSHOT. The suggestion is to push one of these snapshots up to apache (Apache maintains an area where you can publish mvn snapshots; see https://repository.apache.org/content/groups/snapshots/org/apache/hbase/hbase/ ).
          Hide
          stack added a comment -

          You know how to fix this Enis Soztutar?

          Here is what I ran: 
          
          stack:0.94 stack$ mvn -DskipTests -Dhadoop.profile=2.0 -Dclassifier=hadoop2 install
          
          ...and I got this:
          
          ...
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 1:12.527s
          [INFO] Finished at: Tue Oct 09 12:11:41 PDT 2012
          [INFO] Final Memory: 28M/81M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.1:build-classpath (create-mrapp-generated-classpath) on project hbase: not found in any repository: aopalliance:aopalliance:java-source:hadoop2:1.0: Could not find artifact aopalliance:aopalliance:jar:hadoop2:1.0 in apache release (https://repository.apache.org/content/repositories/releases/)
          [ERROR] 
          [ERROR] Try downloading the file manually from the project website.
          [ERROR] 
          [ERROR] Then, install it using the command:
          [ERROR] mvn install:install-file -DgroupId=aopalliance -DartifactId=aopalliance -Dversion=1.0 -Dclassifier=hadoop2 -Dpackaging=java-source -Dfile=/path/to/file
          [ERROR] 
          [ERROR] Alternatively, if you host your own repository you can deploy the file there:
          [ERROR] mvn deploy:deploy-file -DgroupId=aopalliance -DartifactId=aopalliance -Dversion=1.0 -Dclassifier=hadoop2 -Dpackaging=java-source -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
          [ERROR] 
          [ERROR] 
          [ERROR] aopalliance:aopalliance:java-source:1.0
          [ERROR] 
          [ERROR] from the specified remote repositories:
          [ERROR] apache release (https://repository.apache.org/content/repositories/releases/, releases=true, snapshots=true),
          [ERROR] apache non-releases (http://people.apache.org/~stack/m2/repository, releases=true, snapshots=false),
          [ERROR] java.net (http://download.java.net/maven/2/, releases=true, snapshots=false),
          [ERROR] codehaus (http://repository.codehaus.org/, releases=true, snapshots=false),
          [ERROR] repository.jboss.org (http://repository.jboss.org/nexus/content/groups/public-jboss/, releases=true, snapshots=false),
          [ERROR] ghelmling.testing (http://people.apache.org/~garyh/mvn/, releases=true, snapshots=true),
          [ERROR] apache.snapshots (http://repository.apache.org/snapshots, releases=false, snapshots=true),
          [ERROR] central (http://repo1.maven.org/maven2, releases=true, snapshots=false)
          [ERROR] -> [Help 1]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
          
          Show
          stack added a comment - You know how to fix this Enis Soztutar ? Here is what I ran: stack:0.94 stack$ mvn -DskipTests -Dhadoop.profile=2.0 -Dclassifier=hadoop2 install ...and I got this : ... [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1:12.527s [INFO] Finished at: Tue Oct 09 12:11:41 PDT 2012 [INFO] Final Memory: 28M/81M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.1:build-classpath (create-mrapp-generated-classpath) on project hbase: not found in any repository: aopalliance:aopalliance:java-source:hadoop2:1.0: Could not find artifact aopalliance:aopalliance:jar:hadoop2:1.0 in apache release (https: //repository.apache.org/content/repositories/releases/) [ERROR] [ERROR] Try downloading the file manually from the project website. [ERROR] [ERROR] Then, install it using the command: [ERROR] mvn install:install-file -DgroupId=aopalliance -DartifactId=aopalliance -Dversion=1.0 -Dclassifier=hadoop2 -Dpackaging=java-source -Dfile=/path/to/file [ERROR] [ERROR] Alternatively, if you host your own repository you can deploy the file there: [ERROR] mvn deploy:deploy-file -DgroupId=aopalliance -DartifactId=aopalliance -Dversion=1.0 -Dclassifier=hadoop2 -Dpackaging=java-source -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] [ERROR] [ERROR] [ERROR] aopalliance:aopalliance:java-source:1.0 [ERROR] [ERROR] from the specified remote repositories: [ERROR] apache release (https: //repository.apache.org/content/repositories/releases/, releases= true , snapshots= true ), [ERROR] apache non-releases (http: //people.apache.org/~stack/m2/repository, releases= true , snapshots= false ), [ERROR] java.net (http: //download.java.net/maven/2/, releases= true , snapshots= false ), [ERROR] codehaus (http: //repository.codehaus.org/, releases= true , snapshots= false ), [ERROR] repository.jboss.org (http: //repository.jboss.org/nexus/content/groups/ public -jboss/, releases= true , snapshots= false ), [ERROR] ghelmling.testing (http: //people.apache.org/~garyh/mvn/, releases= true , snapshots= true ), [ERROR] apache.snapshots (http: //repository.apache.org/snapshots, releases= false , snapshots= true ), [ERROR] central (http: //repo1.maven.org/maven2, releases= true , snapshots= false ) [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch . [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http: //cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
          Hide
          Lars Hofhansl added a comment -

          We were discussing this internally, and there'd be interest for us to have 0.94 working with Hadoop-2. Is there a top-level jira that I can check out to see the magnitude of the changes needed?

          Show
          Lars Hofhansl added a comment - We were discussing this internally, and there'd be interest for us to have 0.94 working with Hadoop-2. Is there a top-level jira that I can check out to see the magnitude of the changes needed?
          Hide
          Enis Soztutar added a comment -

          @Stack, I think -Dclassifier causes the problem.

          mvn clean install -Dhadoop.profile=2.0
          

          Works for me on 0.94. There should be a pom.xml change for deploying the local jar with a classifier. I'll look into that.
          @Lars, I did not see an umbrella issue, but I think we have to compile a list, and see what comes out of it. If I have some time by the end of week, I'll try to do that.

          Show
          Enis Soztutar added a comment - @Stack, I think -Dclassifier causes the problem. mvn clean install -Dhadoop.profile=2.0 Works for me on 0.94. There should be a pom.xml change for deploying the local jar with a classifier. I'll look into that. @Lars, I did not see an umbrella issue, but I think we have to compile a list, and see what comes out of it. If I have some time by the end of week, I'll try to do that.
          Hide
          Jarek Jarcec Cecho added a comment -

          I would like to support this ticket as Sqoop project also have issues with missing hbase jars compiled against hadoop 2. I would greatly appreciate if this will happen

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - I would like to support this ticket as Sqoop project also have issues with missing hbase jars compiled against hadoop 2. I would greatly appreciate if this will happen Jarcec
          Hide
          stack added a comment - - edited

          Maven makes no sense (I know no one will read and register the statement I just made because all have same opinion).

          -Dclassifier just doesn't work on mvn command line. Maybe hadoop pom needs an amendment so all is just optional or some such voodoo but if you specify -Dclassifier and try to deploy, it doesn't matter what combination seemingly (at least going by my hours spent trying to figure mvn vagaries), it just tries to include dependencies w/ the classifier attached...e.g. randomjar-yourclassifier.jar, and so on. This fails.

          The attached patch modifies the hadooop 2.0 profile so the hbase jar built has an hadoop2 suffix (also ups the version of our apache pom from 8 to 11). It does not decorate the test, source or javadoc jars w/ the classifier suffix.

          I went ahead and uploaded 0.94.3-SNAPSHOTs to maven repository, w/ the hadoop2 classification: https://repository.apache.org/content/repositories/snapshots/org/apache/hbase/hbase/0.94.3-SNAPSHOT/. Is this all that is needed? If so I'll commit this patch as is (I think you'd depend on the hbase-0.94.3-SNAPSHOT but would reference explicitly hbase-0.94.3-SNAPSHOT-hadoop2.jar if an hadoop2 dependency).

          Show
          stack added a comment - - edited Maven makes no sense (I know no one will read and register the statement I just made because all have same opinion). -Dclassifier just doesn't work on mvn command line. Maybe hadoop pom needs an amendment so all is just optional or some such voodoo but if you specify -Dclassifier and try to deploy, it doesn't matter what combination seemingly (at least going by my hours spent trying to figure mvn vagaries), it just tries to include dependencies w/ the classifier attached...e.g. randomjar-yourclassifier.jar, and so on. This fails. The attached patch modifies the hadooop 2.0 profile so the hbase jar built has an hadoop2 suffix (also ups the version of our apache pom from 8 to 11). It does not decorate the test, source or javadoc jars w/ the classifier suffix. I went ahead and uploaded 0.94.3-SNAPSHOTs to maven repository, w/ the hadoop2 classification: https://repository.apache.org/content/repositories/snapshots/org/apache/hbase/hbase/0.94.3-SNAPSHOT/ . Is this all that is needed? If so I'll commit this patch as is (I think you'd depend on the hbase-0.94.3-SNAPSHOT but would reference explicitly hbase-0.94.3-SNAPSHOT-hadoop2.jar if an hadoop2 dependency).
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Sir,
          thank you very much for your time on this issue. Would you mind also providing test artifact compiled against hadoop 2?

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi Sir, thank you very much for your time on this issue. Would you mind also providing test artifact compiled against hadoop 2? Jarcec
          Hide
          stack added a comment -

          I can add hadoop2 to the name of the target artifact but not to the test jar. When maven-jar-plugin makes the jar, it uses a hard-coded 'tests' classifier. I cannot override it w/o changing plugin code (adding a classifier to maven-jar-plugin is just ignored).

          I tried various other things. I tried getting build-helper-maven-plugin to attach the test jar at package time but only seemed to end up doubling the test jars installed (a variant of this suggestion http://stackoverflow.com/questions/8499266/maven-deploy-source-classifiers). They still didn't have the right name on install. I think the maven-jar-plugin attaches the test jar before I can intercede later w/ the build-helper-maven-plugin.

          I tried setting the finalName to hadoop2 when we are using the hadoop2 profile. This gets pretty far. See below:

          [INFO] --- maven-install-plugin:2.3.1:install (default-install) @ hbase ---
          [INFO] Installing /Users/stack/checkouts/0.94/target/hbase-0.94.3-SNAPSHOT-hadoop2.jar to /Users/stack/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT/hbase-0.94.3-SNAPSHOT.jar
          [INFO] Installing /Users/stack/checkouts/0.94/pom.xml to /Users/stack/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT/hbase-0.94.3-SNAPSHOT.pom
          [INFO] Installing /Users/stack/checkouts/0.94/target/hbase-0.94.3-SNAPSHOT-hadoop2-tests.jar to /Users/stack/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT/hbase-0.94.3-SNAPSHOT-tests.jar
          [INFO] Installing /Users/stack/checkouts/0.94/target/hbase-0.94.3-SNAPSHOT-hadoop2-sources.jar to /Users/stack/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT/hbase-0.94.3-SNAPSHOT-sources.jar
          

          I cannot get install though to write the local repo w/ the names I told it to use. Looking at the install plugin, it uses maven itself copying artifacts (The above logging is the output of a repository 'event' as artifacts are copied in). I cannot put config on the install plugin to have it use my final name rather than the one it composes from base pom attributes.

          Anyone else have a suggestion?

          Is the test jar really needed? The hadoop1 built one doesn't work w/ hadoop2?

          Show
          stack added a comment - I can add hadoop2 to the name of the target artifact but not to the test jar. When maven-jar-plugin makes the jar, it uses a hard-coded 'tests' classifier. I cannot override it w/o changing plugin code (adding a classifier to maven-jar-plugin is just ignored). I tried various other things. I tried getting build-helper-maven-plugin to attach the test jar at package time but only seemed to end up doubling the test jars installed (a variant of this suggestion http://stackoverflow.com/questions/8499266/maven-deploy-source-classifiers ). They still didn't have the right name on install. I think the maven-jar-plugin attaches the test jar before I can intercede later w/ the build-helper-maven-plugin. I tried setting the finalName to hadoop2 when we are using the hadoop2 profile. This gets pretty far. See below: [INFO] --- maven-install-plugin:2.3.1:install ( default -install) @ hbase --- [INFO] Installing /Users/stack/checkouts/0.94/target/hbase-0.94.3-SNAPSHOT-hadoop2.jar to /Users/stack/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT/hbase-0.94.3-SNAPSHOT.jar [INFO] Installing /Users/stack/checkouts/0.94/pom.xml to /Users/stack/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT/hbase-0.94.3-SNAPSHOT.pom [INFO] Installing /Users/stack/checkouts/0.94/target/hbase-0.94.3-SNAPSHOT-hadoop2-tests.jar to /Users/stack/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT/hbase-0.94.3-SNAPSHOT-tests.jar [INFO] Installing /Users/stack/checkouts/0.94/target/hbase-0.94.3-SNAPSHOT-hadoop2-sources.jar to /Users/stack/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT/hbase-0.94.3-SNAPSHOT-sources.jar I cannot get install though to write the local repo w/ the names I told it to use. Looking at the install plugin, it uses maven itself copying artifacts (The above logging is the output of a repository 'event' as artifacts are copied in). I cannot put config on the install plugin to have it use my final name rather than the one it composes from base pom attributes. Anyone else have a suggestion? Is the test jar really needed? The hadoop1 built one doesn't work w/ hadoop2?
          Hide
          Lars Hofhansl added a comment -

          Jesse Yates Any ideas, Mr. build expert?

          Show
          Lars Hofhansl added a comment - Jesse Yates Any ideas, Mr. build expert?
          Hide
          Jonathan Hsieh added a comment -

          Since I believe the minicluster code is in the test package, I'm fairly sure any project that depends on this hbase+hadoop 2.0 (flume, sqoop to name a few) will want them.

          Maybe look at how bigtop deals with the different versions?

          Show
          Jonathan Hsieh added a comment - Since I believe the minicluster code is in the test package, I'm fairly sure any project that depends on this hbase+hadoop 2.0 (flume, sqoop to name a few) will want them. Maybe look at how bigtop deals with the different versions?
          Hide
          Jarek Jarcec Cecho added a comment -

          I can confirm that Sqoop needs this testing package to test hbase import feature.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - I can confirm that Sqoop needs this testing package to test hbase import feature. Jarcec
          Hide
          Enis Soztutar added a comment -

          @Stack I cannot agree more on maven not making any sense

          It seems from your tests, classifiers just does not work for tests, sources, etc with hadoop2. Then the only remaining option is to change the version string I guess.

          I am attaching a patch which does that. It seems it can work:

          mvn install -DskipTests  -Dhadoop.profile=2.0
          
          [INFO] --- maven-install-plugin:2.3:install (default-install) @ hbase ---
          [INFO] Installing /Users/enis/projects/hbase-0.94/target/hbase-0.94.3-SNAPSHOT-hadoop2.jar to /Users/enis/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT-hadoop2/hbase-0.94.3-SNAPSHOT-hadoop2.jar
          [INFO] Installing /Users/enis/projects/hbase-0.94/pom.xml to /Users/enis/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT-hadoop2/hbase-0.94.3-SNAPSHOT-hadoop2.pom
          [INFO] Installing /Users/enis/projects/hbase-0.94/target/hbase-0.94.3-SNAPSHOT-hadoop2-tests.jar to /Users/enis/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT-hadoop2/hbase-0.94.3-SNAPSHOT-hadoop2-tests.jar
          [INFO] Installing /Users/enis/projects/hbase-0.94/target/hbase-0.94.3-SNAPSHOT-hadoop2-sources.jar to /Users/enis/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT-hadoop2/hbase-0.94.3-SNAPSHOT-hadoop2-sources.jar
          

          The downside is that the version changes, which can bring it's own set of problems. Wdyt?

          Show
          Enis Soztutar added a comment - @Stack I cannot agree more on maven not making any sense It seems from your tests, classifiers just does not work for tests, sources, etc with hadoop2. Then the only remaining option is to change the version string I guess. I am attaching a patch which does that. It seems it can work: mvn install -DskipTests -Dhadoop.profile=2.0 [INFO] --- maven-install-plugin:2.3:install ( default -install) @ hbase --- [INFO] Installing /Users/enis/projects/hbase-0.94/target/hbase-0.94.3-SNAPSHOT-hadoop2.jar to /Users/enis/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT-hadoop2/hbase-0.94.3-SNAPSHOT-hadoop2.jar [INFO] Installing /Users/enis/projects/hbase-0.94/pom.xml to /Users/enis/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT-hadoop2/hbase-0.94.3-SNAPSHOT-hadoop2.pom [INFO] Installing /Users/enis/projects/hbase-0.94/target/hbase-0.94.3-SNAPSHOT-hadoop2-tests.jar to /Users/enis/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT-hadoop2/hbase-0.94.3-SNAPSHOT-hadoop2-tests.jar [INFO] Installing /Users/enis/projects/hbase-0.94/target/hbase-0.94.3-SNAPSHOT-hadoop2-sources.jar to /Users/enis/.m2/repository/org/apache/hbase/hbase/0.94.3-SNAPSHOT-hadoop2/hbase-0.94.3-SNAPSHOT-hadoop2-sources.jar The downside is that the version changes, which can bring it's own set of problems. Wdyt?
          Hide
          Roman Shaposhnik added a comment -

          Aren't you guys already publishing secure/unsecure bits? Adding a dedicated version would make it 4 then.

          Show
          Roman Shaposhnik added a comment - Aren't you guys already publishing secure/unsecure bits? Adding a dedicated version would make it 4 then.
          Hide
          Enis Soztutar added a comment -

          Since classifiers did not work, I could not find another way other than changing the version number for publishing the jars. I think we haven't published secure jars yet.

          Show
          Enis Soztutar added a comment - Since classifiers did not work, I could not find another way other than changing the version number for publishing the jars. I think we haven't published secure jars yet.
          Hide
          Roman Shaposhnik added a comment -

          So, isn't the lack of secure jars as much a problem as the lack of the ones compiled against different versions of Hadoop? I'm just trying to understand the scope of different 'twists' of HBase that one would have to know/care about.

          Show
          Roman Shaposhnik added a comment - So, isn't the lack of secure jars as much a problem as the lack of the ones compiled against different versions of Hadoop? I'm just trying to understand the scope of different 'twists' of HBase that one would have to know/care about.
          Hide
          Enis Soztutar added a comment -

          @Roman yes, linked HBASE-5341 for this. Specific to 0.94, we have incompatible jars for main, security and hadoop-2. Which should give us 4 jars:

          • main
          • security
          • hadoop-2
          • hadoop-2 security
            although the last one, I am not sure we care at this point. For trunk, there is no security jar, since we require at least hadoop-1 compatibility.

          Note that apache releases and maven repo are related but slightly different. For the release, we released the tarball and sources for main and security. We do not need to release the binaries compiled with hadoop2, since it can be obtained by just compiling the source with hadoop2. At future, we can decide to release the bits with h2 as well.

          Show
          Enis Soztutar added a comment - @Roman yes, linked HBASE-5341 for this. Specific to 0.94, we have incompatible jars for main, security and hadoop-2. Which should give us 4 jars: main security hadoop-2 hadoop-2 security although the last one, I am not sure we care at this point. For trunk, there is no security jar, since we require at least hadoop-1 compatibility. Note that apache releases and maven repo are related but slightly different. For the release, we released the tarball and sources for main and security. We do not need to release the binaries compiled with hadoop2, since it can be obtained by just compiling the source with hadoop2. At future, we can decide to release the bits with h2 as well.
          Hide
          Jarek Jarcec Cecho added a comment -

          Another possible solution would be to provide artifact hbase-test that would have classifier either hadoop1 or hadoop2 based on for which version it was compiled. This solution would be probably step back as for example hadoop was using it in the past and has moved to classifiers, but it should be working without need to have special version for testing artifact compiled against hadoop2.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Another possible solution would be to provide artifact hbase-test that would have classifier either hadoop1 or hadoop2 based on for which version it was compiled. This solution would be probably step back as for example hadoop was using it in the past and has moved to classifiers, but it should be working without need to have special version for testing artifact compiled against hadoop2. Jarcec
          Hide
          Roman Shaposhnik added a comment -

          Long time ago I filed HBASE-4850 but haven't had much time to devote to it since then. Is there any chance that somebody can help me working on it?

          Show
          Roman Shaposhnik added a comment - Long time ago I filed HBASE-4850 but haven't had much time to devote to it since then. Is there any chance that somebody can help me working on it?
          Hide
          Enis Soztutar added a comment -

          Another possible solution would be to provide artifact hbase-test that would have classifier either hadoop1 or hadoop2 based on for which version it was compiled

          We should not do this for h1, since that would break maven compatibility that our previous artifacts (0.94.0, 0.94.1, etc) was released with artifact name hbase, and classifier tests. However, maybe we can do it for hadoop2, not sure how easy it would be to create hbase-tests artifact with classifier hadoop-2. Since maven-deploy hardcodes "tests" classifier, this might not be possible at all.

          Long time ago I filed HBASE-4850 but haven't had much time to devote to it since then.

          What versions of hadoop? If between hadoop1 and hadoop2, then I don't think we can resolve this easily without creating a shim layer for pretty much everything.

          See:
          http://dbeech.github.com/hadoop-api-evolution.html

          Show
          Enis Soztutar added a comment - Another possible solution would be to provide artifact hbase-test that would have classifier either hadoop1 or hadoop2 based on for which version it was compiled We should not do this for h1, since that would break maven compatibility that our previous artifacts (0.94.0, 0.94.1, etc) was released with artifact name hbase, and classifier tests. However, maybe we can do it for hadoop2, not sure how easy it would be to create hbase-tests artifact with classifier hadoop-2. Since maven-deploy hardcodes "tests" classifier, this might not be possible at all. Long time ago I filed HBASE-4850 but haven't had much time to devote to it since then. What versions of hadoop? If between hadoop1 and hadoop2, then I don't think we can resolve this easily without creating a shim layer for pretty much everything. See: http://dbeech.github.com/hadoop-api-evolution.html
          Hide
          Roman Shaposhnik added a comment -

          If between hadoop1 and hadoop2, then I don't think we can resolve this easily

          It is not super trivial, but it isn't insurmountable either. Besides, we can borrow generously from some of the projects that did some of it already. In short – it definitely can be done, but it requires more than just yours truly

          Show
          Roman Shaposhnik added a comment - If between hadoop1 and hadoop2, then I don't think we can resolve this easily It is not super trivial, but it isn't insurmountable either. Besides, we can borrow generously from some of the projects that did some of it already. In short – it definitely can be done, but it requires more than just yours truly
          Hide
          stack added a comment -

          Yes, we have not published a secure jar to maven. We should have. We just shipped two tgz bundles, secure and insecure.

          Yes, this is a problem we need to fix.

          On changing the version being the only way around this issue, I agree. Jon Hsieh and I talked w/ a Maven'y fellow who suggested similar – Andrew Bayer. Even if we could make it work w/ classifiers, it would probably be a deceit; I would not trust that downloaded, maven would pull in the proper dependencies.

          The hadoop-api-evolution table made me violently sick.

          Roman, on HBASE-4850, I don't think we can do what is suggested there (Commented to that effect).

          I like the patch you suggest Enis.

          What do we do in hbase 1.0, say, where minimum required is indeed h2. Don't we want our version to be hbase in that case and not hbase-hadoop2? You think w/ your patch, maven repo will have hbase-hadoop2 and it will be just fine when hbase itself moves to hadoop2? (We'll have a hadoop3 version around this time too?)

          Oh, we don't need to do a security artifact in maven for 0.96. We should have published one for 0.94 and before.

          Show
          stack added a comment - Yes, we have not published a secure jar to maven. We should have. We just shipped two tgz bundles, secure and insecure. Yes, this is a problem we need to fix. On changing the version being the only way around this issue, I agree. Jon Hsieh and I talked w/ a Maven'y fellow who suggested similar – Andrew Bayer. Even if we could make it work w/ classifiers, it would probably be a deceit; I would not trust that downloaded, maven would pull in the proper dependencies. The hadoop-api-evolution table made me violently sick. Roman, on HBASE-4850 , I don't think we can do what is suggested there (Commented to that effect). I like the patch you suggest Enis. What do we do in hbase 1.0, say, where minimum required is indeed h2. Don't we want our version to be hbase in that case and not hbase- hadoop2? You think w/ your patch, maven repo will have hbase -hadoop2 and it will be just fine when hbase itself moves to hadoop2? (We'll have a hadoop3 version around this time too?) Oh, we don't need to do a security artifact in maven for 0.96. We should have published one for 0.94 and before.
          Hide
          Jarek Jarcec Cecho added a comment -

          Another idea - version number can contain string. So instead of creating special dot release, what about creating release "$

          {normalversion}

          -hadoop2"?

          Show
          Jarek Jarcec Cecho added a comment - Another idea - version number can contain string. So instead of creating special dot release, what about creating release "$ {normalversion} -hadoop2"?
          Hide
          Enis Soztutar added a comment -

          @Stack

          What do we do in hbase 1.0, say, where minimum required is indeed h2. Don't we want our version to be hbase in that case and not hbase-hadoop2? You think w/ your patch, maven repo will have hbase-hadoop2 and it will be just fine when hbase itself moves to hadoop2? (We'll have a hadoop3 version around this time too?)

          If Hbase moves to hadoop2, and drops support for h1, then we can just call the main version hbase without the hadoop2 suffix. As long as which version of HBase requires which version of hadoop, it should not be too confusing, wdyt?
          @Jarek

          Another idea - version number can contain string. So instead of creating special dot release, what about creating release "${normalversion}-hadoop2"?

          This is what I did in the patch. We will release for example hbase-0.94.3-hadoop2.

          Show
          Enis Soztutar added a comment - @Stack What do we do in hbase 1.0, say, where minimum required is indeed h2. Don't we want our version to be hbase in that case and not hbase-hadoop2? You think w/ your patch, maven repo will have hbase-hadoop2 and it will be just fine when hbase itself moves to hadoop2? (We'll have a hadoop3 version around this time too?) If Hbase moves to hadoop2, and drops support for h1, then we can just call the main version hbase without the hadoop2 suffix. As long as which version of HBase requires which version of hadoop, it should not be too confusing, wdyt? @Jarek Another idea - version number can contain string. So instead of creating special dot release, what about creating release "${normalversion}-hadoop2"? This is what I did in the patch. We will release for example hbase-0.94.3-hadoop2.
          Hide
          Jarek Jarcec Cecho added a comment -

          Enis Soztutar I see, I originally thought that you want to increase the version number. Adding "-hadoop2" suffix seems as a reasonable workaround to me, but it's up to you HBase guys to agree on the final solution

          Show
          Jarek Jarcec Cecho added a comment - Enis Soztutar I see, I originally thought that you want to increase the version number. Adding "-hadoop2" suffix seems as a reasonable workaround to me, but it's up to you HBase guys to agree on the final solution
          Hide
          Roman Shaposhnik added a comment -

          @stack

          if HBASE-4850 is not an option I believe you guys are on the right track. You definitely don't want to use classifier for this so that leaves mucking with the version string. As long as you come up with a reasonable encoding strategy for pointing at various artifacts (e.g. hadoop2-secure, etc.) you can append that to the version string. Watch out for things in the pom.xml that need to be tweaked for each of these combinations – I got bitten by default settings in specially versioned pom files a couple of times.

          Show
          Roman Shaposhnik added a comment - @stack if HBASE-4850 is not an option I believe you guys are on the right track. You definitely don't want to use classifier for this so that leaves mucking with the version string. As long as you come up with a reasonable encoding strategy for pointing at various artifacts (e.g. hadoop2-secure, etc.) you can append that to the version string. Watch out for things in the pom.xml that need to be tweaked for each of these combinations – I got bitten by default settings in specially versioned pom files a couple of times.
          Hide
          stack added a comment -

          I don't think hbase-4850 possible as of this writing. It may become so by the time we release (you might just have to shuffle the CLASSPATH if we move all h1 and h2 mannerisms successfully back into their respective compatibility modules). Meantime, messing w/ version string seems the way to go.

          Is there a hurry on solving this one? I'd like to do a bit of testing and leave the decision on what we'll publish till close to release if that is ok w/ bigtop and other downstreamers.

          Show
          stack added a comment - I don't think hbase-4850 possible as of this writing. It may become so by the time we release (you might just have to shuffle the CLASSPATH if we move all h1 and h2 mannerisms successfully back into their respective compatibility modules). Meantime, messing w/ version string seems the way to go. Is there a hurry on solving this one? I'd like to do a bit of testing and leave the decision on what we'll publish till close to release if that is ok w/ bigtop and other downstreamers.
          Hide
          stack added a comment -

          Hmm... then again this issue wants 0.94 artifacts. Should we go ahead and commit the Enis patch and publish 0.94 snapshots w/ the hadoop2 versions published with a version that has a hadoop2 suffix? Patch looks good. Should we publish even though all tests do not yet pass against 0.94?

          Show
          stack added a comment - Hmm... then again this issue wants 0.94 artifacts. Should we go ahead and commit the Enis patch and publish 0.94 snapshots w/ the hadoop2 versions published with a version that has a hadoop2 suffix? Patch looks good. Should we publish even though all tests do not yet pass against 0.94?
          Hide
          Jarek Jarcec Cecho added a comment -

          As of now, Sqoop is simply skipping HBase tests on Hadoop 2, so we do have working workaround and we could potentially wait. But I would personally prefer to solve it as soon as possible as I do not like current situation :-/

          Having also HBase 0.92 test jars compiled against Hadoop 2 would be nice, so that we could test against them as well and make sure that everything is working as expected.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - As of now, Sqoop is simply skipping HBase tests on Hadoop 2, so we do have working workaround and we could potentially wait. But I would personally prefer to solve it as soon as possible as I do not like current situation :-/ Having also HBase 0.92 test jars compiled against Hadoop 2 would be nice, so that we could test against them as well and make sure that everything is working as expected. Jarcec
          Hide
          Hari Shreedharan added a comment -

          What is the status of this issue? Right now, even Flume requires that the user builds hbase locally before building Flume. We'd like to avoid this, so it'd be great if this is committed soon. Thanks!

          Show
          Hari Shreedharan added a comment - What is the status of this issue? Right now, even Flume requires that the user builds hbase locally before building Flume. We'd like to avoid this, so it'd be great if this is committed soon. Thanks!
          Hide
          stack added a comment -

          Jarek Jarcec Cecho and Hari Shreedharan I thought Roman said this is not necessary? That it not necessary we publish an hbase 0.94 artifact built against hadoop2? You fellas still need it? If so, we'll publish one but it'll be a version of 0.94.2-hadoop2. Would that work for you?

          Show
          stack added a comment - Jarek Jarcec Cecho and Hari Shreedharan I thought Roman said this is not necessary? That it not necessary we publish an hbase 0.94 artifact built against hadoop2? You fellas still need it? If so, we'll publish one but it'll be a version of 0.94.2-hadoop2. Would that work for you?
          Hide
          Hari Shreedharan added a comment -

          Stack If there are no maven artifacts available, then it would require a user building flume/sqoop against hadoop2 to build hbase against hadoop2 first - which is not something we really desire. If it is possible to push the artifacts it would be great. I can update the Flume pom to point to the hadoop2 version of hbase in the hadoop2 profile - so 0.94.2-hadoop2 would work fine. Also, could you give an ETA, since Flume-1.3.0 is currently being voted on, I'd like this to go into the release - so if this does not take too long, it'd be great, I will ask for another RC.

          Thanks!

          Show
          Hari Shreedharan added a comment - Stack If there are no maven artifacts available, then it would require a user building flume/sqoop against hadoop2 to build hbase against hadoop2 first - which is not something we really desire. If it is possible to push the artifacts it would be great. I can update the Flume pom to point to the hadoop2 version of hbase in the hadoop2 profile - so 0.94.2-hadoop2 would work fine. Also, could you give an ETA, since Flume-1.3.0 is currently being voted on, I'd like this to go into the release - so if this does not take too long, it'd be great, I will ask for another RC. Thanks!
          Hide
          stack added a comment -

          Hari Shreedharan What if I did it as hbase-0.94.2-hadoop2-SNAPSHOT? Would that work? i.e. SNAPSHOT? I'm wary making a hbase-0.94.2-hadoop2 release since no testing of 0.94 against hadoop2 done really.

          Show
          stack added a comment - Hari Shreedharan What if I did it as hbase-0.94.2-hadoop2-SNAPSHOT? Would that work? i.e. SNAPSHOT? I'm wary making a hbase-0.94.2-hadoop2 release since no testing of 0.94 against hadoop2 done really.
          Hide
          Jarek Jarcec Cecho added a comment -

          I definitely agree with Hari - forcing users to build their own HBase in order to test Sqoop do not seems as a good way to go, so I would prefer available upstream jar if possible. I don't mind having version hbase-0.94.2-hadoop2-SNAPSHOT with the "SNAPSHOT" part. I'll properly document why we're depending on SNAPSHOT and link this issue for better understanding...

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - I definitely agree with Hari - forcing users to build their own HBase in order to test Sqoop do not seems as a good way to go, so I would prefer available upstream jar if possible. I don't mind having version hbase-0.94.2-hadoop2-SNAPSHOT with the "SNAPSHOT" part. I'll properly document why we're depending on SNAPSHOT and link this issue for better understanding... Jarcec
          Hide
          Hari Shreedharan added a comment -

          Stack I agree with Jarcec. A snapshot is fine - we shall document it.

          Show
          Hari Shreedharan added a comment - Stack I agree with Jarcec. A snapshot is fine - we shall document it.
          Hide
          Roman Shaposhnik added a comment -

          A couple of points

          1. I was saying that if HBase project would not want to get into the business of certifying that HBase is known to work against Hadoop 2 it can safely NOT publish the artifacts and expect downstream projects to build local copies of HBase
          2. Bigtop 0.4.0 and 0.5.0 do fair amount of testing of the HBase 0.94.X and Hadoop 2.0.X combination
          3. If you keep #1 and #2 in mind – you might be interested in outsourcing the management of the hadoop2 HBase artifact to the Bigtop project. If that is an attractive option – lets talk.
          Show
          Roman Shaposhnik added a comment - A couple of points I was saying that if HBase project would not want to get into the business of certifying that HBase is known to work against Hadoop 2 it can safely NOT publish the artifacts and expect downstream projects to build local copies of HBase Bigtop 0.4.0 and 0.5.0 do fair amount of testing of the HBase 0.94.X and Hadoop 2.0.X combination If you keep #1 and #2 in mind – you might be interested in outsourcing the management of the hadoop2 HBase artifact to the Bigtop project. If that is an attractive option – lets talk.
          Hide
          stack added a comment -

          Roman Shaposhnik I just tried deploying an hbase artifact built against hadoop2 and its a PITA; the pom needs a bunch of work it seems to get it to work (mvn trying to pull in hadoop-core-2.0.2-alpha and ditto w/ hadoop-test).

          We want to certify 0.96 against hadoop2, not 0.94.

          So, I'm up for talking Mr. Roman.

          Show
          stack added a comment - Roman Shaposhnik I just tried deploying an hbase artifact built against hadoop2 and its a PITA; the pom needs a bunch of work it seems to get it to work (mvn trying to pull in hadoop-core-2.0.2-alpha and ditto w/ hadoop-test). We want to certify 0.96 against hadoop2, not 0.94. So, I'm up for talking Mr. Roman.
          Hide
          Roshan Naik added a comment -

          checking to see if hbase is still planning to have hadoop2 binaries published for 0.96 ?

          Show
          Roshan Naik added a comment - checking to see if hbase is still planning to have hadoop2 binaries published for 0.96 ?
          Hide
          stack added a comment -

          Roshan Naik Here is our attempt: https://repository.apache.org/content/groups/snapshots/org/apache/hbase/hbase-client/ What do you think? Will it work for you? We publish two artifacts, one named w/ a hadoop1 suffix and the other with a hadoop2. Would be interested in feedback. What I have so far is that this scheme will not work (our Elliott has it our hadoop dependency is resolved before we have a chance to influence it – profiles).

          Show
          stack added a comment - Roshan Naik Here is our attempt: https://repository.apache.org/content/groups/snapshots/org/apache/hbase/hbase-client/ What do you think? Will it work for you? We publish two artifacts, one named w/ a hadoop1 suffix and the other with a hadoop2. Would be interested in feedback. What I have so far is that this scheme will not work (our Elliott has it our hadoop dependency is resolved before we have a chance to influence it – profiles).
          Hide
          Hari Shreedharan added a comment -

          Stack, This should work for Flume. We have different mvn profiles for hadoop-1 and hadoop-2. So we can just change the hadoop-2 profile to pull in the correct artifact (we already do this for hadoop-common/hadoop-core).

          Show
          Hari Shreedharan added a comment - Stack , This should work for Flume. We have different mvn profiles for hadoop-1 and hadoop-2. So we can just change the hadoop-2 profile to pull in the correct artifact (we already do this for hadoop-common/hadoop-core).
          Hide
          Roshan Naik added a comment -

          there will be a little complexity ..

          Hbase jar has been split up in 0.95.. so in addition to the hadoop2 profile flume (and others such prjs) will now additionally need a profile for hbase0.95.

          the complexity is that neither the hadoop2 profile nor the hbase0.95+ profile can independently conjure up the full name and dependencies list for hbase.

          let me give it a try.. will get back.

          Show
          Roshan Naik added a comment - there will be a little complexity .. Hbase jar has been split up in 0.95.. so in addition to the hadoop2 profile flume (and others such prjs) will now additionally need a profile for hbase0.95. the complexity is that neither the hadoop2 profile nor the hbase0.95+ profile can independently conjure up the full name and dependencies list for hbase. let me give it a try.. will get back.
          Hide
          Hari Shreedharan added a comment -

          Roshan: we should not need anything more than the HBase client jars (I believe - not sure where the minicluster classes for tests are and what we'd need to handle that though). So we should have to do only what we do hadoop-common v/s hadoop-core.

          Show
          Hari Shreedharan added a comment - Roshan: we should not need anything more than the HBase client jars (I believe - not sure where the minicluster classes for tests are and what we'd need to handle that though). So we should have to do only what we do hadoop-common v/s hadoop-core.
          Hide
          Enis Soztutar added a comment -

          I think Flume needs at least the hbase-server tests jar to use the minicluster. We currently do not have hbase-minicluster kind of module.

          Show
          Enis Soztutar added a comment - I think Flume needs at least the hbase-server tests jar to use the minicluster. We currently do not have hbase-minicluster kind of module.
          Hide
          Roshan Naik added a comment -

          Hari: it looks like flume needs these .. hbase-client, hbase-common (HBaseCommonTestingUtility.. at compile time), hbase-server (HMaster,MiniZooKeeperCluster.. at compile time) and hbase-server-test (HBaseTestingUtility..when running tests)

          Show
          Roshan Naik added a comment - Hari: it looks like flume needs these .. hbase-client, hbase-common (HBaseCommonTestingUtility.. at compile time), hbase-server (HMaster,MiniZooKeeperCluster.. at compile time) and hbase-server-test (HBaseTestingUtility..when running tests)
          Hide
          Hari Shreedharan added a comment -

          Yep, we could define those in the hadoop-2 profile as dependencies.

          Show
          Hari Shreedharan added a comment - Yep, we could define those in the hadoop-2 profile as dependencies.
          Hide
          Roshan Naik added a comment -

          these are not hadoop2 specific.... these are hbase0.95+ specific which comes in both hadoop1&2 flovaours. So it seems we will need to support the following combos...

          1) building against hbase 0.94x .. in this case there will be no -hadoop1 suffix on hbase deps

          2) building against hbase 0.95+ with hadoop 1 .. in this case the -hadoop1 suffix will be used with the newer dependencies list

          3) building against hbase 0.95+ with hadoop 2 .. in this case the -hadoop2 suffix will be used with the newer dependency list

          Show
          Roshan Naik added a comment - these are not hadoop2 specific.... these are hbase0.95+ specific which comes in both hadoop1&2 flovaours. So it seems we will need to support the following combos... 1) building against hbase 0.94x .. in this case there will be no -hadoop1 suffix on hbase deps 2) building against hbase 0.95+ with hadoop 1 .. in this case the -hadoop1 suffix will be used with the newer dependencies list 3) building against hbase 0.95+ with hadoop 2 .. in this case the -hadoop2 suffix will be used with the newer dependency list
          Hide
          Andrew Purtell added a comment -

          This came up again today over on phoenix#286: https://github.com/forcedotcom/phoenix/issues/286 . Looking for HBase 0.94 artifacts built against Hadoop 2.

          Show
          Andrew Purtell added a comment - This came up again today over on phoenix#286: https://github.com/forcedotcom/phoenix/issues/286 . Looking for HBase 0.94 artifacts built against Hadoop 2.
          Hide
          stack added a comment -

          We could do something like HBASE-8224, a script that creates a pom.xml.hadoop2 from the original pom.xml w/ interpolated version – e.g. 0.94.12-hadoop2 – and that does other search and replace. We'd then use this new pom to publish SNAPSHOTs and for wrestling with mvn release plugin. Should be easier in 0.94 than in 0.96 since we are not modularized in 0.94. I could help out w/ this after 0.95.2 goes up if folks interested still.

          Show
          stack added a comment - We could do something like HBASE-8224 , a script that creates a pom.xml.hadoop2 from the original pom.xml w/ interpolated version – e.g. 0.94.12-hadoop2 – and that does other search and replace. We'd then use this new pom to publish SNAPSHOTs and for wrestling with mvn release plugin. Should be easier in 0.94 than in 0.96 since we are not modularized in 0.94. I could help out w/ this after 0.95.2 goes up if folks interested still.
          Hide
          Roshan Naik added a comment -

          Would it help to give the hadoop2 based hbase project a different name instead of sufix ? something like hbase-hadoop2-XX.YY.ZZ

          sort of like win32 v/s win64

          Show
          Roshan Naik added a comment - Would it help to give the hadoop2 based hbase project a different name instead of sufix ? something like hbase-hadoop2-XX.YY.ZZ sort of like win32 v/s win64
          Hide
          stack added a comment -

          Roshan Naik A different version – a version with -hadoop2 in it – worked over in hbase-8224.

          Show
          stack added a comment - Roshan Naik A different version – a version with -hadoop2 in it – worked over in hbase-8224.

            People

            • Assignee:
              Unassigned
              Reporter:
              Enis Soztutar
            • Votes:
              3 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:

                Development