HBase
  1. HBase
  2. HBASE-1433

Update hbase build to match core, use ivy, publish jars to maven repo, etc.

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 0.90.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      We are almost close, except for the republication of artifacts to mapreduce snapshot repository from branch-0.21 (after applying the appropriate patch). It would be great to have this out for 0.21 .

      1. HBASE-1433.patch
        49 kB
        Karthik K
      2. HBASE-1433.patch
        38 kB
        Karthik K
      3. HBASE-1433.patch
        39 kB
        Karthik K
      4. HBASE-1433.patch
        38 kB
        Karthik K
      5. HBASE-1433.patch
        41 kB
        Karthik K
      6. HBASE-1433.patch
        39 kB
        Karthik K
      7. HBASE-1433.patch
        40 kB
        Karthik K
      8. HBASE-1433.patch
        41 kB
        Karthik K

        Issue Links

          Activity

          Hide
          Karthik K added a comment -

          stack - curious to see what the status on this is ..

          I have a patch that uses ivy to do the dependency management and pulls the necessary files - except for the following dependencies ( thrift / protocol buffers (contrib-stargate ) and zookeeper ).

          For now - may be we can just version those piece of software until the ivy repositories become available. Else - dependency management using ivy should be available soon.

          Show
          Karthik K added a comment - stack - curious to see what the status on this is .. I have a patch that uses ivy to do the dependency management and pulls the necessary files - except for the following dependencies ( thrift / protocol buffers (contrib-stargate ) and zookeeper ). For now - may be we can just version those piece of software until the ivy repositories become available. Else - dependency management using ivy should be available soon.
          Hide
          Karthik K added a comment -

          Introduce ivy based dependency management - first cut.

          Show
          Karthik K added a comment - Introduce ivy based dependency management - first cut.
          Hide
          Karthik K added a comment -

          Depends on apache snapshots for hadoop-core / hdfs-core / mapreduce-core projects.

          During development - if we want stability - we can maintain another ivy resolver with specific snapshots that we are interested. But for production and releases - this should suffice.

          Protocol Buffers
          Thrift
          Zookeeper do not seem to publish their (maven/ivy) artifacts publicly yet.

          Until that happens - those binaries are maintained separately in version control and after the public repositories are available , they will move away in favor of ivy.xml (yes - it is ugly now !)

          $ ant compile
          succeeds

          $ ant test
          succeeds

          Show
          Karthik K added a comment - Depends on apache snapshots for hadoop-core / hdfs-core / mapreduce-core projects. During development - if we want stability - we can maintain another ivy resolver with specific snapshots that we are interested. But for production and releases - this should suffice. Protocol Buffers Thrift Zookeeper do not seem to publish their (maven/ivy) artifacts publicly yet. Until that happens - those binaries are maintained separately in version control and after the public repositories are available , they will move away in favor of ivy.xml (yes - it is ugly now !) $ ant compile succeeds $ ant test succeeds
          Hide
          Karthik K added a comment -

          protocol buffers seems to be available here- http://google-maven-repository.googlecode.com/svn/repository/com/google/protobuf/protobuf-java/ .

          Apparently - there is an issue with the maven repository keys here at - http://code.google.com/p/protobuf/issues/detail?id=126&q=ivy&colspec=ID%20Type%20Status%20Priority%20FixedIn%20Owner%20Summary .

          After it goes in - protobuf-java can also be deleted from the source control and added in ivy .

          Show
          Karthik K added a comment - protocol buffers seems to be available here- http://google-maven-repository.googlecode.com/svn/repository/com/google/protobuf/protobuf-java/ . Apparently - there is an issue with the maven repository keys here at - http://code.google.com/p/protobuf/issues/detail?id=126&q=ivy&colspec=ID%20Type%20Status%20Priority%20FixedIn%20Owner%20Summary . After it goes in - protobuf-java can also be deleted from the source control and added in ivy .
          Hide
          Karthik K added a comment -

          Voting for it since I have vested interest in upgrading the Lucene index version (2.2 , as of now to 2.9/3.0+) backing the same, and having a proper dependency management system makes it easy .

          Show
          Karthik K added a comment - Voting for it since I have vested interest in upgrading the Lucene index version (2.2 , as of now to 2.9/3.0+) backing the same, and having a proper dependency management system makes it easy .
          Hide
          Karthik K added a comment -

          $ ant test
          succeeds

          Sorry - spoke too soon. Three test cases failed due to dynamic linking issue.

          java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/apache/hadoop/conf/Configuration;Ljavax/net/SocketFactory;)Lorg/apache/hadoop/ipc/VersionedProtocol;

          [junit] Test org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan FAILED
          [junit] Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
          [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 40.874 sec
          [junit] Test org.apache.hadoop.hbase.mapreduce.TestTableMapReduce FAILED
          [junit] Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
          [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 29.85 sec
          [junit] Test org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed FAILED
          [junit] Running org.apache.hadoop.hbase.master.TestMinimumServerCount

          May be a subsequent snapshot of mapreduce library would resolve the inconsistency.

          Show
          Karthik K added a comment - $ ant test succeeds Sorry - spoke too soon. Three test cases failed due to dynamic linking issue. java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/apache/hadoop/conf/Configuration;Ljavax/net/SocketFactory;)Lorg/apache/hadoop/ipc/VersionedProtocol; [junit] Test org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan FAILED [junit] Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 40.874 sec [junit] Test org.apache.hadoop.hbase.mapreduce.TestTableMapReduce FAILED [junit] Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 29.85 sec [junit] Test org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed FAILED [junit] Running org.apache.hadoop.hbase.master.TestMinimumServerCount May be a subsequent snapshot of mapreduce library would resolve the inconsistency.
          Hide
          Karthik K added a comment -

          ZooKeeper needed for contrib/transactional/test as well

          Show
          Karthik K added a comment - ZooKeeper needed for contrib/transactional/test as well
          Hide
          Karthik K added a comment -

          Thrift dependency: THRIFT-363 and Zookeeper dependency: ZOOKEEPER-224 ( for ivy artifacts to be publicly available ).

          Show
          Karthik K added a comment - Thrift dependency: THRIFT-363 and Zookeeper dependency: ZOOKEEPER-224 ( for ivy artifacts to be publicly available ).
          Hide
          stack added a comment -

          @kay kay Checking in the dependencies not available yet in maven repos is the way to go. Let me check out your patch... will be back (Thanks for doing this).

          Show
          stack added a comment - @kay kay Checking in the dependencies not available yet in maven repos is the way to go. Let me check out your patch... will be back (Thanks for doing this).
          Hide
          stack added a comment -

          @kay kay How'd you fix the above "dynamic linking" issue. I get that for core tests at least after I changed your patch so it pulled hadoop 0.21 jars rather than hadoop 0.22 jars (hbase TRUNK uses hadoop 0.21). It seems too like there other jars we could fetch like the content of lib/jsp-2.1 and in stargate, there seems to be jackson-json? Thanks KK.

          Show
          stack added a comment - @kay kay How'd you fix the above "dynamic linking" issue. I get that for core tests at least after I changed your patch so it pulled hadoop 0.21 jars rather than hadoop 0.22 jars (hbase TRUNK uses hadoop 0.21). It seems too like there other jars we could fetch like the content of lib/jsp-2.1 and in stargate, there seems to be jackson-json? Thanks KK.
          Hide
          Karthik K added a comment -

          How'd you fix the above "dynamic linking" issue.

          I have not. That is tricky because it pulls the latest snapshot across three different projects ( core . hdfs, mapreduce ) . Also - one of the hdfs snapshots (0.21 ) seemed to pull (0.22) from core. We also need to have some sort of conflict resolver specific to core / hdfs / mapreduce to prevent similar conflicts ( as opposed to the default one ).

          This specific dll conflict seems to be an artifact of HADOOP-6435 checked in recently and corresponding mismatch in other projects in picking up the same.

          For a release - this would work perfect to avoid the conflicts of different versions. During development though - this would be a headache though during such major API changes. But - may be we can maintain a separate repository during development and flip the order of ivy resolvers for ( core / hdfs/ mapreduce ).

          I get that for core tests at least after I changed your patch so it pulled hadoop 0.21 jars rather than hadoop 0.22 jars

          Yup. I changed it to see if it avoids the dll linkage issue, but if 0.21 is what we want - we also want to add a custom ivy conflict resolvers to allow only release versions ( and maintain major / minor version compatibility ). I will look into the same.

          It seems too like there other jars we could fetch like the content of lib/jsp-2.1 and in stargate, there seems to be jackson-json?

          Yes - lib/jsp-2.1 , can also brought under ivy umbrella .

          For json - there is this -

          <dependency org="com.sun.jersey" name="jersey-json"
          rev="$

          {jersey.version}

          " conf="common->default">

          If we are looking for a specific json implementation - we can change this.

          Show
          Karthik K added a comment - How'd you fix the above "dynamic linking" issue. I have not. That is tricky because it pulls the latest snapshot across three different projects ( core . hdfs, mapreduce ) . Also - one of the hdfs snapshots (0.21 ) seemed to pull (0.22) from core. We also need to have some sort of conflict resolver specific to core / hdfs / mapreduce to prevent similar conflicts ( as opposed to the default one ). This specific dll conflict seems to be an artifact of HADOOP-6435 checked in recently and corresponding mismatch in other projects in picking up the same. For a release - this would work perfect to avoid the conflicts of different versions. During development though - this would be a headache though during such major API changes. But - may be we can maintain a separate repository during development and flip the order of ivy resolvers for ( core / hdfs/ mapreduce ). I get that for core tests at least after I changed your patch so it pulled hadoop 0.21 jars rather than hadoop 0.22 jars Yup. I changed it to see if it avoids the dll linkage issue, but if 0.21 is what we want - we also want to add a custom ivy conflict resolvers to allow only release versions ( and maintain major / minor version compatibility ). I will look into the same. It seems too like there other jars we could fetch like the content of lib/jsp-2.1 and in stargate, there seems to be jackson-json? Yes - lib/jsp-2.1 , can also brought under ivy umbrella . For json - there is this - <dependency org="com.sun.jersey" name="jersey-json" rev="$ {jersey.version} " conf="common->default"> If we are looking for a specific json implementation - we can change this.
          Hide
          Karthik K added a comment -
          It seems too like there other jars we could fetch like the content of lib/jsp-2.1 and in stargate, there seems to be jackson-json?

          jersey-json depends on jackson-json-asl (1.1.1) transitively

          Show
          Karthik K added a comment - It seems too like there other jars we could fetch like the content of lib/jsp-2.1 and in stargate, there seems to be jackson-json? jersey-json depends on jackson-json-asl (1.1.1) transitively
          Hide
          Karthik K added a comment -
          fetch like the content of lib/jsp-2.1

          jsp-2.1 seems to be fetched from the dependency of hadoop-core ( jsp.jar and jsp-api.jar ). So the lib/jsp-2.1/* files could be deleted as well.

          Show
          Karthik K added a comment - fetch like the content of lib/jsp-2.1 jsp-2.1 seems to be fetched from the dependency of hadoop-core ( jsp.jar and jsp-api.jar ). So the lib/jsp-2.1/* files could be deleted as well.
          Hide
          stack added a comment -

          .bq I have not. That is tricky because it pulls the latest snapshot across three different projects ( core . hdfs, mapreduce ) . Also - one of the hdfs snapshots (0.21 ) seemed to pull (0.22) from core. We also need to have some sort of conflict resolver specific to core / hdfs / mapreduce to prevent similar conflicts ( as opposed to the default one ).

          A custom conflict resolver sounds like loads of fun (smile). If a 0.21 hdfs snapshot is pulling in 0.22, that sounds well broke.

          How about putting up a patch that leaves the hadoop stuff as checkins and not pull these from a repo just yet (or, maybe if you just check in commons, and pull the other two, does that help)? I suggest this because the patch as is where it moves the bulk of the jar includes to instead be ivy pulls is a nice fat contribution. It also gets you what you need, an updated lucene?

          Good stuff on the fact that jsp-2.1 is being pulled already and jersery-json is a transitive include.

          Show
          stack added a comment - .bq I have not. That is tricky because it pulls the latest snapshot across three different projects ( core . hdfs, mapreduce ) . Also - one of the hdfs snapshots (0.21 ) seemed to pull (0.22) from core. We also need to have some sort of conflict resolver specific to core / hdfs / mapreduce to prevent similar conflicts ( as opposed to the default one ). A custom conflict resolver sounds like loads of fun (smile). If a 0.21 hdfs snapshot is pulling in 0.22, that sounds well broke. How about putting up a patch that leaves the hadoop stuff as checkins and not pull these from a repo just yet (or, maybe if you just check in commons, and pull the other two, does that help)? I suggest this because the patch as is where it moves the bulk of the jar includes to instead be ivy pulls is a nice fat contribution. It also gets you what you need, an updated lucene? Good stuff on the fact that jsp-2.1 is being pulled already and jersery-json is a transitive include.
          Hide
          Karthik K added a comment -

          MAPREDUCE-1352 and HDFS-869 logged to make sure that the pom published in apache snapshot repository is consistent with the version in branch 0.21 .

          Expected: 0.21.0 for hadoop-core . Actuals: 0.21.0 for hadoop-core in either of them. That gives rise to quite a lot of confusion here.

          Show
          Karthik K added a comment - MAPREDUCE-1352 and HDFS-869 logged to make sure that the pom published in apache snapshot repository is consistent with the version in branch 0.21 . Expected: 0.21.0 for hadoop-core . Actuals: 0.21.0 for hadoop-core in either of them. That gives rise to quite a lot of confusion here.
          Hide
          Karthik K added a comment -

          So far - it seems like the .pom published to mapreduce-0.21 and hdfs-0.21 refers to hadoop-core-0.22 by mistake (when they had intended to mean hadoop-core-0.21 ) . The teams are working on fixing it and republishing artifacts again.

          Independently - mapreduce-0.21 does not have a green build for a long time in hudson due to the same dynamic link issue that we are facing.

          After hdfs gets a green build with hadoop-core-0.21 dependency - mapreduce needs to do the same thing again. At which point- I expect this dynamic link issue to be resolved and ivy based dependency management in place ( except for zookeeper / thrift ) for us.

          Show
          Karthik K added a comment - So far - it seems like the .pom published to mapreduce-0.21 and hdfs-0.21 refers to hadoop-core-0.22 by mistake (when they had intended to mean hadoop-core-0.21 ) . The teams are working on fixing it and republishing artifacts again. Independently - mapreduce-0.21 does not have a green build for a long time in hudson due to the same dynamic link issue that we are facing. After hdfs gets a green build with hadoop-core-0.21 dependency - mapreduce needs to do the same thing again. At which point- I expect this dynamic link issue to be resolved and ivy based dependency management in place ( except for zookeeper / thrift ) for us.
          Hide
          stack added a comment -

          @ Kay Kay: Good stuff. Seems like we should just commit the pom changes over in mapreduce-1352 (though it looks like the pom change may have broken at least one unit test) and same for hdfs-869. Whats there is just broke. Is there an issue up in hadoop for fixing the dynamic link issue or do you think addressing the mapreduce-1352 and hdfs-869 will take care of it?

          Show
          stack added a comment - @ Kay Kay: Good stuff. Seems like we should just commit the pom changes over in mapreduce-1352 (though it looks like the pom change may have broken at least one unit test) and same for hdfs-869. Whats there is just broke. Is there an issue up in hadoop for fixing the dynamic link issue or do you think addressing the mapreduce-1352 and hdfs-869 will take care of it?
          Hide
          Karthik K added a comment -

          I believe mapreduce-1352 and hdfs-869 once fixed - should make this green (hopefully ).

          Show
          Karthik K added a comment - I believe mapreduce-1352 and hdfs-869 once fixed - should make this green (hopefully ).
          Hide
          Karthik K added a comment -
          Expected: 0.21.0 for hadoop-core . Actuals: 0.21.0 for hadoop-core in either of them

          Oops meant - "Expected: 0.21.0 for hadoop-core . Actuals: 0.22.0 for hadoop-core in either of them"

          Show
          Karthik K added a comment - Expected: 0.21.0 for hadoop-core . Actuals: 0.21.0 for hadoop-core in either of them Oops meant - "Expected: 0.21.0 for hadoop-core . Actuals: 0.22.0 for hadoop-core in either of them"
          Hide
          Karthik K added a comment -

          A good number of configurations in ivy.xml have been copied from hadoop-core and hence not necessary. Will revisit that after getting the rest of the dependent issues resolved.

          Show
          Karthik K added a comment - A good number of configurations in ivy.xml have been copied from hadoop-core and hence not necessary. Will revisit that after getting the rest of the dependent issues resolved.
          Hide
          Karthik K added a comment -

          Having 3 separate ivysettings.xml and libraries.properties seems like an overkill. We can do with 1 . More on the TODO list.

          Show
          Karthik K added a comment - Having 3 separate ivysettings.xml and libraries.properties seems like an overkill. We can do with 1 . More on the TODO list.
          Hide
          Karthik K added a comment -

          hadoop-core ivy configuration, conf="common->default" probably needs to be revisited specifically mentioning only the contexts needed for hbase code and not everything else. 'default' is not something we would want. A comma separated , fine-grained dependency is needed.

          Show
          Karthik K added a comment - hadoop-core ivy configuration, conf="common->default" probably needs to be revisited specifically mentioning only the contexts needed for hbase code and not everything else. 'default' is not something we would want. A comma separated , fine-grained dependency is needed.
          Hide
          stack added a comment -

          I applied hdfs-869. Lets see if hudson builds clean and posts new jar to apache repo.

          Show
          stack added a comment - I applied hdfs-869. Lets see if hudson builds clean and posts new jar to apache repo.
          Hide
          Karthik K added a comment -
          I applied hdfs-869. Lets see if hudson builds clean and posts new jar to apache repo.

          Thanks stack. That has been applied successfully as the new jars are available in the repository. mapreduce-0.21 seems to be red ( http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-21-Build/51/#showFailuresLink ) though, failing with the same dll issue seen above (RPC.getProxy() signature ).

          Show
          Karthik K added a comment - I applied hdfs-869. Lets see if hudson builds clean and posts new jar to apache repo. Thanks stack. That has been applied successfully as the new jars are available in the repository. mapreduce-0.21 seems to be red ( http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-21-Build/51/#showFailuresLink ) though, failing with the same dll issue seen above (RPC.getProxy() signature ).
          Hide
          Karthik K added a comment -

          Revised ivy file attached.

          • single place for ivy settings / libraries.properties .

          Zookeeper / thrift jars need to be version controlled. Everything else can be deleted.

          Show
          Karthik K added a comment - Revised ivy file attached. single place for ivy settings / libraries.properties . Zookeeper / thrift jars need to be version controlled. Everything else can be deleted.
          Hide
          Karthik K added a comment -

          So far -after hdfs repositories have been published to apache snapshot repository - ran this patch locally , that completes test without the dynamic link issue with the following caveats.

          • The jars in hadoop-mapred/0.21.0 seem to be corrupted.
          • Checked out branch ( branch-0.21) of mapreduce . manually created the jars (hadoop-mapred and hadoop-mapred-test) and copied them to ~/.ivy2/org.apache.hadoop/hadoop-mapred(-test) to circumvent ivy . Dynamic link issue disappeared (great !!)
          • org.apache.hadoop.hbase.TestInfoServers FAILED (timeout)

          $ cat build/test/TEST-org.apache.hadoop.hbase.TestInfoServers.txt
          Testsuite: org.apache.hadoop.hbase.TestInfoServers
          Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec

          Testcase: testInfoServersAreUp took 0.003 sec
          Caused an ERROR
          Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
          junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.

          But that might be a different issue altogether independent on dependency management that we can handle separately . may be.

          • TODO: Fix MAPREDUCE-1352 with revised checkin to branch and revised publishing of artifacts. That should resolve this issue.
          Show
          Karthik K added a comment - So far -after hdfs repositories have been published to apache snapshot repository - ran this patch locally , that completes test without the dynamic link issue with the following caveats. The jars in hadoop-mapred/0.21.0 seem to be corrupted. Checked out branch ( branch-0.21) of mapreduce . manually created the jars (hadoop-mapred and hadoop-mapred-test) and copied them to ~/.ivy2/org.apache.hadoop/hadoop-mapred(-test) to circumvent ivy . Dynamic link issue disappeared (great !!) org.apache.hadoop.hbase.TestInfoServers FAILED (timeout) $ cat build/test/TEST-org.apache.hadoop.hbase.TestInfoServers.txt Testsuite: org.apache.hadoop.hbase.TestInfoServers Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec Testcase: testInfoServersAreUp took 0.003 sec Caused an ERROR Timeout occurred. Please note the time in the report does not reflect the time until the timeout. junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout. But that might be a different issue altogether independent on dependency management that we can handle separately . may be. TODO: Fix MAPREDUCE-1352 with revised checkin to branch and revised publishing of artifacts. That should resolve this issue.
          Hide
          Karthik K added a comment -

          Remove properties that are redundant.

          Show
          Karthik K added a comment - Remove properties that are redundant.
          Hide
          ryan rawson added a comment -

          One thing you aren't surfacing here is this change will require everyone who wants to build HBase to have internet access on their build machines.

          Show
          ryan rawson added a comment - One thing you aren't surfacing here is this change will require everyone who wants to build HBase to have internet access on their build machines.
          Hide
          Karthik K added a comment -
          One thing you aren't surfacing here is this change will require everyone who wants to build HBase to have internet access on their build machines.

          Nothing theoretically prevents from a build machine from hosting their own internal ivy repository within the firewall and configuring ivysettings.xml appropriately ( as another resolver ). But given the fact that hbase is using no less than 3 snapshots of other open source projects ( + zookeeper , that makes is four ) - I guess the need is to make sure that hbase does not fall back too much from the code of the dependent projects and wait for the single big merge. That will -vely affect the length of release cycles and the confidence of milestone builds in the middle.

          Show
          Karthik K added a comment - One thing you aren't surfacing here is this change will require everyone who wants to build HBase to have internet access on their build machines. Nothing theoretically prevents from a build machine from hosting their own internal ivy repository within the firewall and configuring ivysettings.xml appropriately ( as another resolver ). But given the fact that hbase is using no less than 3 snapshots of other open source projects ( + zookeeper , that makes is four ) - I guess the need is to make sure that hbase does not fall back too much from the code of the dependent projects and wait for the single big merge. That will -vely affect the length of release cycles and the confidence of milestone builds in the middle.
          Hide
          ryan rawson added a comment -

          I just want to make it clear here that Ivy doesn't solve the problems you are listing. Those problems existed before, and were solved before ivy. Ivy solves your problem because you assume/need/depend on a complex dependency management system. For those of us who don't care about ivy in fact lose something - a easy to build hbase.

          Show
          ryan rawson added a comment - I just want to make it clear here that Ivy doesn't solve the problems you are listing. Those problems existed before, and were solved before ivy. Ivy solves your problem because you assume/need/depend on a complex dependency management system. For those of us who don't care about ivy in fact lose something - a easy to build hbase.
          Hide
          Karthik K added a comment -
          I just want to make it clear here that Ivy doesn't solve the problems you are listing. Those problems existed before, and were solved before ivy. Ivy solves your problem because you assume/need/depend on a complex dependency management system. For those of us who don't care about ivy in fact lose something - a easy to build hbase.

          Ivy definitely is not a silver bullet to dependency management issues but it makes a honest attempt towards that. I find the graph visualization / conflict resolution very useful. Agree about hurting slow speed connection users - but you can always set up a proxy dependency server pointing to a main repository to handle it for you, to mitigate the issue.

          Show
          Karthik K added a comment - I just want to make it clear here that Ivy doesn't solve the problems you are listing. Those problems existed before, and were solved before ivy. Ivy solves your problem because you assume/need/depend on a complex dependency management system. For those of us who don't care about ivy in fact lose something - a easy to build hbase. Ivy definitely is not a silver bullet to dependency management issues but it makes a honest attempt towards that. I find the graph visualization / conflict resolution very useful. Agree about hurting slow speed connection users - but you can always set up a proxy dependency server pointing to a main repository to handle it for you, to mitigate the issue.
          Hide
          Andrew Purtell added a comment -

          I agree with Ryan. Why should I have to bother with setting up some kind of ivy proxy (locally?) ? The first thing I do with Hadoop 0.21 is build to pull all the dependencies into build/ivy/.... and then copy them by hand into a lib/ directory so I can put them on the Eclipse build path in a clean manner. Any dependency changes, I have to fix them by hand. Before, SVN add/rm would handle that for me – the committer would deal with the dependency change. I'm not looking forward to needing to do the same thing for HBase.

          My interest here is not to discourage this if people are generally in favor, but, personally, I prefer ant. I like make. I have a good relationship with Eclipse but otherwise IDEs get in the way. Etc. I see no personal benefit for using ivy or maven, just an (unnecessary, IMO) learning curve, additional work if I need to fix something to get it to work for my changes, and a truckload of external dependencies.

          Show
          Andrew Purtell added a comment - I agree with Ryan. Why should I have to bother with setting up some kind of ivy proxy (locally?) ? The first thing I do with Hadoop 0.21 is build to pull all the dependencies into build/ivy/.... and then copy them by hand into a lib/ directory so I can put them on the Eclipse build path in a clean manner. Any dependency changes, I have to fix them by hand. Before, SVN add/rm would handle that for me – the committer would deal with the dependency change. I'm not looking forward to needing to do the same thing for HBase. My interest here is not to discourage this if people are generally in favor, but, personally, I prefer ant. I like make. I have a good relationship with Eclipse but otherwise IDEs get in the way. Etc. I see no personal benefit for using ivy or maven, just an (unnecessary, IMO) learning curve, additional work if I need to fix something to get it to work for my changes, and a truckload of external dependencies.
          Hide
          Karthik K added a comment -

          I agree with Ryan. Why should I have to bother with setting up some kind of ivy proxy (locally?) ? The first thing I do with Hadoop 0.21 is build to pull all the dependencies into build/ivy/.... and then copy them by hand into a lib/ directory so I can put them on the Eclipse build path in a clean manner. Any dependency changes, I have to fix them by hand. Before, SVN add/rm would handle that for me - the committer would deal with the dependency change. I'm not looking forward to needing to do the same thing for HBase.
          My interest here is not to discourage this if people are generally in favor, but, personally, I prefer ant. I like make. I have a good relationship with Eclipse but otherwise IDEs get in the way. Etc. I see no personal benefit for using ivy or maven, just an (unnecessary, IMO) learning curve, additional work if I need to fix something to get it to work for my changes, and a truckload of external dependencies.

          I find this thread interesting given that every other project in the eco-system - hadoop-core / mapreduce / hdfs / mahout ( uses maven) / zookeeper (3.3.0+) go with a dependency manager in one form or the other. It does not conflict with Eclipse in any way since instead of lib/** - people will be required to add build/ivy/common/** to the classpath . SVN add/rm handling dependency management is definitely possible, but say in the case of lucene - there are at least 6 different jars / contribs that are available and if we want to flip between versions- adding and removing versions from source control is not fun at all. And with ivy - there is no need to install any piece of software other than ant itself. Version controlling snapshots of projects will make it hard to keep up with their updates as and when they become available and goes against the very objective of continuous build process. With this patch - people would be modifying ivy.xml / libraries.properties at max , without worrying about anything else and I find that way comfortable , compared to downloading yet another installation and version controlling them (and deleting old versions), not to mention deciding dependency management winners.

          Show
          Karthik K added a comment - I agree with Ryan. Why should I have to bother with setting up some kind of ivy proxy (locally?) ? The first thing I do with Hadoop 0.21 is build to pull all the dependencies into build/ivy/.... and then copy them by hand into a lib/ directory so I can put them on the Eclipse build path in a clean manner. Any dependency changes, I have to fix them by hand. Before, SVN add/rm would handle that for me - the committer would deal with the dependency change. I'm not looking forward to needing to do the same thing for HBase. My interest here is not to discourage this if people are generally in favor, but, personally, I prefer ant. I like make. I have a good relationship with Eclipse but otherwise IDEs get in the way. Etc. I see no personal benefit for using ivy or maven, just an (unnecessary, IMO) learning curve, additional work if I need to fix something to get it to work for my changes, and a truckload of external dependencies. I find this thread interesting given that every other project in the eco-system - hadoop-core / mapreduce / hdfs / mahout ( uses maven) / zookeeper (3.3.0+) go with a dependency manager in one form or the other. It does not conflict with Eclipse in any way since instead of lib/** - people will be required to add build/ivy/common/** to the classpath . SVN add/rm handling dependency management is definitely possible, but say in the case of lucene - there are at least 6 different jars / contribs that are available and if we want to flip between versions- adding and removing versions from source control is not fun at all. And with ivy - there is no need to install any piece of software other than ant itself. Version controlling snapshots of projects will make it hard to keep up with their updates as and when they become available and goes against the very objective of continuous build process. With this patch - people would be modifying ivy.xml / libraries.properties at max , without worrying about anything else and I find that way comfortable , compared to downloading yet another installation and version controlling them (and deleting old versions), not to mention deciding dependency management winners.
          Hide
          Karthik K added a comment -

          And to state the obvious, both ivy/maven caches the jar files locally , to be nice to slow connection scenarios. (happens transparently , without the need for any config.) .

          Show
          Karthik K added a comment - And to state the obvious, both ivy/maven caches the jar files locally , to be nice to slow connection scenarios. (happens transparently , without the need for any config.) .
          Hide
          stack added a comment -

          (Redo of a mangled iphone comment of mine that I just deleted)

          On having to be connected to the net, it should be the case that if the jar is not in the local ivy cache, then the build should not go to the net. If this is not the case, we should fix it. This should do away with having to set up local repo. The only difficulty that I see will be not going to the net to check for new SNAPSHOTs (Kay Kay says this above).

          In deleted comment, I made the silly remark that build may not be more complex, just different. Thinking on it, there'll be more moving parts and more to learn, etc. as Andy says above.

          For me, main reasons we're doing this change is to keep alignment the parent project. Ivy also helps with maven publishing, another means of making hbase available especially for folks embedding (we could write ant tasks ourselves to do this but if we use ivy we can just rob the tested parent projects ant tasks and bring them local).

          Show
          stack added a comment - (Redo of a mangled iphone comment of mine that I just deleted) On having to be connected to the net, it should be the case that if the jar is not in the local ivy cache, then the build should not go to the net. If this is not the case, we should fix it. This should do away with having to set up local repo. The only difficulty that I see will be not going to the net to check for new SNAPSHOTs (Kay Kay says this above). In deleted comment, I made the silly remark that build may not be more complex, just different. Thinking on it, there'll be more moving parts and more to learn, etc. as Andy says above. For me, main reasons we're doing this change is to keep alignment the parent project. Ivy also helps with maven publishing, another means of making hbase available especially for folks embedding (we could write ant tasks ourselves to do this but if we use ivy we can just rob the tested parent projects ant tasks and bring them local).
          Hide
          stack added a comment -

          @kay kay mapreduce-1352 looks likes its committed and resolved but 0.21 build of mapreduce up on hudson looks badly broke. On TestInfoServers, this could be an ordering of jars issue. HBase tries to make use of the hadoop webapps. Have to be careful about which jar comes first (hadoop before hbase IIRC).

          Show
          stack added a comment - @kay kay mapreduce-1352 looks likes its committed and resolved but 0.21 build of mapreduce up on hudson looks badly broke. On TestInfoServers, this could be an ordering of jars issue. HBase tries to make use of the hadoop webapps. Have to be careful about which jar comes first (hadoop before hbase IIRC).
          Hide
          Karthik K added a comment -

          yes - MAPREDUCE-1352 has been committed and ran 'ant test'. The dynamic link issue does not appear and all tests run successfully except for TestInfoServers.

          (As far as the branch-0.21 of mapreduce being broke - it appears to be an environment issue on the build part at the moment , since it does not seem to have hadoop-core-0.21.0 in its CP ).

          So - that leaves with 1 open issue - TestInfoServers , CP issue. I will look into this when I get time.

          Show
          Karthik K added a comment - yes - MAPREDUCE-1352 has been committed and ran 'ant test'. The dynamic link issue does not appear and all tests run successfully except for TestInfoServers. (As far as the branch-0.21 of mapreduce being broke - it appears to be an environment issue on the build part at the moment , since it does not seem to have hadoop-core-0.21.0 in its CP ). So - that leaves with 1 open issue - TestInfoServers , CP issue. I will look into this when I get time.
          Hide
          Karthik K added a comment -

          added jruby-complete .

          rearranged classpath to have hadoop ahead of hbase .

          Show
          Karthik K added a comment - added jruby-complete . rearranged classpath to have hadoop ahead of hbase .
          Hide
          Karthik K added a comment -

          attached a revised patch . Error still persists .

          Test log has no significant information.. Where can we get more debugging information ( web context load issue / classpath issue , say ? ).

          Show
          Karthik K added a comment - attached a revised patch . Error still persists . Test log has no significant information.. Where can we get more debugging information ( web context load issue / classpath issue , say ? ).
          Hide
          stack added a comment -

          If its what I think it is, complaint is that it can't find a wanted webapp. Can you print out CLASSPATH used? Can you make it same as that used running tests in pre-patch days?

          Show
          stack added a comment - If its what I think it is, complaint is that it can't find a wanted webapp. Can you print out CLASSPATH used? Can you make it same as that used running tests in pre-patch days?
          Hide
          stack added a comment -

          One other thing, if there is an instance of hbase running at same time as you are running tests, that test will fail.

          Show
          stack added a comment - One other thing, if there is an instance of hbase running at same time as you are running tests, that test will fail.
          Hide
          Karthik K added a comment -

          If its what I think it is, complaint is that it can't find a wanted webapp. Can you print out CLASSPATH used? Can you make it same as that used running tests in pre-patch days?

          Having src/test : build/test ( in that order ) seems to fix the issue.

          One other thing, if there is an instance of hbase running at same time as you are running tests, that test will fail.

          Did notice this (when I tried to run tests in parallel on contrib/stargate , contrib/transactional etc.

          So - I have a revised patch that works with TestInfoServers as well.

          TODO:

          • Check tests of contrib/stargate and
          • contrib/transactional .
          Show
          Karthik K added a comment - If its what I think it is, complaint is that it can't find a wanted webapp. Can you print out CLASSPATH used? Can you make it same as that used running tests in pre-patch days? Having src/test : build/test ( in that order ) seems to fix the issue. One other thing, if there is an instance of hbase running at same time as you are running tests, that test will fail. Did notice this (when I tried to run tests in parallel on contrib/stargate , contrib/transactional etc. So - I have a revised patch that works with TestInfoServers as well. TODO: Check tests of contrib/stargate and contrib/transactional .
          Hide
          Karthik K added a comment -

          So far -

          contrib/transactional:

          [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
          [junit] Test org.apache.hadoop.hbase.client.transactional.TestTransactions FAILED (timeout)

          [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
          [junit] Test org.apache.hadoop.hbase.regionserver.transactional.TestTHLogRecovery FAILED (timeout)

          Show
          Karthik K added a comment - So far - contrib/transactional: [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec [junit] Test org.apache.hadoop.hbase.client.transactional.TestTransactions FAILED (timeout) [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec [junit] Test org.apache.hadoop.hbase.regionserver.transactional.TestTHLogRecovery FAILED (timeout)
          Hide
          Karthik K added a comment -

          'ant compile' and 'ant test' succeeds , eventually .

          [ Set transitive="false" for all test packages to not pollute the classpath by the presence of hadoop-core* / -hdfs* / -mapreduce* jars twice ( in lib/common and lib/test ). ]

          Need to check other targets for compatibility issues. Ivy - here we come.

          Show
          Karthik K added a comment - 'ant compile' and 'ant test' succeeds , eventually . [ Set transitive="false" for all test packages to not pollute the classpath by the presence of hadoop-core* / -hdfs* / -mapreduce* jars twice ( in lib/common and lib/test ). ] Need to check other targets for compatibility issues. Ivy - here we come.
          Hide
          Karthik K added a comment -

          To make thrift publish their artifacts to repositories.

          Show
          Karthik K added a comment - To make thrift publish their artifacts to repositories.
          Hide
          Karthik K added a comment -

          @stack - The revised patch compiles / tests successfully. Can you help review this and may be try this ( on a hudson build ? ) to see how this goes.

          Show
          Karthik K added a comment - @stack - The revised patch compiles / tests successfully. Can you help review this and may be try this ( on a hudson build ? ) to see how this goes.
          Hide
          stack added a comment -

          @Kay Kay Good on you for figuring the ugly issue. We haven't done the work to setup the hadoop patch-builds up on hudson. For hbase, only way to get an hudson build is to do an actual commit. If all tests pass, then I should commit only its a pretty big change so I just sent a heads-up to the hbase-dev list warning of pending commit. I also asked for commentary in case folks object to the move to ivy.

          Show
          stack added a comment - @Kay Kay Good on you for figuring the ugly issue. We haven't done the work to setup the hadoop patch-builds up on hudson. For hbase, only way to get an hudson build is to do an actual commit. If all tests pass, then I should commit only its a pretty big change so I just sent a heads-up to the hbase-dev list warning of pending commit. I also asked for commentary in case folks object to the move to ivy.
          Hide
          Karthik K added a comment -
          • build-contrib - junit target - if /unless "testcase" retained as before, using $ {testcase}

            macro

          Show
          Karthik K added a comment - build-contrib - junit target - if /unless "testcase" retained as before, using $ {testcase} macro
          Hide
          stack added a comment -

          committed after making sure build passes locally. had to clear my ivy cache but then was fine. thanks for nice fat patch kay kay

          Show
          stack added a comment - committed after making sure build passes locally. had to clear my ivy cache but then was fine. thanks for nice fat patch kay kay
          Hide
          Karthik K added a comment -

          Thanks stack for helping reviewing this one and getting this one in.

          I have put up a primer here , for transition - http://wiki.apache.org/hadoop/Hbase/IvyPrimer . Needs better formatting- but can serve as an initial point to resolve mystery.

          Show
          Karthik K added a comment - Thanks stack for helping reviewing this one and getting this one in. I have put up a primer here , for transition - http://wiki.apache.org/hadoop/Hbase/IvyPrimer . Needs better formatting- but can serve as an initial point to resolve mystery.

            People

            • Assignee:
              Unassigned
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development