HBase
  1. HBase
  2. HBASE-2233

Support both Hadoop 0.20, 0.21, and 0.22

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.92.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      HBase will run unadorned on hadoop 0.22 (HBase does not run on hadoop 0.21, not without forward port of recoverLease, HDFS-1948, and change to httpserver, HADOOP-7351)
      Show
      HBase will run unadorned on hadoop 0.22 (HBase does not run on hadoop 0.21, not without forward port of recoverLease, HDFS-1948 , and change to httpserver, HADOOP-7351 )

      Description

      Since Hadoop 0.21 isn't going to be well supported and that a lot of users may wish to stick on 0.20, the next HBase major release should support both 0.20 and 0.21.

      HDFS-265 support will be swapped for HDFS-200 if running on HDFS 0.20. A cluster without that patchset shouldn't be supported.

      1. 2233.txt
        7 kB
        stack
      2. pom.to.go.against.hadoop.trunk.txt
        4 kB
        stack
      3. 2233-v2.txt
        29 kB
        stack
      4. 2233-v3.txt
        35 kB
        stack
      5. 2233-v4.txt
        42 kB
        stack
      6. 2233-v5.txt
        44 kB
        stack

        Activity

        Hide
        Karthik K added a comment -

        Unfortunate, but do not have an option.

        Show
        Karthik K added a comment - Unfortunate, but do not have an option.
        Hide
        Karthik K added a comment -

        so - at its current stage of HBase trunk - what are the patches suggested/ advisable for hdfs - 0.21 ?

        what would be the similar patches ( if applicable) for hadoop 0.20.x as well ?

        Show
        Karthik K added a comment - so - at its current stage of HBase trunk - what are the patches suggested/ advisable for hdfs - 0.21 ? what would be the similar patches ( if applicable) for hadoop 0.20.x as well ?
        Hide
        Jean-Daniel Cryans added a comment -

        so - at its current stage of HBase trunk - what are the patches suggested/ advisable for hdfs - 0.21 ?

        None that I know of.

        what would be the similar patches ( if applicable) for hadoop 0.20.x as well ?

        It's not all done yet but there's HDFS-200, HDFS-142 and one other that I don't remember. Dhruba said he will tag them "hbase".

        Show
        Jean-Daniel Cryans added a comment - so - at its current stage of HBase trunk - what are the patches suggested/ advisable for hdfs - 0.21 ? None that I know of. what would be the similar patches ( if applicable) for hadoop 0.20.x as well ? It's not all done yet but there's HDFS-200 , HDFS-142 and one other that I don't remember. Dhruba said he will tag them "hbase".
        Hide
        Jean-Daniel Cryans added a comment -

        0.21.0 was released today, we need to decide if we support it in the scope of 0.90.

        Show
        Jean-Daniel Cryans added a comment - 0.21.0 was released today, we need to decide if we support it in the scope of 0.90.
        Hide
        Jonathan Gray added a comment -

        What's plan for this?

        Show
        Jonathan Gray added a comment - What's plan for this?
        Hide
        stack added a comment -

        Moving out

        Show
        stack added a comment - Moving out
        Hide
        stack added a comment -

        This is what I did at the 0.22 hackathon today... There is more to do bringing us over to new append. Asking Hairong about recovery of lease which is not in 0.22, if it makes sense to add it. This code compiles but I need to check tests.

        It looks for hflush in the dfsclient outputstream. If present, we'll call it on each sync invocation (we are not asking SequenceFile to do it as 0.20 does since there is no 'hflush' in SequenceFile)

        Show
        stack added a comment - This is what I did at the 0.22 hackathon today... There is more to do bringing us over to new append. Asking Hairong about recovery of lease which is not in 0.22, if it makes sense to add it. This code compiles but I need to check tests. It looks for hflush in the dfsclient outputstream. If present, we'll call it on each sync invocation (we are not asking SequenceFile to do it as 0.20 does since there is no 'hflush' in SequenceFile)
        Hide
        stack added a comment -

        Hack on pom so it works with hadoop trunk.

        Show
        stack added a comment - Hack on pom so it works with hadoop trunk.
        Hide
        stack added a comment -

        Here is something that actually compiles against 0.22. Doesn't work though yet. I'm working through the unit tests. Has hackups to get around changes in hadoop API including switch to use hadoop's TOP if suitable one is available and removal from RPC of user passing since its Type changed across Hadoop versions (and Gary's security work does this stuff for us – no need to do it here in plain-hbase).

        Show
        stack added a comment - Here is something that actually compiles against 0.22. Doesn't work though yet. I'm working through the unit tests. Has hackups to get around changes in hadoop API including switch to use hadoop's TOP if suitable one is available and removal from RPC of user passing since its Type changed across Hadoop versions (and Gary's security work does this stuff for us – no need to do it here in plain-hbase).
        Hide
        Todd Lipcon added a comment -

        Do any of the 0.22 incompatibilities seem like things that are Hadoop's fault? Or all cases where we've used somewhat internal APIs like UGI?

        Show
        Todd Lipcon added a comment - Do any of the 0.22 incompatibilities seem like things that are Hadoop's fault? Or all cases where we've used somewhat internal APIs like UGI?
        Hide
        stack added a comment -

        + UGI in RPC ConnectionHeader and in some test code.
        + The hackary around the bringing of TotalOrderPartitioner local under hadoopbackport subpackage. In particular the move of TaskAttemptContext from Class to Interface.
        + The HttpServer changes the signature on one of its add webapp methods so you now need to pass a Configuration
        + Some mess where we gave ourselves access to NameNode internals so we could set leases; add hackery to expose something that once was exposed.

        Considering we're jumping two versions, not too bad.

        Still may be more and I need to go over the patch again before its final.

        Show
        stack added a comment - + UGI in RPC ConnectionHeader and in some test code. + The hackary around the bringing of TotalOrderPartitioner local under hadoopbackport subpackage. In particular the move of TaskAttemptContext from Class to Interface. + The HttpServer changes the signature on one of its add webapp methods so you now need to pass a Configuration + Some mess where we gave ourselves access to NameNode internals so we could set leases; add hackery to expose something that once was exposed. Considering we're jumping two versions, not too bad. Still may be more and I need to go over the patch again before its final.
        Hide
        stack added a comment -

        Hmm... some fs behaviors have changed. You can't do a listing of a non-existent directory anymore. Now it throws FileNotFoundException where before it just soldiered on.

        Show
        stack added a comment - Hmm... some fs behaviors have changed. You can't do a listing of a non-existent directory anymore. Now it throws FileNotFoundException where before it just soldiered on.
        Hide
        stack added a comment -

        This patch gets pretty far in the tests. Still running. Need to make sure works w/ 0.20 hadoop next.

        Show
        stack added a comment - This patch gets pretty far in the tests. Still running. Need to make sure works w/ 0.20 hadoop next.
        Hide
        stack added a comment -

        Just TestReplication and TestHFileOutputFormat failing now; issues running mapreduce.

        Show
        stack added a comment - Just TestReplication and TestHFileOutputFormat failing now; issues running mapreduce.
        Hide
        stack added a comment -

        All tests pass now building against 0.22 hadoop but with one kink. TestHFileOutputFormat fails by hanging BECAUSE our avro is 1.4.1 but the avro in hadoop 0.22 is 1.3.2; when this test runs IF YOU ENABLE MR DEBUG you'll see NoSuchMethodException thrown (Thanks to Todd for help in here). I'm tempted to cut avro from hbase core.

        I am now testing hbase up on cluster.

        Still to do is verify this patch passes all tests with hadoop 0.20 under it.

        Show
        stack added a comment - All tests pass now building against 0.22 hadoop but with one kink. TestHFileOutputFormat fails by hanging BECAUSE our avro is 1.4.1 but the avro in hadoop 0.22 is 1.3.2; when this test runs IF YOU ENABLE MR DEBUG you'll see NoSuchMethodException thrown (Thanks to Todd for help in here). I'm tempted to cut avro from hbase core. I am now testing hbase up on cluster. Still to do is verify this patch passes all tests with hadoop 0.20 under it.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/858/
        -----------------------------------------------------------

        Review request for hbase.

        Summary
        -------

        Patch for HBASE TRUNK that makes it so we run on hadoop 0.20.x or hadoop 0.22 branch (as it is now).

        Needed hdfs-1948 and hadoop-7531 applied to hadoop 0.22 branch. Because these issues are NOT in hadoop 0.21, hbase will not work against 0.21 hadoop, not w/o these patches and a rebuild.

        This addresses bug HBASE-2233.
        https://issues.apache.org/jira/browse/HBASE-2233

        Diffs


        src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java c2a7649
        src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 95407d1
        src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/InputSampler.java c3e0a68
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac
        src/main/java/org/apache/hadoop/hbase/master/ServerManager.java f735e41
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6404538
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java db29e56
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java 8dc9a5e
        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108
        src/main/java/org/apache/hadoop/hbase/util/InfoServer.java 6ed9fe6
        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788
        src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java f6a7210
        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java f1f4c6b
        src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 5b487a9
        src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java c8fc065

        Diff: https://reviews.apache.org/r/858/diff

        Testing
        -------

        Ran this patch with hadoop 0.20 and hadoop 0.22 underneath hbase (I used the patch for the pom that is attached to the issue to switch to hadoop 0.22). All unit tests seem to pass except TestHFileOutputFormat when up on hadoop 0.22; hadoop 0.22 uses avro 1.3.2 whereas hbase is up on avro 1.4.1. The test fails with a method not found exception. You need to enable mapreduce logging to see the exception.

        Thanks,

        Michael

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/858/ ----------------------------------------------------------- Review request for hbase. Summary ------- Patch for HBASE TRUNK that makes it so we run on hadoop 0.20.x or hadoop 0.22 branch (as it is now). Needed hdfs-1948 and hadoop-7531 applied to hadoop 0.22 branch. Because these issues are NOT in hadoop 0.21, hbase will not work against 0.21 hadoop, not w/o these patches and a rebuild. This addresses bug HBASE-2233 . https://issues.apache.org/jira/browse/HBASE-2233 Diffs src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java c2a7649 src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 95407d1 src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/InputSampler.java c3e0a68 src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac src/main/java/org/apache/hadoop/hbase/master/ServerManager.java f735e41 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6404538 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java db29e56 src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java 8dc9a5e src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108 src/main/java/org/apache/hadoop/hbase/util/InfoServer.java 6ed9fe6 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788 src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java f6a7210 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java f1f4c6b src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 5b487a9 src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java c8fc065 Diff: https://reviews.apache.org/r/858/diff Testing ------- Ran this patch with hadoop 0.20 and hadoop 0.22 underneath hbase (I used the patch for the pom that is attached to the issue to switch to hadoop 0.22). All unit tests seem to pass except TestHFileOutputFormat when up on hadoop 0.22; hadoop 0.22 uses avro 1.3.2 whereas hbase is up on avro 1.4.1. The test fails with a method not found exception. You need to enable mapreduce logging to see the exception. Thanks, Michael
        Hide
        stack added a comment -

        Looking for a bit of a review before I commit (Patch is not too big – just a little messy)

        Show
        stack added a comment - Looking for a bit of a review before I commit (Patch is not too big – just a little messy)
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/858/#review769
        -----------------------------------------------------------

        nice work just some small comments.

        src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
        <https://reviews.apache.org/r/858/#comment1651>

        is this change related?

        src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
        <https://reviews.apache.org/r/858/#comment1652>

        also this

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
        <https://reviews.apache.org/r/858/#comment1653>

        hrm, I dont understand this line

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
        <https://reviews.apache.org/r/858/#comment1654>

        did our behavior here change? I think we used to return null, but I thought now we returned an empty list?

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
        <https://reviews.apache.org/r/858/#comment1655>

        same q here

        src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java
        <https://reviews.apache.org/r/858/#comment1656>

        CONFI?

        • Todd

        On 2011-06-06 22:27:56, Michael Stack wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/858/

        -----------------------------------------------------------

        (Updated 2011-06-06 22:27:56)

        Review request for hbase.

        Summary

        -------

        Patch for HBASE TRUNK that makes it so we run on hadoop 0.20.x or hadoop 0.22 branch (as it is now).

        Needed hdfs-1948 and hadoop-7531 applied to hadoop 0.22 branch. Because these issues are NOT in hadoop 0.21, hbase will not work against 0.21 hadoop, not w/o these patches and a rebuild.

        This addresses bug HBASE-2233.

        https://issues.apache.org/jira/browse/HBASE-2233

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java c2a7649

        src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 95407d1

        src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/InputSampler.java c3e0a68

        src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac

        src/main/java/org/apache/hadoop/hbase/master/ServerManager.java f735e41

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6404538

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java db29e56

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java 8dc9a5e

        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108

        src/main/java/org/apache/hadoop/hbase/util/InfoServer.java 6ed9fe6

        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788

        src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java f6a7210

        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java f1f4c6b

        src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 5b487a9

        src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java c8fc065

        Diff: https://reviews.apache.org/r/858/diff

        Testing

        -------

        Ran this patch with hadoop 0.20 and hadoop 0.22 underneath hbase (I used the patch for the pom that is attached to the issue to switch to hadoop 0.22). All unit tests seem to pass except TestHFileOutputFormat when up on hadoop 0.22; hadoop 0.22 uses avro 1.3.2 whereas hbase is up on avro 1.4.1. The test fails with a method not found exception. You need to enable mapreduce logging to see the exception.

        Thanks,

        Michael

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/858/#review769 ----------------------------------------------------------- nice work just some small comments. src/main/java/org/apache/hadoop/hbase/master/ServerManager.java < https://reviews.apache.org/r/858/#comment1651 > is this change related? src/main/java/org/apache/hadoop/hbase/master/ServerManager.java < https://reviews.apache.org/r/858/#comment1652 > also this src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java < https://reviews.apache.org/r/858/#comment1653 > hrm, I dont understand this line src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java < https://reviews.apache.org/r/858/#comment1654 > did our behavior here change? I think we used to return null, but I thought now we returned an empty list? src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java < https://reviews.apache.org/r/858/#comment1655 > same q here src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java < https://reviews.apache.org/r/858/#comment1656 > CONFI? Todd On 2011-06-06 22:27:56, Michael Stack wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/858/ ----------------------------------------------------------- (Updated 2011-06-06 22:27:56) Review request for hbase. Summary ------- Patch for HBASE TRUNK that makes it so we run on hadoop 0.20.x or hadoop 0.22 branch (as it is now). Needed hdfs-1948 and hadoop-7531 applied to hadoop 0.22 branch. Because these issues are NOT in hadoop 0.21, hbase will not work against 0.21 hadoop, not w/o these patches and a rebuild. This addresses bug HBASE-2233 . https://issues.apache.org/jira/browse/HBASE-2233 Diffs ----- src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java c2a7649 src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 95407d1 src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/InputSampler.java c3e0a68 src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac src/main/java/org/apache/hadoop/hbase/master/ServerManager.java f735e41 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6404538 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java db29e56 src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java 8dc9a5e src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108 src/main/java/org/apache/hadoop/hbase/util/InfoServer.java 6ed9fe6 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788 src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java f6a7210 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java f1f4c6b src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 5b487a9 src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java c8fc065 Diff: https://reviews.apache.org/r/858/diff Testing ------- Ran this patch with hadoop 0.20 and hadoop 0.22 underneath hbase (I used the patch for the pom that is attached to the issue to switch to hadoop 0.22). All unit tests seem to pass except TestHFileOutputFormat when up on hadoop 0.22; hadoop 0.22 uses avro 1.3.2 whereas hbase is up on avro 1.4.1. The test fails with a method not found exception. You need to enable mapreduce logging to see the exception. Thanks, Michael
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-06-06 22:51:07, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/master/ServerManager.java, line 360

        > <https://reviews.apache.org/r/858/diff/1/?file=20611#file20611line360>

        >

        > is this change related?

        Not related but needed. I could do in separate issue. Saw ugly NPE when stuff came up ugly when not patched properly.

        On 2011-06-06 22:51:07, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java, line 791

        > <https://reviews.apache.org/r/858/diff/1/?file=20613#file20613line791>

        >

        > hrm, I dont understand this line

        Good one. Test should be inverted. Fixing...

        On 2011-06-06 22:51:07, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java, line 1343

        > <https://reviews.apache.org/r/858/diff/1/?file=20613#file20613line1343>

        >

        > did our behavior here change? I think we used to return null, but I thought now we returned an empty list?

        fs.listStatus changed in that if you pass it a dir that does not exist, before it would return empty list but no throws FileNotFoundException.

        Regards our behavior changing around the return out of this method, I don't think so. I looked at 0.90 and it does this too:

        if (files == null) return filesSorted;

        ... returning empty NavigableSet if nothing found.

        On 2011-06-06 22:51:07, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java, line 127

        > <https://reviews.apache.org/r/858/diff/1/?file=20621#file20621line127>

        >

        > CONFI?

        Yeah. Takes variable named 'conf1'. I changed it to be less surprising CONF_WITH_LOCALFS

        • Michael

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/858/#review769
        -----------------------------------------------------------

        On 2011-06-06 22:27:56, Michael Stack wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/858/

        -----------------------------------------------------------

        (Updated 2011-06-06 22:27:56)

        Review request for hbase.

        Summary

        -------

        Patch for HBASE TRUNK that makes it so we run on hadoop 0.20.x or hadoop 0.22 branch (as it is now).

        Needed hdfs-1948 and hadoop-7531 applied to hadoop 0.22 branch. Because these issues are NOT in hadoop 0.21, hbase will not work against 0.21 hadoop, not w/o these patches and a rebuild.

        This addresses bug HBASE-2233.

        https://issues.apache.org/jira/browse/HBASE-2233

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java c2a7649

        src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 95407d1

        src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/InputSampler.java c3e0a68

        src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac

        src/main/java/org/apache/hadoop/hbase/master/ServerManager.java f735e41

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6404538

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java db29e56

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java 8dc9a5e

        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108

        src/main/java/org/apache/hadoop/hbase/util/InfoServer.java 6ed9fe6

        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788

        src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java f6a7210

        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java f1f4c6b

        src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 5b487a9

        src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java c8fc065

        Diff: https://reviews.apache.org/r/858/diff

        Testing

        -------

        Ran this patch with hadoop 0.20 and hadoop 0.22 underneath hbase (I used the patch for the pom that is attached to the issue to switch to hadoop 0.22). All unit tests seem to pass except TestHFileOutputFormat when up on hadoop 0.22; hadoop 0.22 uses avro 1.3.2 whereas hbase is up on avro 1.4.1. The test fails with a method not found exception. You need to enable mapreduce logging to see the exception.

        Thanks,

        Michael

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-06-06 22:51:07, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/master/ServerManager.java, line 360 > < https://reviews.apache.org/r/858/diff/1/?file=20611#file20611line360 > > > is this change related? Not related but needed. I could do in separate issue. Saw ugly NPE when stuff came up ugly when not patched properly. On 2011-06-06 22:51:07, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java, line 791 > < https://reviews.apache.org/r/858/diff/1/?file=20613#file20613line791 > > > hrm, I dont understand this line Good one. Test should be inverted. Fixing... On 2011-06-06 22:51:07, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java, line 1343 > < https://reviews.apache.org/r/858/diff/1/?file=20613#file20613line1343 > > > did our behavior here change? I think we used to return null, but I thought now we returned an empty list? fs.listStatus changed in that if you pass it a dir that does not exist, before it would return empty list but no throws FileNotFoundException. Regards our behavior changing around the return out of this method, I don't think so. I looked at 0.90 and it does this too: if (files == null) return filesSorted; ... returning empty NavigableSet if nothing found. On 2011-06-06 22:51:07, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java, line 127 > < https://reviews.apache.org/r/858/diff/1/?file=20621#file20621line127 > > > CONFI? Yeah. Takes variable named 'conf1'. I changed it to be less surprising CONF_WITH_LOCALFS Michael ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/858/#review769 ----------------------------------------------------------- On 2011-06-06 22:27:56, Michael Stack wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/858/ ----------------------------------------------------------- (Updated 2011-06-06 22:27:56) Review request for hbase. Summary ------- Patch for HBASE TRUNK that makes it so we run on hadoop 0.20.x or hadoop 0.22 branch (as it is now). Needed hdfs-1948 and hadoop-7531 applied to hadoop 0.22 branch. Because these issues are NOT in hadoop 0.21, hbase will not work against 0.21 hadoop, not w/o these patches and a rebuild. This addresses bug HBASE-2233 . https://issues.apache.org/jira/browse/HBASE-2233 Diffs ----- src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java c2a7649 src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 95407d1 src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/InputSampler.java c3e0a68 src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac src/main/java/org/apache/hadoop/hbase/master/ServerManager.java f735e41 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6404538 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java db29e56 src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java 8dc9a5e src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108 src/main/java/org/apache/hadoop/hbase/util/InfoServer.java 6ed9fe6 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788 src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java f6a7210 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java f1f4c6b src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 5b487a9 src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java c8fc065 Diff: https://reviews.apache.org/r/858/diff Testing ------- Ran this patch with hadoop 0.20 and hadoop 0.22 underneath hbase (I used the patch for the pom that is attached to the issue to switch to hadoop 0.22). All unit tests seem to pass except TestHFileOutputFormat when up on hadoop 0.22; hadoop 0.22 uses avro 1.3.2 whereas hbase is up on avro 1.4.1. The test fails with a method not found exception. You need to enable mapreduce logging to see the exception. Thanks, Michael
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/858/#review776
        -----------------------------------------------------------

        Ship it!

        sounds good, +1 after fixing that condition

        • Todd

        On 2011-06-06 22:27:56, Michael Stack wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/858/

        -----------------------------------------------------------

        (Updated 2011-06-06 22:27:56)

        Review request for hbase.

        Summary

        -------

        Patch for HBASE TRUNK that makes it so we run on hadoop 0.20.x or hadoop 0.22 branch (as it is now).

        Needed hdfs-1948 and hadoop-7531 applied to hadoop 0.22 branch. Because these issues are NOT in hadoop 0.21, hbase will not work against 0.21 hadoop, not w/o these patches and a rebuild.

        This addresses bug HBASE-2233.

        https://issues.apache.org/jira/browse/HBASE-2233

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java c2a7649

        src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 95407d1

        src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/InputSampler.java c3e0a68

        src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac

        src/main/java/org/apache/hadoop/hbase/master/ServerManager.java f735e41

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6404538

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java db29e56

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java 8dc9a5e

        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108

        src/main/java/org/apache/hadoop/hbase/util/InfoServer.java 6ed9fe6

        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788

        src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java f6a7210

        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java f1f4c6b

        src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 5b487a9

        src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java c8fc065

        Diff: https://reviews.apache.org/r/858/diff

        Testing

        -------

        Ran this patch with hadoop 0.20 and hadoop 0.22 underneath hbase (I used the patch for the pom that is attached to the issue to switch to hadoop 0.22). All unit tests seem to pass except TestHFileOutputFormat when up on hadoop 0.22; hadoop 0.22 uses avro 1.3.2 whereas hbase is up on avro 1.4.1. The test fails with a method not found exception. You need to enable mapreduce logging to see the exception.

        Thanks,

        Michael

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/858/#review776 ----------------------------------------------------------- Ship it! sounds good, +1 after fixing that condition Todd On 2011-06-06 22:27:56, Michael Stack wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/858/ ----------------------------------------------------------- (Updated 2011-06-06 22:27:56) Review request for hbase. Summary ------- Patch for HBASE TRUNK that makes it so we run on hadoop 0.20.x or hadoop 0.22 branch (as it is now). Needed hdfs-1948 and hadoop-7531 applied to hadoop 0.22 branch. Because these issues are NOT in hadoop 0.21, hbase will not work against 0.21 hadoop, not w/o these patches and a rebuild. This addresses bug HBASE-2233 . https://issues.apache.org/jira/browse/HBASE-2233 Diffs ----- src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java c2a7649 src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 95407d1 src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/InputSampler.java c3e0a68 src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac src/main/java/org/apache/hadoop/hbase/master/ServerManager.java f735e41 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6404538 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java db29e56 src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java 8dc9a5e src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108 src/main/java/org/apache/hadoop/hbase/util/InfoServer.java 6ed9fe6 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788 src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java f6a7210 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java f1f4c6b src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 5b487a9 src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java c8fc065 Diff: https://reviews.apache.org/r/858/diff Testing ------- Ran this patch with hadoop 0.20 and hadoop 0.22 underneath hbase (I used the patch for the pom that is attached to the issue to switch to hadoop 0.22). All unit tests seem to pass except TestHFileOutputFormat when up on hadoop 0.22; hadoop 0.22 uses avro 1.3.2 whereas hbase is up on avro 1.4.1. The test fails with a method not found exception. You need to enable mapreduce logging to see the exception. Thanks, Michael
        Hide
        stack added a comment -

        OK. Did a bit more testing first. There may be some tests that need addressing if hadoop 0.22 is under hbase. I think I got them all. TestHSplitLog was the last one I was working on; it seemed to have failing unit tests but coming back to it, they pass now (TestHFileOutputFormat does not pass, not w/o reverting the avro version to that of hadoop 0.22). Will create new issues for tests that fail against hadoop 0.22. Committed to TRUNK

        Thanks for review Todd.

        As said above we do not work against hadoop 0.21, not w/o backport of "HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461" and "HDFS-1948 Forward port 'hdfs-1520 lightweight namenode operation to trigger lease recovery'"

        Show
        stack added a comment - OK. Did a bit more testing first. There may be some tests that need addressing if hadoop 0.22 is under hbase. I think I got them all. TestHSplitLog was the last one I was working on; it seemed to have failing unit tests but coming back to it, they pass now (TestHFileOutputFormat does not pass, not w/o reverting the avro version to that of hadoop 0.22). Will create new issues for tests that fail against hadoop 0.22. Committed to TRUNK Thanks for review Todd. As said above we do not work against hadoop 0.21, not w/o backport of " HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461 " and " HDFS-1948 Forward port 'hdfs-1520 lightweight namenode operation to trigger lease recovery'"
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #1976 (See https://builds.apache.org/job/HBase-TRUNK/1976/)

        Show
        Hudson added a comment - Integrated in HBase-TRUNK #1976 (See https://builds.apache.org/job/HBase-TRUNK/1976/ )

          People

          • Assignee:
            stack
            Reporter:
            Jean-Daniel Cryans
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development