HBase
  1. HBase
  2. HBASE-4797

[availability] Skip recovered.edits files with edits we know older than what region currently has

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0
    • Component/s: Performance
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Tags:
      noob

      Description

      Testing 0.92, I crashed all servers out. Another bug makes it so WALs are not getting cleaned so I had 7000 regions to replay. The distributed split code did a nice job and cluster came back but interesting is that some hot regions ended up having loads of recovered.edits files – tens if not hundreds – to replay against the region (can we bulk load recovered.edits instead of replaying them?). Each recovered.edits file is taking about a second to process (though only about 30 odd edits per file it seems). The region is unavailable during this time.

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #163 (See https://builds.apache.org/job/HBase-0.92/163/)
        HBASE-4869 Backport to 0.92: HBASE-4797 [availability] Skip recovered.edits
        files with edits older than what region currently has (Jimmy Xiang)

        tedyu :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #163 (See https://builds.apache.org/job/HBase-0.92/163/ ) HBASE-4869 Backport to 0.92: HBASE-4797 [availability] Skip recovered.edits files with edits older than what region currently has (Jimmy Xiang) tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #22 (See https://builds.apache.org/job/HBase-0.92-security/22/)
        HBASE-4869 Backport to 0.92: HBASE-4797 [availability] Skip recovered.edits
        files with edits older than what region currently has (Jimmy Xiang)

        tedyu :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #22 (See https://builds.apache.org/job/HBase-0.92-security/22/ ) HBASE-4869 Backport to 0.92: HBASE-4797 [availability] Skip recovered.edits files with edits older than what region currently has (Jimmy Xiang) tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2474 (See https://builds.apache.org/job/HBase-TRUNK/2474/)
        HBASE-4797 [availability] Skip recovered.edits files with edits we know older than what region currently has (Jimmy Jiang)

        tedyu :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2474 (See https://builds.apache.org/job/HBase-TRUNK/2474/ ) HBASE-4797 [availability] Skip recovered.edits files with edits we know older than what region currently has (Jimmy Jiang) tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #6 (See https://builds.apache.org/job/HBase-TRUNK-security/6/)
        HBASE-4797 [availability] Skip recovered.edits files with edits we know older than what region currently has (Jimmy Jiang)

        tedyu :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #6 (See https://builds.apache.org/job/HBase-TRUNK-security/6/ ) HBASE-4797 [availability] Skip recovered.edits files with edits we know older than what region currently has (Jimmy Jiang) tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
        Hide
        Ted Yu added a comment - - edited

        Integrated to TRUNK.

        Thanks for the patch Jimmy.

        Thanks for the review, Lars and Stack.

        Show
        Ted Yu added a comment - - edited Integrated to TRUNK. Thanks for the patch Jimmy. Thanks for the review, Lars and Stack.
        Hide
        Jimmy Xiang added a comment -

        Can someone check in the patch?

        Show
        Jimmy Xiang added a comment - Can someone check in the patch?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504779/0001-HBASE-4797-availability-skip-files-with-edits-we-kno.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated -162 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 66 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.master.TestMasterFailover
        org.apache.hadoop.hbase.client.TestAdmin

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/337//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/337//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/337//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504779/0001-HBASE-4797-availability-skip-files-with-edits-we-kno.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 66 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestMasterFailover org.apache.hadoop.hbase.client.TestAdmin Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/337//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/337//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/337//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504777/0001-HBASE-4797-availability-skip-files-with-edits-we-kno.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated -162 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 66 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestInstantSchemaChange
        org.apache.hadoop.hbase.client.TestAdmin

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/336//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/336//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/336//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504777/0001-HBASE-4797-availability-skip-files-with-edits-we-kno.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 66 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestInstantSchemaChange org.apache.hadoop.hbase.client.TestAdmin Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/336//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/336//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/336//console This message is automatically generated.
        Hide
        Jimmy Xiang added a comment -

        Thanks! I cancel and resubmit the patch.

        Show
        Jimmy Xiang added a comment - Thanks! I cancel and resubmit the patch.
        Hide
        stack added a comment -

        @Jimmy Just FYI, since you are new, to trigger the build again, you need to re-upload the original patch or a new one (which you did), then (I think) you need to cancel and resubmit the patch.

        Show
        stack added a comment - @Jimmy Just FYI, since you are new, to trigger the build again, you need to re-upload the original patch or a new one (which you did), then (I think) you need to cancel and resubmit the patch.
        Hide
        Jimmy Xiang added a comment -

        This patch is done with --no-prefix.

        Please check it in.

        Thanks a lot!

        Show
        Jimmy Xiang added a comment - This patch is done with --no-prefix. Please check it in. Thanks a lot!
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2906/#review3425
        -----------------------------------------------------------

        +1 on patch

        • ramkrishna

        On 2011-11-22 01:02:17, Jimmy Xiang wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2906/

        -----------------------------------------------------------

        (Updated 2011-11-22 01:02:17)

        Review request for hbase, Todd Lipcon and Michael Stack.

        Summary

        -------

        If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored.

        This addresses bug HBASE-4797.

        https://issues.apache.org/jira/browse/HBASE-4797

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661

        src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b

        Diff: https://reviews.apache.org/r/2906/diff

        Testing

        -------

        Added test case to TestHRegion, and all the tests in this test are passed.

        Thanks,

        Jimmy

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/#review3425 ----------------------------------------------------------- +1 on patch ramkrishna On 2011-11-22 01:02:17, Jimmy Xiang wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/ ----------------------------------------------------------- (Updated 2011-11-22 01:02:17) Review request for hbase, Todd Lipcon and Michael Stack. Summary ------- If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored. This addresses bug HBASE-4797 . https://issues.apache.org/jira/browse/HBASE-4797 Diffs ----- src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b Diff: https://reviews.apache.org/r/2906/diff Testing ------- Added test case to TestHRegion, and all the tests in this test are passed. Thanks, Jimmy
        Hide
        stack added a comment -

        No objection from me. Jimmy, want to attach patch with --no-prefix so hadoopqa runs?

        Show
        stack added a comment - No objection from me. Jimmy, want to attach patch with --no-prefix so hadoopqa runs?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504687/0001-HBASE-4797-%5Bavailability%5D-skip-older-edits.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/326//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504687/0001-HBASE-4797-%5Bavailability%5D-skip-older-edits.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/326//console This message is automatically generated.
        Hide
        Lars Hofhansl added a comment -

        I'm happy to commit this (pending the test run). Any objections?

        Show
        Lars Hofhansl added a comment - I'm happy to commit this (pending the test run). Any objections?
        Hide
        Jimmy Xiang added a comment -

        Removed white spaces in TestHRegion.java

        Show
        Jimmy Xiang added a comment - Removed white spaces in TestHRegion.java
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2906/
        -----------------------------------------------------------

        (Updated 2011-11-22 01:02:17.373022)

        Review request for hbase, Todd Lipcon and Michael Stack.

        Changes
        -------

        Removed white spaces in TestHRegion.java

        Summary
        -------

        If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored.

        This addresses bug HBASE-4797.
        https://issues.apache.org/jira/browse/HBASE-4797

        Diffs (updated)


        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661
        src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b

        Diff: https://reviews.apache.org/r/2906/diff

        Testing
        -------

        Added test case to TestHRegion, and all the tests in this test are passed.

        Thanks,

        Jimmy

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/ ----------------------------------------------------------- (Updated 2011-11-22 01:02:17.373022) Review request for hbase, Todd Lipcon and Michael Stack. Changes ------- Removed white spaces in TestHRegion.java Summary ------- If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored. This addresses bug HBASE-4797 . https://issues.apache.org/jira/browse/HBASE-4797 Diffs (updated) src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b Diff: https://reviews.apache.org/r/2906/diff Testing ------- Added test case to TestHRegion, and all the tests in this test are passed. Thanks, Jimmy
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2906/#review3416
        -----------------------------------------------------------

        Ship it!

        Looks good to me.
        Thanks for fixing the whitespace too (although it made the patch harder to read).
        You also left some whitespace in testSkipRecoveredEditsReplay.

        • Lars

        On 2011-11-22 00:32:48, Jimmy Xiang wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2906/

        -----------------------------------------------------------

        (Updated 2011-11-22 00:32:48)

        Review request for hbase, Todd Lipcon and Michael Stack.

        Summary

        -------

        If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored.

        This addresses bug HBASE-4797.

        https://issues.apache.org/jira/browse/HBASE-4797

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661

        src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b

        Diff: https://reviews.apache.org/r/2906/diff

        Testing

        -------

        Added test case to TestHRegion, and all the tests in this test are passed.

        Thanks,

        Jimmy

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/#review3416 ----------------------------------------------------------- Ship it! Looks good to me. Thanks for fixing the whitespace too (although it made the patch harder to read). You also left some whitespace in testSkipRecoveredEditsReplay. Lars On 2011-11-22 00:32:48, Jimmy Xiang wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/ ----------------------------------------------------------- (Updated 2011-11-22 00:32:48) Review request for hbase, Todd Lipcon and Michael Stack. Summary ------- If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored. This addresses bug HBASE-4797 . https://issues.apache.org/jira/browse/HBASE-4797 Diffs ----- src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b Diff: https://reviews.apache.org/r/2906/diff Testing ------- Added test case to TestHRegion, and all the tests in this test are passed. Thanks, Jimmy
        Hide
        Jimmy Xiang added a comment -

        Cool! Thanks.

        On Mon, Nov 21, 2011 at 2:51 PM, Kannan Muthukkaruppan (Commented) (JIRA) <

        Show
        Jimmy Xiang added a comment - Cool! Thanks. On Mon, Nov 21, 2011 at 2:51 PM, Kannan Muthukkaruppan (Commented) (JIRA) <
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2906/
        -----------------------------------------------------------

        (Updated 2011-11-22 00:32:48.813017)

        Review request for hbase, Todd Lipcon and Michael Stack.

        Changes
        -------

        Revised patch with changes per review.

        Summary
        -------

        If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored.

        This addresses bug HBASE-4797.
        https://issues.apache.org/jira/browse/HBASE-4797

        Diffs (updated)


        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661
        src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b

        Diff: https://reviews.apache.org/r/2906/diff

        Testing
        -------

        Added test case to TestHRegion, and all the tests in this test are passed.

        Thanks,

        Jimmy

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/ ----------------------------------------------------------- (Updated 2011-11-22 00:32:48.813017) Review request for hbase, Todd Lipcon and Michael Stack. Changes ------- Revised patch with changes per review. Summary ------- If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored. This addresses bug HBASE-4797 . https://issues.apache.org/jira/browse/HBASE-4797 Diffs (updated) src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b Diff: https://reviews.apache.org/r/2906/diff Testing ------- Added test case to TestHRegion, and all the tests in this test are passed. Thanks, Jimmy
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-11-21 22:47:55, Michael Stack wrote:

        > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 2456

        > <https://reviews.apache.org/r/2906/diff/2/?file=59652#file59652line2456>

        >

        > So, are these already sorted in right order from oldest edit to newest?

        All these files are under the same folder, if these files have the same name pattern as defined in HLog: String.format("%019d", seqid);
        yes, they are sorted in the right order based on the sequence id number.

        If this is not true, then the order to reapply these edits is already wrong.

        On 2011-11-21 22:47:55, Michael Stack wrote:

        > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 2475

        > <https://reviews.apache.org/r/2906/diff/2/?file=59652#file59652line2475>

        >

        > Possilbe should be Possible.

        >

        > I'd be more assertive in this message. "Maximum possible sequenceid for this log is " + + ", skipping ..

        Sure, I will fix it.

        On 2011-11-21 22:47:55, Michael Stack wrote:

        > src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java, line 2855

        > <https://reviews.apache.org/r/2906/diff/2/?file=59653#file59653line2855>

        >

        > Any more asserts we can do in here? Assert we replayed N of the M files?

        Sure, I added more test cases.

        • Jimmy

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2906/#review3409
        -----------------------------------------------------------

        On 2011-11-21 22:38:39, Jimmy Xiang wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2906/

        -----------------------------------------------------------

        (Updated 2011-11-21 22:38:39)

        Review request for hbase, Todd Lipcon and Michael Stack.

        Summary

        -------

        If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored.

        This addresses bug HBASE-4797.

        https://issues.apache.org/jira/browse/HBASE-4797

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661

        src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b

        Diff: https://reviews.apache.org/r/2906/diff

        Testing

        -------

        Added test case to TestHRegion, and all the tests in this test are passed.

        Thanks,

        Jimmy

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-11-21 22:47:55, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 2456 > < https://reviews.apache.org/r/2906/diff/2/?file=59652#file59652line2456 > > > So, are these already sorted in right order from oldest edit to newest? All these files are under the same folder, if these files have the same name pattern as defined in HLog: String.format("%019d", seqid); yes, they are sorted in the right order based on the sequence id number. If this is not true, then the order to reapply these edits is already wrong. On 2011-11-21 22:47:55, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 2475 > < https://reviews.apache.org/r/2906/diff/2/?file=59652#file59652line2475 > > > Possilbe should be Possible. > > I'd be more assertive in this message. "Maximum possible sequenceid for this log is " + + ", skipping .. Sure, I will fix it. On 2011-11-21 22:47:55, Michael Stack wrote: > src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java, line 2855 > < https://reviews.apache.org/r/2906/diff/2/?file=59653#file59653line2855 > > > Any more asserts we can do in here? Assert we replayed N of the M files? Sure, I added more test cases. Jimmy ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/#review3409 ----------------------------------------------------------- On 2011-11-21 22:38:39, Jimmy Xiang wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/ ----------------------------------------------------------- (Updated 2011-11-21 22:38:39) Review request for hbase, Todd Lipcon and Michael Stack. Summary ------- If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored. This addresses bug HBASE-4797 . https://issues.apache.org/jira/browse/HBASE-4797 Diffs ----- src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b Diff: https://reviews.apache.org/r/2906/diff Testing ------- Added test case to TestHRegion, and all the tests in this test are passed. Thanks, Jimmy
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-11-21 23:23:07, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 2468

        > <https://reviews.apache.org/r/2906/diff/2/?file=59652#file59652line2468>

        >

        > maxSedId should be named maxSeqId

        Good catch.

        • Jimmy

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2906/#review3413
        -----------------------------------------------------------

        On 2011-11-21 22:38:39, Jimmy Xiang wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2906/

        -----------------------------------------------------------

        (Updated 2011-11-21 22:38:39)

        Review request for hbase, Todd Lipcon and Michael Stack.

        Summary

        -------

        If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored.

        This addresses bug HBASE-4797.

        https://issues.apache.org/jira/browse/HBASE-4797

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661

        src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b

        Diff: https://reviews.apache.org/r/2906/diff

        Testing

        -------

        Added test case to TestHRegion, and all the tests in this test are passed.

        Thanks,

        Jimmy

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-11-21 23:23:07, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 2468 > < https://reviews.apache.org/r/2906/diff/2/?file=59652#file59652line2468 > > > maxSedId should be named maxSeqId Good catch. Jimmy ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/#review3413 ----------------------------------------------------------- On 2011-11-21 22:38:39, Jimmy Xiang wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/ ----------------------------------------------------------- (Updated 2011-11-21 22:38:39) Review request for hbase, Todd Lipcon and Michael Stack. Summary ------- If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored. This addresses bug HBASE-4797 . https://issues.apache.org/jira/browse/HBASE-4797 Diffs ----- src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b Diff: https://reviews.apache.org/r/2906/diff Testing ------- Added test case to TestHRegion, and all the tests in this test are passed. Thanks, Jimmy
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2906/#review3413
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        <https://reviews.apache.org/r/2906/#comment7642>

        maxSedId should be named maxSeqId

        • Ted

        On 2011-11-21 22:38:39, Jimmy Xiang wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2906/

        -----------------------------------------------------------

        (Updated 2011-11-21 22:38:39)

        Review request for hbase, Todd Lipcon and Michael Stack.

        Summary

        -------

        If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored.

        This addresses bug HBASE-4797.

        https://issues.apache.org/jira/browse/HBASE-4797

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661

        src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b

        Diff: https://reviews.apache.org/r/2906/diff

        Testing

        -------

        Added test case to TestHRegion, and all the tests in this test are passed.

        Thanks,

        Jimmy

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/#review3413 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/2906/#comment7642 > maxSedId should be named maxSeqId Ted On 2011-11-21 22:38:39, Jimmy Xiang wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/ ----------------------------------------------------------- (Updated 2011-11-21 22:38:39) Review request for hbase, Todd Lipcon and Michael Stack. Summary ------- If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored. This addresses bug HBASE-4797 . https://issues.apache.org/jira/browse/HBASE-4797 Diffs ----- src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b Diff: https://reviews.apache.org/r/2906/diff Testing ------- Added test case to TestHRegion, and all the tests in this test are passed. Thanks, Jimmy
        Hide
        stack added a comment -

        The region opening is tried periodically. The waiting interval is about 1/3 of the assignment time out. I think that's fine.

        From the log snippet above though Jimmy, it seems like we are updating the znode every second almost. Thats too much?

        Show
        stack added a comment - The region opening is tried periodically. The waiting interval is about 1/3 of the assignment time out. I think that's fine. From the log snippet above though Jimmy, it seems like we are updating the znode every second almost. Thats too much?
        Hide
        stack added a comment -

        Updated JIRA subject.

        Show
        stack added a comment - Updated JIRA subject.
        Hide
        Kannan Muthukkaruppan added a comment -

        The "title" for the bug can be updated given that we are no longer renaming the files in recovered.edits. [That concerned me initially -- but reading through the details, looks like you have come up with a way to avoid new name format. That's always smoother for upgrades and such..]

        Show
        Kannan Muthukkaruppan added a comment - The "title" for the bug can be updated given that we are no longer renaming the files in recovered.edits. [That concerned me initially -- but reading through the details, looks like you have come up with a way to avoid new name format. That's always smoother for upgrades and such..]
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2906/#review3409
        -----------------------------------------------------------

        Very nice patch.

        In future, would suggest you confine your change just to what you are adding. The white space cleanup is nice but it distracts from your patch. It also bloats it and makes it look intimidating to review (smile).

        Minor fixups only.

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        <https://reviews.apache.org/r/2906/#comment7635>

        So, are these already sorted in right order from oldest edit to newest?

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        <https://reviews.apache.org/r/2906/#comment7636>

        Possilbe should be Possible.

        I'd be more assertive in this message. "Maximum possible sequenceid for this log is " + + ", skipping ..

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        <https://reviews.apache.org/r/2906/#comment7637>

        Good.

        src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
        <https://reviews.apache.org/r/2906/#comment7638>

        Any more asserts we can do in here? Assert we replayed N of the M files?

        • Michael

        On 2011-11-21 22:38:39, Jimmy Xiang wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2906/

        -----------------------------------------------------------

        (Updated 2011-11-21 22:38:39)

        Review request for hbase, Todd Lipcon and Michael Stack.

        Summary

        -------

        If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored.

        This addresses bug HBASE-4797.

        https://issues.apache.org/jira/browse/HBASE-4797

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661

        src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b

        Diff: https://reviews.apache.org/r/2906/diff

        Testing

        -------

        Added test case to TestHRegion, and all the tests in this test are passed.

        Thanks,

        Jimmy

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/#review3409 ----------------------------------------------------------- Very nice patch. In future, would suggest you confine your change just to what you are adding. The white space cleanup is nice but it distracts from your patch. It also bloats it and makes it look intimidating to review (smile). Minor fixups only. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/2906/#comment7635 > So, are these already sorted in right order from oldest edit to newest? src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/2906/#comment7636 > Possilbe should be Possible. I'd be more assertive in this message. "Maximum possible sequenceid for this log is " + + ", skipping .. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/2906/#comment7637 > Good. src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java < https://reviews.apache.org/r/2906/#comment7638 > Any more asserts we can do in here? Assert we replayed N of the M files? Michael On 2011-11-21 22:38:39, Jimmy Xiang wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/ ----------------------------------------------------------- (Updated 2011-11-21 22:38:39) Review request for hbase, Todd Lipcon and Michael Stack. Summary ------- If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored. This addresses bug HBASE-4797 . https://issues.apache.org/jira/browse/HBASE-4797 Diffs ----- src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b Diff: https://reviews.apache.org/r/2906/diff Testing ------- Added test case to TestHRegion, and all the tests in this test are passed. Thanks, Jimmy
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2906/
        -----------------------------------------------------------

        Review request for hbase, Todd Lipcon and Michael Stack.

        Summary
        -------

        If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored.

        This addresses bug HBASE-4797.
        https://issues.apache.org/jira/browse/HBASE-4797

        Diffs


        src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661
        src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b

        Diff: https://reviews.apache.org/r/2906/diff

        Testing
        -------

        Added test case to TestHRegion, and all the tests in this test are passed.

        Thanks,

        Jimmy

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2906/ ----------------------------------------------------------- Review request for hbase, Todd Lipcon and Michael Stack. Summary ------- If there are multiple recovered edits files, I used the file name to find the initial sequence id. After these files are sorted, we can find a file's possible maximum sequence id based on the next file's initial sequence id. If the maximum sequence id is smaller than the current sequence id, the whole recovered edits file is old and ignored. This addresses bug HBASE-4797 . https://issues.apache.org/jira/browse/HBASE-4797 Diffs src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8b89661 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 5daa02b Diff: https://reviews.apache.org/r/2906/diff Testing ------- Added test case to TestHRegion, and all the tests in this test are passed. Thanks, Jimmy
        Hide
        Jimmy Xiang added a comment -

        The region opening is tried periodically. The waiting interval is about 1/3 of the assignment time out. I think that's fine.

        Show
        Jimmy Xiang added a comment - The region opening is tried periodically. The waiting interval is about 1/3 of the assignment time out. I think that's fine.
        Hide
        Jimmy Xiang added a comment -

        Yes, that's what I was thinking. The file name has the start seq id. If
        there are multiple files, there should be multiple start seq ids. That
        implies the max seq ids in
        some of these files, if sorted. I can use these information to filter out
        some files safely.

        On Mon, Nov 21, 2011 at 10:52 AM, stack (Commented) (JIRA)

        Show
        Jimmy Xiang added a comment - Yes, that's what I was thinking. The file name has the start seq id. If there are multiple files, there should be multiple start seq ids. That implies the max seq ids in some of these files, if sorted. I can use these information to filter out some files safely. On Mon, Nov 21, 2011 at 10:52 AM, stack (Commented) (JIRA)
        Hide
        stack added a comment -

        Thanks Jimmy for taking this on. Looks like you don't have to rename the files; just sort them and figure which set to apply (and do what Todd suggests rewriting the znode less often – or asynchronously).

        Show
        stack added a comment - Thanks Jimmy for taking this on. Looks like you don't have to rename the files; just sort them and figure which set to apply (and do what Todd suggests rewriting the znode less often – or asynchronously).
        Hide
        Todd Lipcon added a comment -

        Plus, looks like we're burning a lot of time with synchronous updates to the region opening "twiddle". Perhaps add a little timestamp in there that we only twiddle it every 5 seconds (or do it async)

        Show
        Todd Lipcon added a comment - Plus, looks like we're burning a lot of time with synchronous updates to the region opening "twiddle". Perhaps add a little timestamp in there that we only twiddle it every 5 seconds (or do it async)
        Hide
        stack added a comment -

        Thinking some more on this, we don't need to rename recovered.edits files. The files are named for the first sequenceid in the file, so, we could just do file listing and sort the return. Then we'd have range of sequenceids per file. We could then just pass on files with edits that are smaller than regions current seqid.

        Show
        stack added a comment - Thinking some more on this, we don't need to rename recovered.edits files. The files are named for the first sequenceid in the file, so, we could just do file listing and sort the return. Then we'd have range of sequenceids per file. We could then just pass on files with edits that are smaller than regions current seqid.
        Hide
        stack added a comment -

        Oh... i suppose its a bit worse than I though. I'm looking at a region that has nearly 6k recovered.edits files to replay. The RegionServer is doing this per file:

        2011-11-16 03:06:02,403 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 0, skipped 33, firstSequenceidInLog=296860, maxSequenceidInLog=351600, path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296860
        2011-11-16 03:06:02,405 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296914; minSequenceid=351600; path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296914
        2011-11-16 03:06:05,097 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x133a5bab186271f Attempting to transition node 69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
        2011-11-16 03:06:05,278 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x133a5bab186271f Successfully transitioned node 69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
        2011-11-16 03:06:05,278 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 0, skipped 33, firstSequenceidInLog=296914, maxSequenceidInLog=351600, path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296914
        2011-11-16 03:06:05,279 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296970; minSequenceid=351600; path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296970
        2011-11-16 03:06:05,952 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x133a5bab186271f Attempting to transition node 69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
        2011-11-16 03:06:06,093 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x133a5bab186271f Successfully transitioned node 69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
        2011-11-16 03:06:06,093 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 0, skipped 44, firstSequenceidInLog=296970, maxSequenceidInLog=351600, path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296970
        2011-11-16 03:06:06,094 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000297041; minSequenceid=351600; path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000297041
        2011-11-16 03:06:06,795 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x133a5bab186271f Attempting to transition node 69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
        2011-11-16 03:06:06,810 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x133a5bab186271f Successfully transitioned node 69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
        
        Show
        stack added a comment - Oh... i suppose its a bit worse than I though. I'm looking at a region that has nearly 6k recovered.edits files to replay. The RegionServer is doing this per file: 2011-11-16 03:06:02,403 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 0, skipped 33, firstSequenceidInLog=296860, maxSequenceidInLog=351600, path=hdfs: //sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296860 2011-11-16 03:06:02,405 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs: //sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296914; minSequenceid=351600; path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296914 2011-11-16 03:06:05,097 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x133a5bab186271f Attempting to transition node 69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-11-16 03:06:05,278 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x133a5bab186271f Successfully transitioned node 69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-11-16 03:06:05,278 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 0, skipped 33, firstSequenceidInLog=296914, maxSequenceidInLog=351600, path=hdfs: //sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296914 2011-11-16 03:06:05,279 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs: //sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296970; minSequenceid=351600; path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296970 2011-11-16 03:06:05,952 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x133a5bab186271f Attempting to transition node 69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-11-16 03:06:06,093 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x133a5bab186271f Successfully transitioned node 69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-11-16 03:06:06,093 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 0, skipped 44, firstSequenceidInLog=296970, maxSequenceidInLog=351600, path=hdfs: //sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000296970 2011-11-16 03:06:06,094 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs: //sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000297041; minSequenceid=351600; path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0000000000000297041 2011-11-16 03:06:06,795 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x133a5bab186271f Attempting to transition node 69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-11-16 03:06:06,810 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x133a5bab186271f Successfully transitioned node 69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING

          People

          • Assignee:
            Jimmy Xiang
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development