HBase
  1. HBase
  2. HBASE-7006

[MTTR] Improve Region Server Recovery Time - Distributed Log Replay

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.0, 0.95.1
    • Component/s: MTTR
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Distributed Log Replay Description:

      After a region server fails, we firstly assign a failed region to another region server with recovering state marked in ZooKeeper. Then a SplitLogWorker directly replays edits from WAL(Write-Ahead-Log)s of the failed region server to the region after it's re-opened in the new location. When a region is in recovering state, it can also accept writes but no reads(including Append and Increment), region split or merge.

      The feature piggybacks on existing distributed log splitting framework and directly replay WAL edits to another region server instead of creating recovered.edits files.

      The advantages over existing log splitting recovered edits implementation:
      1) Eliminate the steps to write and read recovered.edits files. There could be thousands of recovered.edits files are created and written concurrently during a region server recovery. Many small random writes could degrade the overall system performance.
      2) Allow writes even when a region is in recovering state. It only takes seconds for a failed over region to accept writes again.

      The feature can be enabled by setting hbase.master.distributed.log.replay to true (by default is false)
      Show
      Distributed Log Replay Description: After a region server fails, we firstly assign a failed region to another region server with recovering state marked in ZooKeeper. Then a SplitLogWorker directly replays edits from WAL(Write-Ahead-Log)s of the failed region server to the region after it's re-opened in the new location. When a region is in recovering state, it can also accept writes but no reads(including Append and Increment), region split or merge. The feature piggybacks on existing distributed log splitting framework and directly replay WAL edits to another region server instead of creating recovered.edits files. The advantages over existing log splitting recovered edits implementation: 1) Eliminate the steps to write and read recovered.edits files. There could be thousands of recovered.edits files are created and written concurrently during a region server recovery. Many small random writes could degrade the overall system performance. 2) Allow writes even when a region is in recovering state. It only takes seconds for a failed over region to accept writes again. The feature can be enabled by setting hbase.master.distributed.log.replay to true (by default is false)

      Description

      Just saw interesting issue where a cluster went down hard and 30 nodes had 1700 WALs to replay. Replay took almost an hour. It looks like it could run faster that much of the time is spent zk'ing and nn'ing.

      Putting in 0.96 so it gets a look at least. Can always punt.

      1. 7006-addendum-3.txt
        2 kB
        Ted Yu
      2. hbase-7006-addendum.patch
        1.0 kB
        Jeffrey Zhong
      3. hbase-7006-combined.patch
        234 kB
        Jeffrey Zhong
      4. hbase-7006-combined-v1.patch
        246 kB
        Jeffrey Zhong
      5. hbase-7006-combined-v4.patch
        298 kB
        Jeffrey Zhong
      6. hbase-7006-combined-v5.patch
        307 kB
        Jeffrey Zhong
      7. hbase-7006-combined-v6.patch
        315 kB
        Jeffrey Zhong
      8. hbase-7006-combined-v7.patch
        315 kB
        Jeffrey Zhong
      9. hbase-7006-combined-v8.patch
        311 kB
        Jeffrey Zhong
      10. hbase-7006-combined-v9.patch
        312 kB
        Jeffrey Zhong
      11. LogSplitting Comparison.pdf
        50 kB
        Jeffrey Zhong
      12. ProposaltoimprovelogsplittingprocessregardingtoHBASE-7006-v2.pdf
        130 kB
        Jeffrey Zhong

        Issue Links

          Activity

          Hide
          Nicolas Liochon added a comment -

          Nothing related to HBASE-6738?
          There is not a limit of 32 WALs per node (hence 900 wals)? Or have you lost more nodes?

          Show
          Nicolas Liochon added a comment - Nothing related to HBASE-6738 ? There is not a limit of 32 WALs per node (hence 900 wals)? Or have you lost more nodes?
          Hide
          stack added a comment -

          Nicolas Liochon No sir. Limit was 8 WALs but write rate overran the limit so almost 40 WALs each.

          Show
          stack added a comment - Nicolas Liochon No sir. Limit was 8 WALs but write rate overran the limit so almost 40 WALs each.
          Hide
          Jeffrey Zhong added a comment -

          Hey,

          I've came up a proposal to improve current log splitting process. Feedbacks are welcome!

          I temporarily assign the JIRA to me so that I can implement the proposal once it gets green light.

          Thanks,
          -Jeffrey

          Show
          Jeffrey Zhong added a comment - Hey, I've came up a proposal to improve current log splitting process. Feedbacks are welcome! I temporarily assign the JIRA to me so that I can implement the proposal once it gets green light. Thanks, -Jeffrey
          Hide
          stack added a comment -

          Excellent write up Jeffrey.

          Was thinking myself that we might do what Nicolas suggests on the end.

          It looks like you handle failures properly.

          Savings will be large I'd think.

          Actually simplifies the log splitting process I'd say.

          What if we do multiple WALs per regionserver? That shouldn't change your processing model far as I can see.

          Show
          stack added a comment - Excellent write up Jeffrey. Was thinking myself that we might do what Nicolas suggests on the end. It looks like you handle failures properly. Savings will be large I'd think. Actually simplifies the log splitting process I'd say. What if we do multiple WALs per regionserver? That shouldn't change your processing model far as I can see.
          Hide
          stack added a comment -

          Making major rather than critical. If done in time, that'd be excellent but won't hold up 0.96 for it.

          Show
          stack added a comment - Making major rather than critical. If done in time, that'd be excellent but won't hold up 0.96 for it.
          Hide
          Jeffrey Zhong added a comment -

          Thanks Stack for reviewing the proposal!

          What if we do multiple WALs per regionserver? That shouldn't change your processing model far as I can see.

          Yeah, you're right. multiples WALs per RS won't affect the proposal.

          Thanks,
          -Jeffrey

          Show
          Jeffrey Zhong added a comment - Thanks Stack for reviewing the proposal! What if we do multiple WALs per regionserver? That shouldn't change your processing model far as I can see. Yeah, you're right. multiples WALs per RS won't affect the proposal. Thanks, -Jeffrey
          Hide
          Jeffrey Zhong added a comment -

          The new implementation increases performance about 2 times and become much better when more regions to be recovered.

          The new implementation allows writes in recovery so we won't have data loss for "time series data" usage pattern. The down time for writes normally is about one or two seconds.

          Thanks,
          -Jeffrey

          Show
          Jeffrey Zhong added a comment - The new implementation increases performance about 2 times and become much better when more regions to be recovered. The new implementation allows writes in recovery so we won't have data loss for "time series data" usage pattern. The down time for writes normally is about one or two seconds. Thanks, -Jeffrey
          Hide
          Ted Yu added a comment -

          This is encouraging.

          Looking forward to your patch.

          Show
          Ted Yu added a comment - This is encouraging. Looking forward to your patch.
          Hide
          Enis Soztutar added a comment -

          These are excellent results, especially with large # of regions. Also we will benefit from other improvements on connection management, region discovery, etc, which means that those numbers can go even lower. Let's try to get this in with the current set of changes, then as we debug more and learn more, we can do follow ups.

          One thing we did not test is to not write a file per region per WAL file, but do the bigtable approach. Namely, for each WAL file, read up until DFS block size (128MB), sort the edits per region in memory, and write a file per block. The files have a simple index per region. Not sure how we can test that easily though.

          Show
          Enis Soztutar added a comment - These are excellent results, especially with large # of regions. Also we will benefit from other improvements on connection management, region discovery, etc, which means that those numbers can go even lower. Let's try to get this in with the current set of changes, then as we debug more and learn more, we can do follow ups. One thing we did not test is to not write a file per region per WAL file, but do the bigtable approach. Namely, for each WAL file, read up until DFS block size (128MB), sort the edits per region in memory, and write a file per block. The files have a simple index per region. Not sure how we can test that easily though.
          Hide
          Jeffrey Zhong added a comment -

          The big table approach is kind of middle ground approach between the existing implementation and the proposal in the JIRA. The file block implementation seems need more work though. Each region server has to read all those newly created block files to replay edits but cut writes significantly so it should have improvements over existing approach(not the new proposal as it still read recovery data twice: one is in log splitting and the other is in replay phase and incur some extra writes).

          Thanks,
          -Jeffrey

          Show
          Jeffrey Zhong added a comment - The big table approach is kind of middle ground approach between the existing implementation and the proposal in the JIRA. The file block implementation seems need more work though. Each region server has to read all those newly created block files to replay edits but cut writes significantly so it should have improvements over existing approach(not the new proposal as it still read recovery data twice: one is in log splitting and the other is in replay phase and incur some extra writes). Thanks, -Jeffrey
          Hide
          Enis Soztutar added a comment -

          Agreed that it is the middle ground. On region open, RS has to do a read on the index, and seek, and sequential read for each region. However, in your approach as you reported off-list, we are paying for re-locating the regions, and the rpc overhead instead of just streaming sequential writes to hdfs. I was just curious, given the current implementation, which one would be faster. I am not suggesting that we should prototype that as well, especially given that we can open the regions for writes in 1-2 secs with this.

          Show
          Enis Soztutar added a comment - Agreed that it is the middle ground. On region open, RS has to do a read on the index, and seek, and sequential read for each region. However, in your approach as you reported off-list, we are paying for re-locating the regions, and the rpc overhead instead of just streaming sequential writes to hdfs. I was just curious, given the current implementation, which one would be faster. I am not suggesting that we should prototype that as well, especially given that we can open the regions for writes in 1-2 secs with this.
          Hide
          Ted Yu added a comment -

          @Jeff:
          Do you plan to publish your patch in sub-task of this JIRA ?

          Show
          Ted Yu added a comment - @Jeff: Do you plan to publish your patch in sub-task of this JIRA ?
          Hide
          Jeffrey Zhong added a comment -

          @Ted,

          Yes, my first patch will include major logic for this JIRA and will be attached to a sub task JIRA(to be created) and be submitted within these two days. There will be two more sub JIRAs: one is to create a replay command and the other is to add metrics for better reporting purpose.

          Thanks,
          -Jeffrey

          Show
          Jeffrey Zhong added a comment - @Ted, Yes, my first patch will include major logic for this JIRA and will be attached to a sub task JIRA(to be created) and be submitted within these two days. There will be two more sub JIRAs: one is to create a replay command and the other is to add metrics for better reporting purpose. Thanks, -Jeffrey
          Hide
          Jeffrey Zhong added a comment -

          Mark it critical so that we can ship this into 0.96.

          Thanks,
          -Jeffrey

          Show
          Jeffrey Zhong added a comment - Mark it critical so that we can ship this into 0.96. Thanks, -Jeffrey
          Hide
          Jimmy Xiang added a comment -

          I read the proposal and have some questions. At first, it sounds we trade disk io to network io, which should have better performance. As to the memstore flush write saving after recovered.edits have been replayed, the proposal needs to do the same, right? You just write them to another WAL file, isn't it true?

          Suppose a region server failed again in the middle, does a split worker need to split the WAL again? This means a WAL may be read/split multiple times?

          In the attached performance testing, do we have a breakdown on how many time it spends on reading the log file, writing to the recovered edits file? How did you measure the log splitting time?

          Show
          Jimmy Xiang added a comment - I read the proposal and have some questions. At first, it sounds we trade disk io to network io, which should have better performance. As to the memstore flush write saving after recovered.edits have been replayed, the proposal needs to do the same, right? You just write them to another WAL file, isn't it true? Suppose a region server failed again in the middle, does a split worker need to split the WAL again? This means a WAL may be read/split multiple times? In the attached performance testing, do we have a breakdown on how many time it spends on reading the log file, writing to the recovered edits file? How did you measure the log splitting time?
          Hide
          Jeffrey Zhong added a comment -

          it sounds we trade disk io to network io

          No, we cut both disk io and network ios relating to recovered.edits files creations & deletions.

          Currently we replay the wal to the destination region server while in old way the destination RS reads recovered edits from underlying hdfs. In terms of network io, they're same because the old way still need read recovered edits file across wire. The difference is that in distributed replay wal edits are pushed to the destination RS while the old way is pulling edits from recovered edits(which are intermediate files).

          In summary, the IOs related to recovered.edits files are all gone without any extra IOs. I think this question is common and I'll include this in the write up.

          Suppose a region server failed again in the middle, does a split worker need to split the WAL again? This means a WAL may be read/split multiple times

          We handle sequential RS failures like a new RS failure and replay its WALs left behind. We may read a WAL multiple times in sequential failures but not replay multiple times if edits are flushed.

          In the attached performance testing, do we have a breakdown on how many time it spends on reading the log file, writing to the recovered edits file? How did you measure the log splitting time?

          I don't have the break down since reading and writing happen at the same time. In normal cases, writing finish several secs after reading is done. We have metrics in splitlogmanager which measures the total splitting time and that's what I used in the testing.

          The latest combined patch is attached in 7837.

          Show
          Jeffrey Zhong added a comment - it sounds we trade disk io to network io No, we cut both disk io and network ios relating to recovered.edits files creations & deletions. Currently we replay the wal to the destination region server while in old way the destination RS reads recovered edits from underlying hdfs. In terms of network io, they're same because the old way still need read recovered edits file across wire. The difference is that in distributed replay wal edits are pushed to the destination RS while the old way is pulling edits from recovered edits(which are intermediate files). In summary, the IOs related to recovered.edits files are all gone without any extra IOs. I think this question is common and I'll include this in the write up. Suppose a region server failed again in the middle, does a split worker need to split the WAL again? This means a WAL may be read/split multiple times We handle sequential RS failures like a new RS failure and replay its WALs left behind. We may read a WAL multiple times in sequential failures but not replay multiple times if edits are flushed. In the attached performance testing, do we have a breakdown on how many time it spends on reading the log file, writing to the recovered edits file? How did you measure the log splitting time? I don't have the break down since reading and writing happen at the same time. In normal cases, writing finish several secs after reading is done. We have metrics in splitlogmanager which measures the total splitting time and that's what I used in the testing. The latest combined patch is attached in 7837.
          Hide
          Jimmy Xiang added a comment -

          Cool, that's great!

          Show
          Jimmy Xiang added a comment - Cool, that's great!
          Hide
          Jimmy Xiang added a comment -

          You mentioned that this patch depends on some assumption. Have you verified it? If so, which patch should be reviewed and committed at first? Or you want them all be reviewed and committed together?

          Show
          Jimmy Xiang added a comment - You mentioned that this patch depends on some assumption. Have you verified it? If so, which patch should be reviewed and committed at first? Or you want them all be reviewed and committed together?
          Hide
          Jeffrey Zhong added a comment -

          Jimmy Xiang Thanks in advance for reviewing. The assumption documented in the write-up is verified and relies on idempotence of hbase. I think it makes sense to review the combined patch to reduce reviewing effort but I fully relies on each reviewer's preferences.

          Show
          Jeffrey Zhong added a comment - Jimmy Xiang Thanks in advance for reviewing. The assumption documented in the write-up is verified and relies on idempotence of hbase. I think it makes sense to review the combined patch to reduce reviewing effort but I fully relies on each reviewer's preferences.
          Hide
          Jimmy Xiang added a comment -

          I prefer small patches, otherwise, it is hard to review.

          Show
          Jimmy Xiang added a comment - I prefer small patches, otherwise, it is hard to review.
          Hide
          Jeffrey Zhong added a comment -

          Update the write-up to reflect the implementations.

          Show
          Jeffrey Zhong added a comment - Update the write-up to reflect the implementations.
          Hide
          Jeffrey Zhong added a comment -

          Rebase combined patch

          Show
          Jeffrey Zhong added a comment - Rebase combined patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579488/hbase-7006-combined.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.io.TestHeapSize

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579488/hbase-7006-combined.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 2 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.io.TestHeapSize Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5356//console This message is automatically generated.
          Hide
          Anoop Sam John added a comment -

          Will start reviewing the patch by tomorrow Jeffrey Zhong. This will be an interesting stuff in MTTR.

          Show
          Anoop Sam John added a comment - Will start reviewing the patch by tomorrow Jeffrey Zhong. This will be an interesting stuff in MTTR.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579497/hbase-7006-combined.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.io.TestHeapSize

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579497/hbase-7006-combined.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 2 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.io.TestHeapSize Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5360//console This message is automatically generated.
          Hide
          Jonathan Hsieh added a comment -

          Jeffrey Zhong Do we have any numbers of how this improves our recovery time?

          Show
          Jonathan Hsieh added a comment - Jeffrey Zhong Do we have any numbers of how this improves our recovery time?
          Hide
          Jeffrey Zhong added a comment -

          Jonathan Hsieh The initial performance number is in the attachment 'LogSplitting Comparison.pdf'. Thanks.

          Show
          Jeffrey Zhong added a comment - Jonathan Hsieh The initial performance number is in the attachment 'LogSplitting Comparison.pdf'. Thanks.
          Hide
          Jonathan Hsieh added a comment -

          Lovely. Thanks!

          Show
          Jonathan Hsieh added a comment - Lovely. Thanks!
          Hide
          Ted Yu added a comment -

          For ReplicationZookeeper.java :

          +  public static byte[] toByteArray(
          +      final long position) {
          

          Considering lockToByteArray() method that follows above, maybe rename above as positionToByteArray()

          +  public static final String REGION_ASSIGNMENT_TIME_OUT = "hbase.master.region.assignment.time.out";
          

          How about "hbase.master.region.assignment.timeout" ?

          +  static final String REPLAY_BATCH_SIZE_DESC = "Number of changes of each replay batch.";
          

          ""Number of changes of each" -> ""Number of changes in each"

          For AssignmentManager.java :

          +    long end = (timeOut <= 0) ? Long.MAX_VALUE : System.currentTimeMillis() + timeOut;
          ...
          +      if (System.currentTimeMillis() > end) {
          

          Please use EnvironmentEdge.

          Show
          Ted Yu added a comment - For ReplicationZookeeper.java : + public static byte [] toByteArray( + final long position) { Considering lockToByteArray() method that follows above, maybe rename above as positionToByteArray() + public static final String REGION_ASSIGNMENT_TIME_OUT = "hbase.master.region.assignment.time.out" ; How about "hbase.master.region.assignment.timeout" ? + static final String REPLAY_BATCH_SIZE_DESC = " Number of changes of each replay batch." ; ""Number of changes of each" -> ""Number of changes in each" For AssignmentManager.java : + long end = (timeOut <= 0) ? Long .MAX_VALUE : System .currentTimeMillis() + timeOut; ... + if ( System .currentTimeMillis() > end) { Please use EnvironmentEdge.
          Hide
          stack added a comment -

          Jeffrey Zhong Nice numbers in posted doc.

          What does below mean sir?

          +  // make current mutation as a distributed log replay change
          +  protected boolean isReplay = false;
          

          Why we have this isReplay in a Mutation? Because these edits get treated differently over on serverside?

          Suggest calling the data member replay or logReplay or walReplay and then the accessor is isLogReply or isWALReplay. isReplay is the name of a method that returns whether the data member replay is true or not.

          Does this define belong in this patch?

          + /** Conf key that specifies region assignment timeout value */
          + public static final String REGION_ASSIGNMENT_TIME_OUT = "hbase.master.region.assignment.time.out";

          Why we timing out assignments in this patch?

          Is this log splitting that is referred to in the metric name below?

          + void updateMetaSplitTime(long time);

          If so, should it be updateMetaWALSplitTime? And given what this patch is about, should it be WALReplay?

          Ditto for updateMetaSplitSize

          Excuse me if I am not following what is going on w/ the above (because I see later that you have replay metrics going on....)

          Default is false?

          +    distributedLogReplay = this.conf.getBoolean(HConstants.DISTRIBUTED_LOG_REPLAY_KEY, false);
          

          Should we turn it on in trunk and off in 0.95? (Should we turn it on in 0.95 so it gets a bit of testing?)

          Something wrong w/ license in WALEditsReplaySink

          Skimmed the patch. Let me come back w/ a decent review. Looks good J.

          Show
          stack added a comment - Jeffrey Zhong Nice numbers in posted doc. What does below mean sir? + // make current mutation as a distributed log replay change + protected boolean isReplay = false ; Why we have this isReplay in a Mutation? Because these edits get treated differently over on serverside? Suggest calling the data member replay or logReplay or walReplay and then the accessor is isLogReply or isWALReplay. isReplay is the name of a method that returns whether the data member replay is true or not. Does this define belong in this patch? + /** Conf key that specifies region assignment timeout value */ + public static final String REGION_ASSIGNMENT_TIME_OUT = "hbase.master.region.assignment.time.out"; Why we timing out assignments in this patch? Is this log splitting that is referred to in the metric name below? + void updateMetaSplitTime(long time); If so, should it be updateMetaWALSplitTime? And given what this patch is about, should it be WALReplay? Ditto for updateMetaSplitSize Excuse me if I am not following what is going on w/ the above (because I see later that you have replay metrics going on....) Default is false? + distributedLogReplay = this .conf.getBoolean(HConstants.DISTRIBUTED_LOG_REPLAY_KEY, false ); Should we turn it on in trunk and off in 0.95? (Should we turn it on in 0.95 so it gets a bit of testing?) Something wrong w/ license in WALEditsReplaySink Skimmed the patch. Let me come back w/ a decent review. Looks good J.
          Hide
          Anoop Sam John added a comment - - edited

          The comparison numbers looks promising! So now we make the region available for writes immediately. Have you run the test with clients doing the writes to region soon after it is opened for write? (Impact on total recovery time) Going through the patch..

          Show
          Anoop Sam John added a comment - - edited The comparison numbers looks promising! So now we make the region available for writes immediately. Have you run the test with clients doing the writes to region soon after it is opened for write? (Impact on total recovery time) Going through the patch..
          Hide
          Jeffrey Zhong added a comment -

          Thanks Stack and Anoop Sam John for reviewing!

          I included the following changes in the v1 patch:
          1) Support for recovering wal edits of regions in disabling/disabled table(Theoretically there is no need to recover wal edits of regions on a disabled table but I keep it to be compatible with before).
          2) Review feedbacks from Ted and Stack.

          From this point, I'll write more unit tests and start run integration tests.

          Below are answers to the latest feedbacks:

          Why we have this isReplay in a Mutation

          This is used inside HRegionServer#batchMutate for special handling of a reply mutation. For example, skip "readonly" check and coprocessor in the normal write path. I'll change this to "logReplay" per your suggestion. The other option is to add an addition "logReplay" argument for all functions in the write path, which isn't as clean as current way IMHO.

          Does this define belong in this patch?
          + /** Conf key that specifies region assignment timeout value */
          + public static final String REGION_ASSIGNMENT_TIME_OUT = "hbase.master.region.assignment.time.out";

          I think the name is confusing and I changed it to "hbase.master.log.replay.wait.region.timeout". It's used by logReplay to wait for a region ready before we can replay wal edits against the region.

          If so, should it be updateMetaWALSplitTime? And given what this patch is about, should it be WALReplay?

          Good point. Fixed.

          Should we turn it on in trunk and off in 0.95?

          A good suggestion to bake it in trunk a little bit.

          Something wrong w/ license in WALEditsReplaySink

          Fixed. Good catch!

          Have you run the test with clients doing the writes to region soon after it is opened for write?

          No, I haven't yet. The performance test I run is against a cluster without load to easily compare results. I'll conduct more performance tests when the feature is fully ready.

          Show
          Jeffrey Zhong added a comment - Thanks Stack and Anoop Sam John for reviewing! I included the following changes in the v1 patch: 1) Support for recovering wal edits of regions in disabling/disabled table(Theoretically there is no need to recover wal edits of regions on a disabled table but I keep it to be compatible with before). 2) Review feedbacks from Ted and Stack. From this point, I'll write more unit tests and start run integration tests. Below are answers to the latest feedbacks: Why we have this isReplay in a Mutation This is used inside HRegionServer#batchMutate for special handling of a reply mutation. For example, skip "readonly" check and coprocessor in the normal write path. I'll change this to "logReplay" per your suggestion. The other option is to add an addition "logReplay" argument for all functions in the write path, which isn't as clean as current way IMHO. Does this define belong in this patch? + /** Conf key that specifies region assignment timeout value */ + public static final String REGION_ASSIGNMENT_TIME_OUT = "hbase.master.region.assignment.time.out"; I think the name is confusing and I changed it to "hbase.master.log.replay.wait.region.timeout". It's used by logReplay to wait for a region ready before we can replay wal edits against the region. If so, should it be updateMetaWALSplitTime? And given what this patch is about, should it be WALReplay? Good point. Fixed. Should we turn it on in trunk and off in 0.95? A good suggestion to bake it in trunk a little bit. Something wrong w/ license in WALEditsReplaySink Fixed. Good catch! Have you run the test with clients doing the writes to region soon after it is opened for write? No, I haven't yet. The performance test I run is against a cluster without load to easily compare results. I'll conduct more performance tests when the feature is fully ready.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12580153/hbase-7006-combined-v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 21 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580153/hbase-7006-combined-v1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 21 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 2 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5417//console This message is automatically generated.
          Hide
          ramkrishna.s.vasudevan added a comment -

          Patch looks good on a high level. Will go through the patch

          Show
          ramkrishna.s.vasudevan added a comment - Patch looks good on a high level. Will go through the patch
          Hide
          Anoop Sam John added a comment -

          Jeffrey Zhong Do we need the metric like req count to be affected by the replay requests?

          Show
          Anoop Sam John added a comment - Jeffrey Zhong Do we need the metric like req count to be affected by the replay requests?
          Hide
          ramkrishna.s.vasudevan added a comment -

          It is more dependent on ZK now. will these exceptions cause any problem if it happens always

          catch (KeeperException e) {
          +        LOG.warn("Cannot get lastFlushedSequenceId from ZooKeeper for server=" + regionServerName
          +            + "; region=" + encodedRegionName, e);
          +      }
          
          +    } catch (KeeperException e) {
          +      LOG.warn("Cannot remove recovering regions from ZooKeeper", e);
          +    }
          
          Show
          ramkrishna.s.vasudevan added a comment - It is more dependent on ZK now. will these exceptions cause any problem if it happens always catch (KeeperException e) { + LOG.warn( "Cannot get lastFlushedSequenceId from ZooKeeper for server=" + regionServerName + + "; region=" + encodedRegionName, e); + } + } catch (KeeperException e) { + LOG.warn( "Cannot remove recovering regions from ZooKeeper" , e); + }
          Hide
          Jeffrey Zhong added a comment -

          Anoop Sam John Are you suggesting to add a req counter at the receiving RS to see how many replays is happening? I think it's a good idea. In addition, I don't see there is such counter for each individual command such as put, get, scan etc. I can add new counters for all client commands in RS. Thanks.

          Show
          Jeffrey Zhong added a comment - Anoop Sam John Are you suggesting to add a req counter at the receiving RS to see how many replays is happening? I think it's a good idea. In addition, I don't see there is such counter for each individual command such as put, get, scan etc. I can add new counters for all client commands in RS. Thanks.
          Hide
          Jeffrey Zhong added a comment -

          Hey Ram,
          Thanks for the good questions. Below are the answers:
          1)

          catch (KeeperException e) {
          +        LOG.warn("Cannot get lastFlushedSequenceId from ZooKeeper for server=" + regionServerName
          +            + "; region=" + encodedRegionName, e);
          +      }
          

          In this scenario, we can't get last flushed sequence Id so we'll replay all edits in the wal. There will be some duplicated replay while it won't affect correctness.

          +    } catch (KeeperException e) {
          +      LOG.warn("Cannot remove recovering regions from ZooKeeper", e);
          +    }
          

          We have other place to do stale data GC. Therefore, after a little bit, the recovering ZK node should be removed:
          In SplitLogManager, we have following code:

          // Garbage collect left-over /hbase/recovering-regions/... znode
          if (tot == 0 && inflightWorkItems.size() == 0 && tasks.size() == 0)

          { removeRecoveringRegionsFromZK(null); }

          -Jeffrey

          Show
          Jeffrey Zhong added a comment - Hey Ram, Thanks for the good questions. Below are the answers: 1) catch (KeeperException e) { + LOG.warn( "Cannot get lastFlushedSequenceId from ZooKeeper for server=" + regionServerName + + "; region=" + encodedRegionName, e); + } In this scenario, we can't get last flushed sequence Id so we'll replay all edits in the wal. There will be some duplicated replay while it won't affect correctness. + } catch (KeeperException e) { + LOG.warn( "Cannot remove recovering regions from ZooKeeper" , e); + } We have other place to do stale data GC. Therefore, after a little bit, the recovering ZK node should be removed: In SplitLogManager, we have following code: // Garbage collect left-over /hbase/recovering-regions/... znode if (tot == 0 && inflightWorkItems.size() == 0 && tasks.size() == 0) { removeRecoveringRegionsFromZK(null); } -Jeffrey
          Hide
          Jeffrey Zhong added a comment -

          In the v2 patch:

          1) Incorporate Anoop Sam John feedbacks by adding updateReplay histogram
          2) Add a new test case for disabling table
          3) Bug fixes relating to sequential RS failures scenario

          Thanks,
          -Jeffrey

          Show
          Jeffrey Zhong added a comment - In the v2 patch: 1) Incorporate Anoop Sam John feedbacks by adding updateReplay histogram 2) Add a new test case for disabling table 3) Bug fixes relating to sequential RS failures scenario Thanks, -Jeffrey
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12580612/hbase-7006-combined-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 21 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.backup.TestHFileArchiving
          org.apache.hadoop.hbase.security.access.TestAccessController

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580612/hbase-7006-combined-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 21 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 2 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.backup.TestHFileArchiving org.apache.hadoop.hbase.security.access.TestAccessController Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5459//console This message is automatically generated.
          Hide
          Himanshu Vashishtha added a comment -

          This has really bloated now. Can you please rb it. Thanks.

          Show
          Himanshu Vashishtha added a comment - This has really bloated now. Can you please rb it. Thanks.
          Hide
          stack added a comment -

          Jeffrey Zhong Yeah, rb it please sir (A few of us were talking about it today... we are all fired up for reviewing more!). Thanks J.

          Show
          stack added a comment - Jeffrey Zhong Yeah, rb it please sir (A few of us were talking about it today... we are all fired up for reviewing more!). Thanks J.
          Hide
          Jeffrey Zhong added a comment -

          Sure, I'll put the latest combined patch in the review board this weekend.

          Show
          Jeffrey Zhong added a comment - Sure, I'll put the latest combined patch in the review board this weekend.
          Hide
          Jeffrey Zhong added a comment -

          I posted the latest patch(v3) to review board at https://reviews.apache.org/r/10832

          In the v3 patch, it has following changes:
          1) Enable distributedLogReplay by default
          2) Bug fixes of some tricky issues from recent tests

          Thanks,
          -Jeffrey

          Show
          Jeffrey Zhong added a comment - I posted the latest patch(v3) to review board at https://reviews.apache.org/r/10832 In the v3 patch, it has following changes: 1) Enable distributedLogReplay by default 2) Bug fixes of some tricky issues from recent tests Thanks, -Jeffrey
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12580940/hbase-7006-combined-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 21 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 4 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.replication.TestReplicationQueueFailover
          org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
          org.apache.hadoop.hbase.backup.TestHFileArchiving
          org.apache.hadoop.hbase.catalog.TestMetaReaderEditor

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580940/hbase-7006-combined-v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 21 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 4 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplicationQueueFailover org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort org.apache.hadoop.hbase.backup.TestHFileArchiving org.apache.hadoop.hbase.catalog.TestMetaReaderEditor Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5482//console This message is automatically generated.
          Hide
          stack added a comment -

          Are some of the above failures because of your patch J? (Reviewing now...)

          Show
          stack added a comment - Are some of the above failures because of your patch J? (Reviewing now...)
          Hide
          Jeffrey Zhong added a comment -

          TestMetaReaderEditor is related and the other three passed locally. I'll include fixes in the next patch. Thanks.

          Show
          Jeffrey Zhong added a comment - TestMetaReaderEditor is related and the other three passed locally. I'll include fixes in the next patch. Thanks.
          Hide
          Anoop Sam John added a comment -

          Added some comments in RB. Not yet completed the review..
          Mutation.replay -> This new state varianle is needed really? For the replay we call replay interface addded in HRS from another HRS. So all the Mutations in that call are replay mutations.

          Show
          Anoop Sam John added a comment - Added some comments in RB. Not yet completed the review.. Mutation.replay -> This new state varianle is needed really? For the replay we call replay interface addded in HRS from another HRS. So all the Mutations in that call are replay mutations.
          Hide
          Jeffrey Zhong added a comment -

          Thanks Anoop Sam John for the reviewing!

          For the replay we call replay interface addded in HRS from another HRS. So all the Mutations in that call are replay mutations.

          Agree. In fact, current implementation is this way. The replay flag is NOT added into MutationProto protobuf message but in the Mutation class. So client doesn't need to specify the flag while the receiving region server set the flag so that write path code can do special logic for the replay otherwise I have to add a new 'replay' flag input argument to all functions along the write path.

          Show
          Jeffrey Zhong added a comment - Thanks Anoop Sam John for the reviewing! For the replay we call replay interface addded in HRS from another HRS. So all the Mutations in that call are replay mutations. Agree. In fact, current implementation is this way. The replay flag is NOT added into MutationProto protobuf message but in the Mutation class. So client doesn't need to specify the flag while the receiving region server set the flag so that write path code can do special logic for the replay otherwise I have to add a new 'replay' flag input argument to all functions along the write path.
          Hide
          stack added a comment -

          Some comments on the design doc:

          + Nit: Add author, date, and add issue number so can go back to the hosting issue should I trip over the doc w/o any other context.
          + Is your assumption about out-of-order replay of edits new to this feature? I suppose in the old/current way of log splitting, we do stuff in sequenceid order because we wrote the recovered.edits files named by sequenceid... so they were ordered when the regionserver read them in? We should highlight your assumption more. I think if we move to multiple-WALs we'll want to also take on this assumption doing recovery.
          + Given the assumption, we should list the problematic scenarios (or point to where we list them already – I think the 'Current Limitations' section here http://hbase.apache.org/book.html#version.delete should have the list we currently know).
          + "...check if all WALs of a failed region server have been successfully replayed." How is this done?
          + How will a crashed regionserver "...... and appending itself into the list of...": i.e. append itself to list of crashed servers (am I reading this wrong)?

          "For each region per failed region server, we stores the last flushed sequence Id from the region server before it failed."

          This is the mechanism that has the regionserver telling the master its current sequenceid everytime it flushes to an hfile? So when server crashes, master writes a znode under the recovering-regions with the last reported seq id? if a new regionserver hosting a recovery of regions then crashes, it gets a new znode w/ its current sequenceid? Now we have two crashed servers with (probably) two different sequenceids whose logs we are recovering. The two sequenceids are never related right? They are only applied to the logs of the server who passed the particular sequenceid to the master?

          Question: So it looks like we replay the WALs of a crashed regionserver by playing them into the new region host servers. There does not seem to be a flush when the replay of the old crashed servers WALs is done. Is your thinking that it is not needed since the old edits are now in the new servers WAL? Would there be any advantage NOT writing the WAL on replay and only when done, then flush (I suppose not, thinking about it, and in fact, it would probably make replay more complicated since we'd have to have this new operation to do; a flush-when-all-WALS-recovered).

          Good stuff.

          Show
          stack added a comment - Some comments on the design doc: + Nit: Add author, date, and add issue number so can go back to the hosting issue should I trip over the doc w/o any other context. + Is your assumption about out-of-order replay of edits new to this feature? I suppose in the old/current way of log splitting, we do stuff in sequenceid order because we wrote the recovered.edits files named by sequenceid... so they were ordered when the regionserver read them in? We should highlight your assumption more. I think if we move to multiple-WALs we'll want to also take on this assumption doing recovery. + Given the assumption, we should list the problematic scenarios (or point to where we list them already – I think the 'Current Limitations' section here http://hbase.apache.org/book.html#version.delete should have the list we currently know). + "...check if all WALs of a failed region server have been successfully replayed." How is this done? + How will a crashed regionserver "...... and appending itself into the list of...": i.e. append itself to list of crashed servers (am I reading this wrong)? "For each region per failed region server, we stores the last flushed sequence Id from the region server before it failed." This is the mechanism that has the regionserver telling the master its current sequenceid everytime it flushes to an hfile? So when server crashes, master writes a znode under the recovering-regions with the last reported seq id? if a new regionserver hosting a recovery of regions then crashes, it gets a new znode w/ its current sequenceid? Now we have two crashed servers with (probably) two different sequenceids whose logs we are recovering. The two sequenceids are never related right? They are only applied to the logs of the server who passed the particular sequenceid to the master? Question: So it looks like we replay the WALs of a crashed regionserver by playing them into the new region host servers. There does not seem to be a flush when the replay of the old crashed servers WALs is done. Is your thinking that it is not needed since the old edits are now in the new servers WAL? Would there be any advantage NOT writing the WAL on replay and only when done, then flush (I suppose not, thinking about it, and in fact, it would probably make replay more complicated since we'd have to have this new operation to do; a flush-when-all-WALS-recovered). Good stuff.
          Hide
          Jeffrey Zhong added a comment -

          The v4 patch includes following changes:

          1) fixes for test failures from last QA run
          2) incorporate Anoop Sam John feedbacks

          The remaining item is to see if need support HBASE-2231 where I raised some questions there.

          Show
          Jeffrey Zhong added a comment - The v4 patch includes following changes: 1) fixes for test failures from last QA run 2) incorporate Anoop Sam John feedbacks The remaining item is to see if need support HBASE-2231 where I raised some questions there.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581430/hbase-7006-combined-v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 21 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestOpenedRegionHandler
          org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581430/hbase-7006-combined-v4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 21 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.TestOpenedRegionHandler org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5527//console This message is automatically generated.
          Hide
          Jeffrey Zhong added a comment -

          Stack Good comments! Please see my responses in reverse order of your feedbacks:

          Would there be any advantage NOT writing the WAL on replay and only when done, then flush

          This is very good question. Actually I was thinking to evaluate this after this feature is in as a possible optimization. Currently receiving RS does a WAL sync for each replay batch. In the optimization scenario, we could replay mutaions with SKIP_WAL durability and flush at the end. The gain mostly depends on the "sequential" write performance of wal syncs. I think it's worth a try here.

          The two sequenceids are never related right? They are only applied to the logs of the server who passed the particular sequenceid to the master?

          No, sequenceIds from different RSs are totally un-related. Yes. Currently we use the up-to-date flushed sequence id when we open the region by looking all the store files as we do today.

          + "...check if all WALs of a failed region server have been successfully replayed." How is this done?

          We rely on the fact that when log split for a failed RS is done then all its wal files are recovered so we don't really does the check.

          + How will a crashed regionserver "...... and appending itself into the list of...": i.e. append itself to list of crashed servers (am I reading this wrong)?

          Master SSH does the work not the dead RS.

          + Is your assumption about out-of-order replay of edits new to this feature?

          Yes. I'll amend the design doc based on your other comments. Thanks.

          Show
          Jeffrey Zhong added a comment - Stack Good comments! Please see my responses in reverse order of your feedbacks: Would there be any advantage NOT writing the WAL on replay and only when done, then flush This is very good question. Actually I was thinking to evaluate this after this feature is in as a possible optimization. Currently receiving RS does a WAL sync for each replay batch. In the optimization scenario, we could replay mutaions with SKIP_WAL durability and flush at the end. The gain mostly depends on the "sequential" write performance of wal syncs. I think it's worth a try here. The two sequenceids are never related right? They are only applied to the logs of the server who passed the particular sequenceid to the master? No, sequenceIds from different RSs are totally un-related. Yes. Currently we use the up-to-date flushed sequence id when we open the region by looking all the store files as we do today. + "...check if all WALs of a failed region server have been successfully replayed." How is this done? We rely on the fact that when log split for a failed RS is done then all its wal files are recovered so we don't really does the check. + How will a crashed regionserver "...... and appending itself into the list of...": i.e. append itself to list of crashed servers (am I reading this wrong)? Master SSH does the work not the dead RS. + Is your assumption about out-of-order replay of edits new to this feature? Yes. I'll amend the design doc based on your other comments. Thanks.
          Hide
          Anoop Sam John added a comment -

          Jeffrey Zhong I also had the same question as from Stack regarding the WAL. This might be very important. Also now we will allow the writes on the recovering region when this replay is happening. These other writes + replays might be doing flushes in btw.. Any way replays alone also might be doing flushes in between(because of memstore sizes).. When this replays are in progress for some regions opened in a RS, now the replay requests from other RS taking some handlers. Whether this will affect the normal functioning of the RS? May be we can test this also IMO. The cluster is normal functioning with read,writes and then this RS down happens. So whether/how it will impact the normal read write throughput.

          Show
          Anoop Sam John added a comment - Jeffrey Zhong I also had the same question as from Stack regarding the WAL. This might be very important. Also now we will allow the writes on the recovering region when this replay is happening. These other writes + replays might be doing flushes in btw.. Any way replays alone also might be doing flushes in between(because of memstore sizes).. When this replays are in progress for some regions opened in a RS, now the replay requests from other RS taking some handlers. Whether this will affect the normal functioning of the RS? May be we can test this also IMO. The cluster is normal functioning with read,writes and then this RS down happens. So whether/how it will impact the normal read write throughput.
          Hide
          Anoop Sam John added a comment -

          Do we need a cleaner abstraction layer for RS->RS communication? May be later when we can do a HLog to region opening RS collocation (RS where the region is newly assigned only doing the HLog split) we can do stuff in this layer so as to avoid the RS connection based calls but just get the Region ref from RS and do direct writes)

          As I mentioned in some above comment when we can do the multi WAL and if we go with fixed regions for a WAL (we are infact doing virtula groups of regions in RS), we can try(max try) assigning all regions in one group to a RS and give the log splitting work for those WAL to this RS then it will be 100% locality wrt the replay commands. Sounds sensible? May be in such a case the replay can create the HFiles directly avoiding the memstore write and then flushes? (Like the bulk loading way) Some thoughts coming.. Pls correct me if I am going wrong.

          Show
          Anoop Sam John added a comment - Do we need a cleaner abstraction layer for RS->RS communication? May be later when we can do a HLog to region opening RS collocation (RS where the region is newly assigned only doing the HLog split) we can do stuff in this layer so as to avoid the RS connection based calls but just get the Region ref from RS and do direct writes) As I mentioned in some above comment when we can do the multi WAL and if we go with fixed regions for a WAL (we are infact doing virtula groups of regions in RS), we can try(max try) assigning all regions in one group to a RS and give the log splitting work for those WAL to this RS then it will be 100% locality wrt the replay commands. Sounds sensible? May be in such a case the replay can create the HFiles directly avoiding the memstore write and then flushes? (Like the bulk loading way) Some thoughts coming.. Pls correct me if I am going wrong.
          Hide
          Ted Yu added a comment -

          we can do a HLog to region opening RS collocation

          Without multi WAL, the above implies that all regions from one failed region server be assigned to one active region server. This negates the performance benefit of distributed log splitting.

          assigning all regions in one group to a RS

          I guess the underlying assumption above is that there are several region groups in multi WAL such that we gain parallelism across multiple active region servers.

          Show
          Ted Yu added a comment - we can do a HLog to region opening RS collocation Without multi WAL, the above implies that all regions from one failed region server be assigned to one active region server. This negates the performance benefit of distributed log splitting. assigning all regions in one group to a RS I guess the underlying assumption above is that there are several region groups in multi WAL such that we gain parallelism across multiple active region servers.
          Hide
          Anoop Sam John added a comment -

          Ted Yu

          Without multi WAL, the above implies that all regions from one failed region server be assigned to one active region server.

          Yes with multi WAL only.. I was just saying it for future consideration

          I guess the underlying assumption above is that there are several region groups in multi WAL

          Yes that is the assumption I have made.

          Show
          Anoop Sam John added a comment - Ted Yu Without multi WAL, the above implies that all regions from one failed region server be assigned to one active region server. Yes with multi WAL only.. I was just saying it for future consideration I guess the underlying assumption above is that there are several region groups in multi WAL Yes that is the assumption I have made.
          Hide
          Jeffrey Zhong added a comment -

          This might be very important. Also now we will allow the writes on the recovering region when this replay is happening. These other writes + replays might be doing flushes in btw.

          This is valid concern. Let's compare the new way with old way. old log splitting appends each WAL edit into a recovered.edits file while the new way flush disk only when memstore reaching certain size. Therefore, even with allowing writes during recovery, new distributed log replay still has better disk writing characteristics(assuming normal situations).
          While your concern is more relevant when a system close to its disk IO or other capacity. Allowing writes could deteriorate whole system even more. I think a system operator should rate limiting in a higher level not using recovery logic to reject traffic because nodes are expected to be down at anytime and we don't want our users get affected even a system is in recovery. Being said that, we could provide a config flag to disallow writes during recovery.

          Show
          Jeffrey Zhong added a comment - This might be very important. Also now we will allow the writes on the recovering region when this replay is happening. These other writes + replays might be doing flushes in btw. This is valid concern. Let's compare the new way with old way. old log splitting appends each WAL edit into a recovered.edits file while the new way flush disk only when memstore reaching certain size. Therefore, even with allowing writes during recovery, new distributed log replay still has better disk writing characteristics(assuming normal situations). While your concern is more relevant when a system close to its disk IO or other capacity. Allowing writes could deteriorate whole system even more. I think a system operator should rate limiting in a higher level not using recovery logic to reject traffic because nodes are expected to be down at anytime and we don't want our users get affected even a system is in recovery. Being said that, we could provide a config flag to disallow writes during recovery.
          Hide
          stack added a comment -

          Thinking on it, flushing after all logs recovered is a bad idea because it a special case. Replay mutations, as is, are treated like any other inbound edit. I think this good.

          Turning off WALs and flushing on the end and trying to figure what we failed to write or writing hfiles directly – if you could, and I don't think you can since edits need to be sorted in an hfile – and by-passing memstore and then telling the Region to pick up the new hfile when done all introduce new states that we will have to manage complicating critical recovery.

          Show
          stack added a comment - Thinking on it, flushing after all logs recovered is a bad idea because it a special case. Replay mutations, as is, are treated like any other inbound edit. I think this good. Turning off WALs and flushing on the end and trying to figure what we failed to write or writing hfiles directly – if you could, and I don't think you can since edits need to be sorted in an hfile – and by-passing memstore and then telling the Region to pick up the new hfile when done all introduce new states that we will have to manage complicating critical recovery.
          Hide
          Jeffrey Zhong added a comment -

          v4 patch incorporating Stack's comments.

          Show
          Jeffrey Zhong added a comment - v4 patch incorporating Stack's comments.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581779/hbase-7006-combined-v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581779/hbase-7006-combined-v4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5554//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581779/hbase-7006-combined-v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestDistributedLogSplitting

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581779/hbase-7006-combined-v4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.TestDistributedLogSplitting Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5563//console This message is automatically generated.
          Hide
          Jeffrey Zhong added a comment -

          Including following changes:
          1) Incorporated Stack's comments
          2) Set replayed WAL edits replication scope to null so that WAL edits created by replay command won't be double replicated.
          3) Add support for non-existence table and column family during replay

          Show
          Jeffrey Zhong added a comment - Including following changes: 1) Incorporated Stack's comments 2) Set replayed WAL edits replication scope to null so that WAL edits created by replay command won't be double replicated. 3) Add support for non-existence table and column family during replay
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12582262/hbase-7006-combined-v5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.regionserver.TestAtomicOperation
          org.apache.hadoop.hbase.security.access.TestAccessController

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582262/hbase-7006-combined-v5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 27 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestAtomicOperation org.apache.hadoop.hbase.security.access.TestAccessController Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5589//console This message is automatically generated.
          Hide
          stack added a comment -

          2) Set replayed WAL edits replication scope to null so that WAL edits created by replay command won't be double replicated.

          How can you be sure all edits in WALs from crashed server were replicated already?

          Show
          stack added a comment - 2) Set replayed WAL edits replication scope to null so that WAL edits created by replay command won't be double replicated. How can you be sure all edits in WALs from crashed server were replicated already?
          Hide
          Jeffrey Zhong added a comment -

          How can you be sure all edits in WALs from crashed server were replicated already?

          This is guaranteed by the replication fail over logic. Replication waits for log splitting finish and then resume replication on those wal files from failed RS. The above change just make sure we don't replicate WAL edits created by replay command again because those edits will be replicated from the original wal file.

          Show
          Jeffrey Zhong added a comment - How can you be sure all edits in WALs from crashed server were replicated already? This is guaranteed by the replication fail over logic. Replication waits for log splitting finish and then resume replication on those wal files from failed RS. The above change just make sure we don't replicate WAL edits created by replay command again because those edits will be replicated from the original wal file.
          Hide
          stack added a comment -

          Jeffrey Zhong Nice. Good one.

          Up on rb, you may have missed another set of reviews of mine. Thanks.

          Show
          stack added a comment - Jeffrey Zhong Nice. Good one. Up on rb, you may have missed another set of reviews of mine. Thanks.
          Hide
          Jeffrey Zhong added a comment -

          It seems that I forgot to publish it. You should have it now. Thanks.

          Show
          Jeffrey Zhong added a comment - It seems that I forgot to publish it. You should have it now. Thanks.
          Hide
          stack added a comment -

          Jeffrey Zhong Thanks.

          I asked about zxid.

          "I think you mean the zxid? That's a 64bit number where the lower
          32bits are the xid and the upper 32 bits are the epoch. The xid
          increases for each write, the epoch increases when there is a leader
          change. The zxid should always only increase. There was a bug where
          the lower 32bits could roll over, however that resulted in the epoch
          number increasing as well (64bits++) - so the constraint was
          maintained (but the cluster would fail/lockup for another issue, I
          fixed that in recent releases though...... Now
          when that is about to happen it forces a new leader election)."

          Above is from our Patrick Hunt. Says fix is in Apache ZK (3.3.5, 3.4.4).

          If you look at tail of the below issue, you will see an hbase favorite user running into rollover issue:

          https://issues.apache.org/jira/browse/ZOOKEEPER-1277

          Let me make sure we add to notes that folks should upgrade to these versions of zk.

          Show
          stack added a comment - Jeffrey Zhong Thanks. I asked about zxid. "I think you mean the zxid? That's a 64bit number where the lower 32bits are the xid and the upper 32 bits are the epoch. The xid increases for each write, the epoch increases when there is a leader change. The zxid should always only increase. There was a bug where the lower 32bits could roll over, however that resulted in the epoch number increasing as well (64bits++) - so the constraint was maintained (but the cluster would fail/lockup for another issue, I fixed that in recent releases though...... Now when that is about to happen it forces a new leader election)." Above is from our Patrick Hunt. Says fix is in Apache ZK (3.3.5, 3.4.4). If you look at tail of the below issue, you will see an hbase favorite user running into rollover issue: https://issues.apache.org/jira/browse/ZOOKEEPER-1277 Let me make sure we add to notes that folks should upgrade to these versions of zk.
          Hide
          stack added a comment -

          I added note to refguide that folks should run w/ newer zks and point to ZOOKEEPER-1277 as a justification.

          Show
          stack added a comment - I added note to refguide that folks should run w/ newer zks and point to ZOOKEEPER-1277 as a justification.
          Hide
          Jeffrey Zhong added a comment -

          The v6 patch comes with the following changes:
          1) Incorporated Ted's comments
          2) Add support to remove stale recovering region znodes from previous run during master initialization
          3) retry znode deletion
          4) bug fixes

          Show
          Jeffrey Zhong added a comment - The v6 patch comes with the following changes: 1) Incorporated Ted's comments 2) Add support to remove stale recovering region znodes from previous run during master initialization 3) retry znode deletion 4) bug fixes
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12582559/hbase-7006-combined-v6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 31 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.util.TestHBaseFsck

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582559/hbase-7006-combined-v6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 31 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.util.TestHBaseFsck Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5620//console This message is automatically generated.
          Hide
          Jeffrey Zhong added a comment -

          Incorporated Ted's latest comments.

          Show
          Jeffrey Zhong added a comment - Incorporated Ted's latest comments.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12582663/hbase-7006-combined-v7.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 31 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestHCM

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582663/hbase-7006-combined-v7.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 31 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestHCM Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5625//console This message is automatically generated.
          Hide
          stack added a comment -

          Where are we w/ this Jeffrey Zhong? What is your sense? Can you summarize. Are all the reviews up in rb addressed in this v7?

          Show
          stack added a comment - Where are we w/ this Jeffrey Zhong ? What is your sense? Can you summarize. Are all the reviews up in rb addressed in this v7?
          Hide
          Jeffrey Zhong added a comment -

          V8 patch: Rebased and one line change add "checkOpen()" into replay command(I found "multi, mutate and multiGet" all missing checkOpen call. I'll file a separate JIRA on this).

          Summary on the JIRA:

          All review comments are addressed. Unit tests passed and no issue is found so far from integration test IntegrationTestDataIngestWithChaosMonkey(will keeping running for a while).

          I think we can check it in now.

          Next steps are:

          1) Re-run performance tests to avoid any performance degradation from recent changes.
          2) Add configuration to "disallow writes during recovery", which I intentionally leave it out from the combined patch.

          Thanks.

          Show
          Jeffrey Zhong added a comment - V8 patch: Rebased and one line change add "checkOpen()" into replay command(I found "multi, mutate and multiGet" all missing checkOpen call. I'll file a separate JIRA on this). Summary on the JIRA: All review comments are addressed. Unit tests passed and no issue is found so far from integration test IntegrationTestDataIngestWithChaosMonkey(will keeping running for a while). I think we can check it in now. Next steps are: 1) Re-run performance tests to avoid any performance degradation from recent changes. 2) Add configuration to "disallow writes during recovery", which I intentionally leave it out from the combined patch. Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12582743/hbase-7006-combined-v8.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 31 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582743/hbase-7006-combined-v8.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 31 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5635//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          I ran test suite through patch v6 twice, through v8 once.
          Result was good.

          TestHBaseFsck failed once in v8 run but wasn't reproduced.

          Show
          Ted Yu added a comment - I ran test suite through patch v6 twice, through v8 once. Result was good. TestHBaseFsck failed once in v8 run but wasn't reproduced.
          Hide
          stack added a comment -

          Jeffrey Zhong The 'next steps' are to be done in other issues, not on this one? Have you run the IntegrationTestBigLinkedList with this in place killing servers as it runs? I am game for checking it in. It has a good bit of review and if we have not found egregious error by now, then we get what we deserve. Nice one.

          Show
          stack added a comment - Jeffrey Zhong The 'next steps' are to be done in other issues, not on this one? Have you run the IntegrationTestBigLinkedList with this in place killing servers as it runs? I am game for checking it in. It has a good bit of review and if we have not found egregious error by now, then we get what we deserve. Nice one.
          Hide
          Jeffrey Zhong added a comment -

          v9 patch is a rebase patch. I've run the integration test IntegrationTestDataIngestWithChaosMonkey which does data ingestion with verification and randomly kill a region server/master periodically.

          I think we can check the v9 patch in. Meanwhile I'll run more integration tests.

          Show
          Jeffrey Zhong added a comment - v9 patch is a rebase patch. I've run the integration test IntegrationTestDataIngestWithChaosMonkey which does data ingestion with verification and randomly kill a region server/master periodically. I think we can check the v9 patch in. Meanwhile I'll run more integration tests.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12583262/hbase-7006-combined-v9.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 34 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12583262/hbase-7006-combined-v9.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 34 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5693//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          +1 on patch v9.

          Show
          Ted Yu added a comment - +1 on patch v9.
          Hide
          stack added a comment -

          Committed to 0.95 and trunk. Thanks for the nice feature Jeffrey. Nice work.

          Please add a fat release note on what this facility adds and do some cleanup of the old issues outstanding.

          Show
          stack added a comment - Committed to 0.95 and trunk. Thanks for the nice feature Jeffrey. Nice work. Please add a fat release note on what this facility adds and do some cleanup of the old issues outstanding.
          Hide
          Jeffrey Zhong added a comment -

          Stack Thanks! I'll keep working on the follow up items and add a release note accordingly.

          Show
          Jeffrey Zhong added a comment - Stack Thanks! I'll keep working on the follow up items and add a release note accordingly.
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95 #194 (See https://builds.apache.org/job/hbase-0.95/194/)
          HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay (Revision 1482676)

          Result = SUCCESS
          stack :
          Files :

          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RegionInRecoveryException.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/branches/0.95/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSource.java
          • /hbase/branches/0.95/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
          • /hbase/branches/0.95/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySource.java
          • /hbase/branches/0.95/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java
          • /hbase/branches/0.95/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
          • /hbase/branches/0.95/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java
          • /hbase/branches/0.95/hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource
          • /hbase/branches/0.95/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java
          • /hbase/branches/0.95/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
          • /hbase/branches/0.95/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java
          • /hbase/branches/0.95/hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource
          • /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java
          • /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
          • /hbase/branches/0.95/hbase-protocol/src/main/protobuf/Admin.proto
          • /hbase/branches/0.95/hbase-protocol/src/main/protobuf/hbase.proto
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMaster.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALEditsReplay.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoveringRegionWatcher.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenMasterInitializing.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
          Show
          Hudson added a comment - Integrated in hbase-0.95 #194 (See https://builds.apache.org/job/hbase-0.95/194/ ) HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay (Revision 1482676) Result = SUCCESS stack : Files : /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RegionInRecoveryException.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/branches/0.95/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSource.java /hbase/branches/0.95/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java /hbase/branches/0.95/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySource.java /hbase/branches/0.95/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java /hbase/branches/0.95/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java /hbase/branches/0.95/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java /hbase/branches/0.95/hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource /hbase/branches/0.95/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java /hbase/branches/0.95/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java /hbase/branches/0.95/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java /hbase/branches/0.95/hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java /hbase/branches/0.95/hbase-protocol/src/main/protobuf/Admin.proto /hbase/branches/0.95/hbase-protocol/src/main/protobuf/hbase.proto /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMaster.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALEditsReplay.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoveringRegionWatcher.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenMasterInitializing.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #4119 (See https://builds.apache.org/job/HBase-TRUNK/4119/)
          HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay (Revision 1482675)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RegionInRecoveryException.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSource.java
          • /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
          • /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySource.java
          • /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java
          • /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
          • /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java
          • /hbase/trunk/hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource
          • /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java
          • /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
          • /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java
          • /hbase/trunk/hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource
          • /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java
          • /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
          • /hbase/trunk/hbase-protocol/src/main/protobuf/Admin.proto
          • /hbase/trunk/hbase-protocol/src/main/protobuf/hbase.proto
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMaster.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALEditsReplay.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoveringRegionWatcher.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenMasterInitializing.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #4119 (See https://builds.apache.org/job/HBase-TRUNK/4119/ ) HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay (Revision 1482675) Result = FAILURE stack : Files : /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RegionInRecoveryException.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSource.java /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySource.java /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java /hbase/trunk/hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java /hbase/trunk/hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java /hbase/trunk/hbase-protocol/src/main/protobuf/Admin.proto /hbase/trunk/hbase-protocol/src/main/protobuf/hbase.proto /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALEditsReplay.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoveringRegionWatcher.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenMasterInitializing.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95-on-hadoop2 #100 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/100/)
          HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay (Revision 1482676)

          Result = FAILURE
          stack :
          Files :

          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RegionInRecoveryException.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/branches/0.95/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSource.java
          • /hbase/branches/0.95/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
          • /hbase/branches/0.95/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySource.java
          • /hbase/branches/0.95/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java
          • /hbase/branches/0.95/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
          • /hbase/branches/0.95/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java
          • /hbase/branches/0.95/hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource
          • /hbase/branches/0.95/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java
          • /hbase/branches/0.95/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
          • /hbase/branches/0.95/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java
          • /hbase/branches/0.95/hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource
          • /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java
          • /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
          • /hbase/branches/0.95/hbase-protocol/src/main/protobuf/Admin.proto
          • /hbase/branches/0.95/hbase-protocol/src/main/protobuf/hbase.proto
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMaster.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALEditsReplay.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoveringRegionWatcher.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenMasterInitializing.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
          Show
          Hudson added a comment - Integrated in hbase-0.95-on-hadoop2 #100 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/100/ ) HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay (Revision 1482676) Result = FAILURE stack : Files : /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RegionInRecoveryException.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/branches/0.95/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSource.java /hbase/branches/0.95/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java /hbase/branches/0.95/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySource.java /hbase/branches/0.95/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java /hbase/branches/0.95/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java /hbase/branches/0.95/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java /hbase/branches/0.95/hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource /hbase/branches/0.95/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java /hbase/branches/0.95/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java /hbase/branches/0.95/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java /hbase/branches/0.95/hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java /hbase/branches/0.95/hbase-protocol/src/main/protobuf/Admin.proto /hbase/branches/0.95/hbase-protocol/src/main/protobuf/hbase.proto /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMaster.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALEditsReplay.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoveringRegionWatcher.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenMasterInitializing.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #531 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/531/)
          HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay (Revision 1482675)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RegionInRecoveryException.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSource.java
          • /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
          • /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySource.java
          • /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java
          • /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
          • /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java
          • /hbase/trunk/hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource
          • /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java
          • /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
          • /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java
          • /hbase/trunk/hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource
          • /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java
          • /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
          • /hbase/trunk/hbase-protocol/src/main/protobuf/Admin.proto
          • /hbase/trunk/hbase-protocol/src/main/protobuf/hbase.proto
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMaster.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALEditsReplay.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoveringRegionWatcher.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenMasterInitializing.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #531 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/531/ ) HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay (Revision 1482675) Result = FAILURE stack : Files : /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RegionInRecoveryException.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSource.java /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySource.java /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java /hbase/trunk/hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterSourceImpl.java /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySourceImpl.java /hbase/trunk/hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.regionserver.wal.MetricsEditsReplaySource /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java /hbase/trunk/hbase-protocol/src/main/protobuf/Admin.proto /hbase/trunk/hbase-protocol/src/main/protobuf/hbase.proto /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALEditsReplay.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoveringRegionWatcher.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenMasterInitializing.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
          Show
          Jeffrey Zhong added a comment - Ted Yu Attached a NPE fix for the issue that Ted found an intermittent failure http://54.241.6.143/job/HBase-0.95/org.apache.hbase$hbase-server/269/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testWorkerAbort/ . Thanks.
          Hide
          stack added a comment -

          I applied the addendum to trunk and 0.95. Thanks lads.

          Show
          stack added a comment - I applied the addendum to trunk and 0.95. Thanks lads.
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95 #196 (See https://builds.apache.org/job/hbase-0.95/196/)
          HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay; ADDENDUM (Revision 1483009)

          Result = FAILURE
          stack :
          Files :

          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
          Show
          Hudson added a comment - Integrated in hbase-0.95 #196 (See https://builds.apache.org/job/hbase-0.95/196/ ) HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay; ADDENDUM (Revision 1483009) Result = FAILURE stack : Files : /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #4122 (See https://builds.apache.org/job/HBase-TRUNK/4122/)
          HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay; ADDENDUM (Revision 1483011)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #4122 (See https://builds.apache.org/job/HBase-TRUNK/4122/ ) HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay; ADDENDUM (Revision 1483011) Result = FAILURE stack : Files : /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #532 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/532/)
          HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay; ADDENDUM (Revision 1483011)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #532 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/532/ ) HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay; ADDENDUM (Revision 1483011) Result = FAILURE stack : Files : /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95-on-hadoop2 #101 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/101/)
          HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay; ADDENDUM (Revision 1483009)

          Result = FAILURE
          stack :
          Files :

          • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
          Show
          Hudson added a comment - Integrated in hbase-0.95-on-hadoop2 #101 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/101/ ) HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay; ADDENDUM (Revision 1483009) Result = FAILURE stack : Files : /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
          Hide
          rajeshbabu added a comment -

          HMaster changes broken TestMasterShutdown.testMasterShutdownBeforeStartingAnyRegionServer

          Show
          rajeshbabu added a comment - HMaster changes broken TestMasterShutdown.testMasterShutdownBeforeStartingAnyRegionServer
          Hide
          rajeshbabu added a comment -

          Jeffrey Zhong
          Is it fine to continue master initialization even .META. region not assigned?

          Show
          rajeshbabu added a comment - Jeffrey Zhong Is it fine to continue master initialization even .META. region not assigned?
          Hide
          Jeffrey Zhong added a comment -

          rajeshbabu After I reset my build to the point right after 7006 check in using "git reset --hard a3e0004e26a68cf58031330a477344b225aaf901", I see the test pass. I guess some checkins after my patch caused the test failure. I'll debug a little bit more to see what caused the issue.

          Show
          Jeffrey Zhong added a comment - rajeshbabu After I reset my build to the point right after 7006 check in using "git reset --hard a3e0004e26a68cf58031330a477344b225aaf901", I see the test pass. I guess some checkins after my patch caused the test failure. I'll debug a little bit more to see what caused the issue.
          Hide
          rajeshbabu added a comment -

          Jeffrey Zhong
          Earlier If META assignment failed then we will return from initialization

          -    if (!assignMeta(status)) return;
          

          In side assignMeta

                 // Make sure a .META. location is set.
          -      if (!isMetaLocation()) return false;
          -      // This guarantees that the transition assigning .META. has completed
          -      this.assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
          

          But now in this patch the logic got changed and even if META not assigned also initialization getting continued and hanging at .META. migration while locating meta from zk.

              // Make sure meta assigned before proceeding.
              status.setStatus("Assigning Meta Region");
              assignMeta(status);
          
          Show
          rajeshbabu added a comment - Jeffrey Zhong Earlier If META assignment failed then we will return from initialization - if (!assignMeta(status)) return ; In side assignMeta // Make sure a .META. location is set. - if (!isMetaLocation()) return false ; - // This guarantees that the transition assigning .META. has completed - this .assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO); But now in this patch the logic got changed and even if META not assigned also initialization getting continued and hanging at .META. migration while locating meta from zk. // Make sure meta assigned before proceeding. status.setStatus( "Assigning Meta Region" ); assignMeta(status);
          Hide
          Jeffrey Zhong added a comment -

          rajeshbabu That's a bug before. Though we return false to break the initialization but there is no return from caller finishInitialization so we continue even master isn't initialized fully. The change is expecting assignMeta to throw exception to stop master starts up.

                  finishInitialization(startupStatus, false);
                  loop();
          
          Show
          Jeffrey Zhong added a comment - rajeshbabu That's a bug before. Though we return false to break the initialization but there is no return from caller finishInitialization so we continue even master isn't initialized fully. The change is expecting assignMeta to throw exception to stop master starts up. finishInitialization(startupStatus, false ); loop();
          Hide
          rajeshbabu added a comment -

          if any failures during .META. assignment other than master shutdown/stop we are infinitely waiting until .META. is assigned.

              enableServerShutdownHandler();
              this.catalogTracker.waitForMeta();
              // Above check waits for general meta availability but this does not
              // guarantee that the transition has completed
              this.assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
          

          Only chance we will come out from initialization before assigning META is master shutdown/stop.
          then any way we will not block in loop as well.

            private void loop() {
              long lastMsgTs = 0l;
              long now = 0l;
              while (!this.stopped) {
                now = System.currentTimeMillis();
                if ((now - lastMsgTs) >= this.msgInterval) {
                  doMetrics();
                  lastMsgTs = System.currentTimeMillis();
                }
                stopSleeper.sleep();
              }
            }
          
          Show
          rajeshbabu added a comment - if any failures during .META. assignment other than master shutdown/stop we are infinitely waiting until .META. is assigned. enableServerShutdownHandler(); this .catalogTracker.waitForMeta(); // Above check waits for general meta availability but this does not // guarantee that the transition has completed this .assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO); Only chance we will come out from initialization before assigning META is master shutdown/stop. then any way we will not block in loop as well. private void loop() { long lastMsgTs = 0l; long now = 0l; while (! this .stopped) { now = System .currentTimeMillis(); if ((now - lastMsgTs) >= this .msgInterval) { doMetrics(); lastMsgTs = System .currentTimeMillis(); } stopSleeper.sleep(); } }
          Hide
          stack added a comment -

          I'm looking at TestMasterShutdown too... lets do it over in new issue HBASE-8560

          Show
          stack added a comment - I'm looking at TestMasterShutdown too... lets do it over in new issue HBASE-8560
          Hide
          rajeshbabu added a comment -

          Jeffrey Zhong
          Throwing exception also fine too..its not a problem...

          Show
          rajeshbabu added a comment - Jeffrey Zhong Throwing exception also fine too..its not a problem...
          Hide
          Jeffrey Zhong added a comment -

          rajeshbabu Thanks for the good catch. It turns out my refactoring missed to handle the master shutting down case. I submitted a patch under hbase-8560.

          Show
          Jeffrey Zhong added a comment - rajeshbabu Thanks for the good catch. It turns out my refactoring missed to handle the master shutting down case. I submitted a patch under hbase-8560.
          Hide
          chunhui shen added a comment -

          Great work!

          One question:
          From the doc, I see "last flushed sequence id of the region is stored in ZK"
          I think we should store the last flushed sequence id for each store of region, otherwise would cause the problem of data correctness when replaying logs.

          Show
          chunhui shen added a comment - Great work! One question: From the doc, I see "last flushed sequence id of the region is stored in ZK" I think we should store the last flushed sequence id for each store of region, otherwise would cause the problem of data correctness when replaying logs.
          Hide
          Jeffrey Zhong added a comment -

          chunhui shen

          I think we should store the last flushed sequence id for each store of region, otherwise would cause the problem of data correctness when replaying logs.

          You're right. Actually yesterday Enis Soztutar showed me HBASE-6059, I realized that we need to do the above as you suggested. I considered that before and thought storing one id per region is enough which turns out not true. I'll fix that as a follow up issue. Thanks for mentioning this!

          Show
          Jeffrey Zhong added a comment - chunhui shen I think we should store the last flushed sequence id for each store of region, otherwise would cause the problem of data correctness when replaying logs. You're right. Actually yesterday Enis Soztutar showed me HBASE-6059 , I realized that we need to do the above as you suggested. I considered that before and thought storing one id per region is enough which turns out not true. I'll fix that as a follow up issue. Thanks for mentioning this!
          Hide
          ramkrishna.s.vasudevan added a comment -

          Good one Chunhui.

          Show
          ramkrishna.s.vasudevan added a comment - Good one Chunhui.
          Hide
          Ted Yu added a comment -

          I created HBASE-8573 for storing last flushed sequence id for each store of region in zookeeper.

          Show
          Ted Yu added a comment - I created HBASE-8573 for storing last flushed sequence id for each store of region in zookeeper.
          Hide
          Ted Yu added a comment -

          Before HBASE-8573 is resolved, I think we should change the following to false:

          +  public static final boolean DEFAULT_DISTRIBUTED_LOG_REPLAY_CONFIG = true;
          
          Show
          Ted Yu added a comment - Before HBASE-8573 is resolved, I think we should change the following to false: + public static final boolean DEFAULT_DISTRIBUTED_LOG_REPLAY_CONFIG = true ;
          Hide
          Ted Yu added a comment -

          Addendum 2 turns off distributed log replay by default.

          Show
          Ted Yu added a comment - Addendum 2 turns off distributed log replay by default.
          Hide
          stack added a comment -

          chunhui shen Thanks for keeping an eye out.

          +1 on addendum for now (adjust release note to say this is off for now)

          Show
          stack added a comment - chunhui shen Thanks for keeping an eye out. +1 on addendum for now (adjust release note to say this is off for now)
          Hide
          Jeffrey Zhong added a comment -

          Looks good to me(+1). Thanks.

          Show
          Jeffrey Zhong added a comment - Looks good to me(+1). Thanks.
          Hide
          Ted Yu added a comment -

          I got the following test failure:

          testHBaseFsck(org.apache.hadoop.hbase.util.TestHBaseFsck)  Time elapsed: 0 sec  <<< FAILURE!
          java.lang.AssertionError: expected:<[]> but was:<[EXPIRED_TABLE_LOCK]>
                  at org.junit.Assert.fail(Assert.java:88)
                  at org.junit.Assert.failNotEquals(Assert.java:743)
                  at org.junit.Assert.assertEquals(Assert.java:118)
                  at org.junit.Assert.assertEquals(Assert.java:144)
                  at org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.assertNoErrors(HbckTestingUtil.java:85)
                  at org.apache.hadoop.hbase.util.TestHBaseFsck.testHBaseFsck(TestHBaseFsck.java:146)
          

          I think the above was not related to addendum.

          Show
          Ted Yu added a comment - I got the following test failure: testHBaseFsck(org.apache.hadoop.hbase.util.TestHBaseFsck) Time elapsed: 0 sec <<< FAILURE! java.lang.AssertionError: expected:<[]> but was:<[EXPIRED_TABLE_LOCK]> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.assertNoErrors(HbckTestingUtil.java:85) at org.apache.hadoop.hbase.util.TestHBaseFsck.testHBaseFsck(TestHBaseFsck.java:146) I think the above was not related to addendum.
          Hide
          Ted Yu added a comment -

          Addendum integrated to 0.95 and trunk.

          Thanks for the reviews, Stack and Jeff.

          Show
          Ted Yu added a comment - Addendum integrated to 0.95 and trunk. Thanks for the reviews, Stack and Jeff.
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95 #202 (See https://builds.apache.org/job/hbase-0.95/202/)
          HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay, Addendum disables the feature by default (Revision 1484021)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java
          Show
          Hudson added a comment - Integrated in hbase-0.95 #202 (See https://builds.apache.org/job/hbase-0.95/202/ ) HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay, Addendum disables the feature by default (Revision 1484021) Result = FAILURE tedyu : Files : /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #4128 (See https://builds.apache.org/job/HBase-TRUNK/4128/)
          HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay, Addendum disables the feature by default (Revision 1484022)

          Result = SUCCESS
          tedyu :
          Files :

          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #4128 (See https://builds.apache.org/job/HBase-TRUNK/4128/ ) HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay, Addendum disables the feature by default (Revision 1484022) Result = SUCCESS tedyu : Files : /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95-on-hadoop2 #103 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/103/)
          HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay, Addendum disables the feature by default (Revision 1484021)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java
          Show
          Hudson added a comment - Integrated in hbase-0.95-on-hadoop2 #103 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/103/ ) HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay, Addendum disables the feature by default (Revision 1484021) Result = FAILURE tedyu : Files : /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #534 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/534/)
          HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay, Addendum disables the feature by default (Revision 1484022)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #534 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/534/ ) HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay, Addendum disables the feature by default (Revision 1484022) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java
          Hide
          Jeffrey Zhong added a comment -

          I conducted a performance test against trunk code + hbase-8680 and hbase-8666. The goal is to verify there is no unexpected performance regression. In the tests, 44 WAL files are recovered in about total 274M size on a 5 EC2 m1.large nodes environment(1 master, 4 region servers).

          A recap of the two recovering mode in comparison:

          1) recovered edits files creation recovery includes following steps:

          • creating recovered edits files per wal per region
          • re-assign failed regions
          • replay recovered edits files during post region open task
          • region are opened for writes & reads

          2) logReplay contains following steps:

          • re-assign failed regions
          • regions are opened for writes
          • replay wal edits directly to assigned regions
          • regions are opened for reads
          Log Splitting with creation of recovered edits files
          Number of Regions in each WAL file Log Splitting Elapsed Time(ms) Recovered Edits Replay Elapsed Time(ms)* Recovery Time For Reads(ms) Recovery Time For Writes(ms)
          80 201263 100631 301894 301894
          160 256325 128160 384485 384485
          320 283514 141757 425271 425271
          • During tests I didn't capture the recovered edits file replay(need to use bulk assign sync mode) accurately so I based on previous runs and used 0.5 * splitting time as an estimate
          Distributed Log Replay
          Number of Regions in each WAL file Log Splitting Elapsed Time(ms) Recovered Edits Replay Elapsed Time(ms)* Recovery Time For Reads(ms) Recovery Time For Writes*(ms)
          80 80953 n/a 80953 < 5000
          160 91656 n/a 91656 < 5000
          320 95053 n/a 95053 < 5000
          • Due to the same above reason that I didn't use bulk assign sync mode so I didn't get the accurate numbers. Opening region normally takes about several secs when there is no recovered edits file replay.

          The cluster I used seems have slow disk throughput while it's good enough to make sure that distributedLogReplay has better performance than recovered edits creation mode after checked into trunk.

          Show
          Jeffrey Zhong added a comment - I conducted a performance test against trunk code + hbase-8680 and hbase-8666. The goal is to verify there is no unexpected performance regression. In the tests, 44 WAL files are recovered in about total 274M size on a 5 EC2 m1.large nodes environment(1 master, 4 region servers). A recap of the two recovering mode in comparison: 1) recovered edits files creation recovery includes following steps: creating recovered edits files per wal per region re-assign failed regions replay recovered edits files during post region open task region are opened for writes & reads 2) logReplay contains following steps: re-assign failed regions regions are opened for writes replay wal edits directly to assigned regions regions are opened for reads Log Splitting with creation of recovered edits files Number of Regions in each WAL file Log Splitting Elapsed Time(ms) Recovered Edits Replay Elapsed Time(ms)* Recovery Time For Reads(ms) Recovery Time For Writes(ms) 80 201263 100631 301894 301894 160 256325 128160 384485 384485 320 283514 141757 425271 425271 During tests I didn't capture the recovered edits file replay(need to use bulk assign sync mode) accurately so I based on previous runs and used 0.5 * splitting time as an estimate Distributed Log Replay Number of Regions in each WAL file Log Splitting Elapsed Time(ms) Recovered Edits Replay Elapsed Time(ms)* Recovery Time For Reads(ms) Recovery Time For Writes*(ms) 80 80953 n/a 80953 < 5000 160 91656 n/a 91656 < 5000 320 95053 n/a 95053 < 5000 Due to the same above reason that I didn't use bulk assign sync mode so I didn't get the accurate numbers. Opening region normally takes about several secs when there is no recovered edits file replay. The cluster I used seems have slow disk throughput while it's good enough to make sure that distributedLogReplay has better performance than recovered edits creation mode after checked into trunk.
          Hide
          Elliott Clark added a comment - - edited

          So talking about this patch with Himanshu and I had a few concerns.

          Concern One

          I'm pretty sure there is an issue with opening a region for edits before all logs are finished replaying. To illustrate:

          Say there's a table with cf that has VERSIONS = 2.
          For all edits the rowkey is the same.

          1. There's a log with: [ A (ts = 0), B (ts = 0) ]
          2. Replay the first half of the log.
          3. A user puts in C (ts = 0)
          4. Memstore has to flush
          5. A new Hfile will be created with [ C, A ] and MaxSequenceId = C's seqid.
          6. Replay the rest of the Log.
          7. Flush

          You'll get either C, A when a get is issued.

          C, B is the expected result.

          We have promised that edits will be ordered by timestamp, then sequence id.

          Concern Two

          I think there's an issue with duplicating edits if there is a failure while replaying. To illustrate:

          Say there's a table with a column family with Versions = 3

          1. There's a log with edits who's timestamps are [ 10, 11, 12 ]
          2. assign the region for replay
          3. Start replaying
          4. Fail after [ 10, 11 ]
          5. Now there are two logs [ 10, 11, 12] [ 10, 11 ]
          6. Master sees that replaying failed and that the rs hosting the region failed.
          7. it will replay both logs.
          8. You will now have [ 12, 11, 11 ]

          Any get to that table will get [ 12, 11, 11]

          [12, 11, 10] is expected.

          This is fixable if we:

          1. Don't replay wal edits with isReaply = true
          2. and only remove old logs after all the memstores that the log got replayed into have fully flushed.

          This is hard since the memstores are all over and hard to keep track of.

          or:

          1. Don't append the replayed edits to the wal
          2. while replaying if the memstore needs to flush, flush the hfiles out to a temp location.
          3. Move the Hfiles in after all the edits are recovered.

          This is hard as we'll have to meddle with how we flush the memstore.

          Show
          Elliott Clark added a comment - - edited So talking about this patch with Himanshu and I had a few concerns. Concern One I'm pretty sure there is an issue with opening a region for edits before all logs are finished replaying. To illustrate: Say there's a table with cf that has VERSIONS = 2. For all edits the rowkey is the same. There's a log with: [ A (ts = 0), B (ts = 0) ] Replay the first half of the log. A user puts in C (ts = 0) Memstore has to flush A new Hfile will be created with [ C, A ] and MaxSequenceId = C's seqid. Replay the rest of the Log. Flush You'll get either C, A when a get is issued. C, B is the expected result. We have promised that edits will be ordered by timestamp, then sequence id. Concern Two I think there's an issue with duplicating edits if there is a failure while replaying. To illustrate: Say there's a table with a column family with Versions = 3 There's a log with edits who's timestamps are [ 10, 11, 12 ] assign the region for replay Start replaying Fail after [ 10, 11 ] Now there are two logs [ 10, 11, 12] [ 10, 11 ] Master sees that replaying failed and that the rs hosting the region failed. it will replay both logs. You will now have [ 12, 11, 11 ] Any get to that table will get [ 12, 11, 11] [12, 11, 10] is expected. This is fixable if we: Don't replay wal edits with isReaply = true and only remove old logs after all the memstores that the log got replayed into have fully flushed. This is hard since the memstores are all over and hard to keep track of. or: Don't append the replayed edits to the wal while replaying if the memstore needs to flush, flush the hfiles out to a temp location. Move the Hfiles in after all the edits are recovered. This is hard as we'll have to meddle with how we flush the memstore.
          Hide
          Ted Yu added a comment -

          for concern #1, step 2 (log replay) implies the crash of original region server.
          In step 3, how would C carry timestamp 0 which is the same as A and B ?

          Show
          Ted Yu added a comment - for concern #1, step 2 (log replay) implies the crash of original region server. In step 3, how would C carry timestamp 0 which is the same as A and B ?
          Hide
          Jeffrey Zhong added a comment -

          Thanks for the concerns!

          Since a row is keyed by row, column family, column qualifier and timestamp. If we send the same put like the following repeatedly

          put 't3', 'row1', 'test_cf:c1','1', 11
          

          scan will only return one row. I tested with the following raw scan

          scan 't3', {RAW=>true, VERSIONS=>100}
          

          The test steps I'm using are:
          1)

          create 't3', {NAME => 'test_cf', VERSIONS => 5}

          2)

          put 't3', 'row1', 'test_cf:c1','1', 10

          3)

          put 't3', 'row1', 'test_cf:c1','1', 11 (10 times)

          4)

          scan 't3', {RAW=>true, VERSIONS=>100}

          I only got:
          row1 column=test_cf:c1, timestamp=11, value=1
          row1 column=test_cf:c1, timestamp=10, value=1

          For your first concern, I think the result should be either B or C.

          Show
          Jeffrey Zhong added a comment - Thanks for the concerns! Since a row is keyed by row, column family, column qualifier and timestamp. If we send the same put like the following repeatedly put 't3', 'row1', 'test_cf:c1','1', 11 scan will only return one row. I tested with the following raw scan scan 't3', {RAW=> true , VERSIONS=>100} The test steps I'm using are: 1) create 't3', {NAME => 'test_cf', VERSIONS => 5} 2) put 't3', 'row1', 'test_cf:c1','1', 10 3) put 't3', 'row1', 'test_cf:c1','1', 11 (10 times) 4) scan 't3', {RAW=> true , VERSIONS=>100} I only got: row1 column=test_cf:c1, timestamp=11, value=1 row1 column=test_cf:c1, timestamp=10, value=1 For your first concern, I think the result should be either B or C.
          Hide
          Elliott Clark added a comment -

          In step 3, how would C carry timestamp 0 which is the same as A and B ?

          You can specify the timestamp on put's. 0 isn't special here. It could be any timestamp that's the same.

          So I bring up the seqId as a concern because of:

          Things in the memstore may be upserted (Probably a bug that should be doc'd or addressed). But sequence Id sorting is the reason that compaction must always choose contiguous files. If that changes then the compaction algorithm can be very different from what it currently is.

          Show
          Elliott Clark added a comment - In step 3, how would C carry timestamp 0 which is the same as A and B ? You can specify the timestamp on put's. 0 isn't special here. It could be any timestamp that's the same. So I bring up the seqId as a concern because of: http://archive.apache.org/dist/hbase/docs/versions.html#d397e2960 "To overwrite an existing value, do a put at exactly the same row, column, and version as that of the cell you would overshadow." https://issues.apache.org/jira/browse/HBASE-7763?focusedCommentId=13572592&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13572592 http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/KeyValueHeap.html#176 Things in the memstore may be upserted (Probably a bug that should be doc'd or addressed). But sequence Id sorting is the reason that compaction must always choose contiguous files. If that changes then the compaction algorithm can be very different from what it currently is.
          Hide
          stack added a comment -

          On 'Concern One', we would get back B. The B would be in memstore when region was flipped readable. memstore gets precedence over hfiles. We should get back C though. Even if we held up flushes until the region were flipped to readable, we'd have a problem (because B coming in later would overwrite C). But Jeffrey, don't you have replayed edits go into a different memstore? Or if you don't, maybe they should... and this other memstore should sort after the first memstore? You could flush the memstore of replayed edits but not the memstore of new edits, not until the region had flipped readable?

          (Good find E and H).

          On concern two, what Jeffrey says above, we suppress versions of same timestamp so only the latest version of a timestamp returned so we'd get 12, 11, 10.

          Show
          stack added a comment - On 'Concern One', we would get back B. The B would be in memstore when region was flipped readable. memstore gets precedence over hfiles. We should get back C though. Even if we held up flushes until the region were flipped to readable, we'd have a problem (because B coming in later would overwrite C). But Jeffrey, don't you have replayed edits go into a different memstore? Or if you don't, maybe they should... and this other memstore should sort after the first memstore? You could flush the memstore of replayed edits but not the memstore of new edits, not until the region had flipped readable? (Good find E and H). On concern two, what Jeffrey says above, we suppress versions of same timestamp so only the latest version of a timestamp returned so we'd get 12, 11, 10.
          Hide
          Jeffrey Zhong added a comment -

          I think about the issue whole morning. Also I discussed this with other folks. Basically the root issue is to maintain the receiving order during recovery for puts with exact same key + version(timestamp). Since log recovery process could work on multiple wal files at same time, the order of replay isn't guaranteed to be in the receiving order. I'm listing several options below to see how others think.

          Option one(the simplest one)

          Document this limitation in the release note. Assuming the same version update is a rare usage pattern.

          Option two(still simple but hacky)

          a) disallow writes during recovery
          b) hold flush till all wals of a recovering region are replayed. Memstore should hold because we only recover unflushed wal edits.

          Option three(multiple memstores)

          a) Let splitlogworker pick wals of a failed RS in order instead of random. Say a failed RS has WAL1, WAL2, WAL3,... WALk. a worker will only pick WAL2 if WAL1 is done(or errored) etc.
          b) During replay, we pass original wal sequence ids of edits to the receiving RS
          c) In receiving RS, we bucket WAL files to a different memstore during replaying and use the original sequence Ids. Say wal1-wal4 to memstore1, wal5-wal10 to memstore2 etc. We only flush the bucket memstore when all wals inside the bucket are replayed. all wals can be replayed concurrently.
          d) writes from normal traffic(allow writes during recovery) are put in a different memstore as of today and flush normally with new sequenceIds.

          Option four

          a) During replay, we pass original wal sequence ids
          b) for each wal edit, we store each edit's sequence id along with its key.
          c) during scanning, we use the original sequence id if it's present otherwise its store file sequence Id
          d) compaction can just leave put with max sequence id

          Show
          Jeffrey Zhong added a comment - I think about the issue whole morning. Also I discussed this with other folks. Basically the root issue is to maintain the receiving order during recovery for puts with exact same key + version(timestamp). Since log recovery process could work on multiple wal files at same time, the order of replay isn't guaranteed to be in the receiving order. I'm listing several options below to see how others think. Option one(the simplest one) Document this limitation in the release note. Assuming the same version update is a rare usage pattern. Option two(still simple but hacky) a) disallow writes during recovery b) hold flush till all wals of a recovering region are replayed. Memstore should hold because we only recover unflushed wal edits. Option three(multiple memstores) a) Let splitlogworker pick wals of a failed RS in order instead of random. Say a failed RS has WAL1, WAL2, WAL3,... WALk. a worker will only pick WAL2 if WAL1 is done(or errored) etc. b) During replay, we pass original wal sequence ids of edits to the receiving RS c) In receiving RS, we bucket WAL files to a different memstore during replaying and use the original sequence Ids. Say wal1-wal4 to memstore1, wal5-wal10 to memstore2 etc. We only flush the bucket memstore when all wals inside the bucket are replayed. all wals can be replayed concurrently. d) writes from normal traffic(allow writes during recovery) are put in a different memstore as of today and flush normally with new sequenceIds. Option four a) During replay, we pass original wal sequence ids b) for each wal edit, we store each edit's sequence id along with its key. c) during scanning, we use the original sequence id if it's present otherwise its store file sequence Id d) compaction can just leave put with max sequence id
          Hide
          Jeffrey Zhong added a comment -

          A small correction for option three:

          a worker will only pick WAL2 if WAL1 is done(or errored) etc.

          should be

          a worker will only pick WAL2 if WAL1 is taken by another worker

          Show
          Jeffrey Zhong added a comment - A small correction for option three: a worker will only pick WAL2 if WAL1 is done(or errored) etc. should be a worker will only pick WAL2 if WAL1 is taken by another worker
          Hide
          stack added a comment -

          On option two, if WALs are being replayed without order, couldn't an edit from WAL 1 (an old WAL) overwrite an edit from WAL 3 (a newer WAL) because memstore does not consider sequenceid?

          I do not think option three will work. We want to be able to put in place multiple WALs per server in the near future and in this case the sequenceids will be spread about amongst a few logs (probably two is enough). Since the sequenceids will be spread across N WALs, splitlogworker will not be able to deduce WAL order since some WALs will be contemporaneous having been written to in // (In other words, replay is bringing on sooner a problem we are going to need to solve anyways).

          In Option three, how will you bucket WALs? You will need to pass in the the WAL file name when you do the Put? How will you signal the regionserver the WAL is done? A special edit?

          On replay, do you need a memstore that considers sequenceid such that when two edits w/ same coordinate, the one w/ the latest sequenceid is retained rather than the last written?

          What is the worst case if we could not flush until all WALs replayed?

          Lets say 2k regions on two servers? That means one server will need to take all edits from 1k regions? Lets say there were 256k WALs? At 128M per WAL that is 32G of edits we'd have to keep in memory w/o flushing? If were also taking writes for all 2k regions, that would be extra memory pressure. We'd fall over in this case.

          Could the replay tell the RS it was replaying a single WAL and when it was done? For WAL it could pass the sequence ids and a hash of the WAL path. Not sure how it would flag the replay is done since in distributed split, a RS could be taking on multiple WAL edits at a time... (so can not treat the arrival of a new WAL file hash as meaning we are done w/ the old file). Region server could take on the edits into a special single-WAL memstore. Region server could keep taking on edits from WALs and keep them in memory until it hit memory barrier. We could then flush these per WAL memstores as hfiles w/ their sequence ids. If the flush didn't get all of a WAL, that should be fine. Would be lots of hfiles possibly but having to flush would be rare I'd say (RS w/ 1k regions and 256 WALs would be rare).

          Show
          stack added a comment - On option two, if WALs are being replayed without order, couldn't an edit from WAL 1 (an old WAL) overwrite an edit from WAL 3 (a newer WAL) because memstore does not consider sequenceid? I do not think option three will work. We want to be able to put in place multiple WALs per server in the near future and in this case the sequenceids will be spread about amongst a few logs (probably two is enough). Since the sequenceids will be spread across N WALs, splitlogworker will not be able to deduce WAL order since some WALs will be contemporaneous having been written to in // (In other words, replay is bringing on sooner a problem we are going to need to solve anyways). In Option three, how will you bucket WALs? You will need to pass in the the WAL file name when you do the Put? How will you signal the regionserver the WAL is done? A special edit? On replay, do you need a memstore that considers sequenceid such that when two edits w/ same coordinate, the one w/ the latest sequenceid is retained rather than the last written? What is the worst case if we could not flush until all WALs replayed? Lets say 2k regions on two servers? That means one server will need to take all edits from 1k regions? Lets say there were 256k WALs? At 128M per WAL that is 32G of edits we'd have to keep in memory w/o flushing? If were also taking writes for all 2k regions, that would be extra memory pressure. We'd fall over in this case. Could the replay tell the RS it was replaying a single WAL and when it was done? For WAL it could pass the sequence ids and a hash of the WAL path. Not sure how it would flag the replay is done since in distributed split, a RS could be taking on multiple WAL edits at a time... (so can not treat the arrival of a new WAL file hash as meaning we are done w/ the old file). Region server could take on the edits into a special single-WAL memstore. Region server could keep taking on edits from WALs and keep them in memory until it hit memory barrier. We could then flush these per WAL memstores as hfiles w/ their sequence ids. If the flush didn't get all of a WAL, that should be fine. Would be lots of hfiles possibly but having to flush would be rare I'd say (RS w/ 1k regions and 256 WALs would be rare).
          Hide
          Jeffrey Zhong added a comment -

          On option two, if WALs are being replayed without order, couldn't an edit from WAL 1 (an old WAL) overwrite an edit from WAL 3 (a newer WAL) because memstore does not consider sequenceid?

          You're right. Option2 has to consider original wal sequenceId as well.

          In Option three, how will you bucket WALs? You will need to pass in the the WAL file name when you do the Put? How will you signal the regionserver the WAL is done? A special edit?

          I need to pass WAL file name(or its hash) inside each replay batch. Receiving RS can put watcher on the split log file deletion/done ZK events and flush a bucket memstore when all log files of the bucket are recovered. Bucket logic is controlled by receiving RS and configuration. Since all the info are in ZK so receiving RS can determine which files belong to which bucket.

          Lets say 2k regions on two servers? That means one server will need to take all edits from 1k regions? Lets say there were 256k WALs? At 128M per WAL that is 32G of edits we'd have to keep in memory w/o flushing? If were also taking writes for all 2k regions, that would be extra memory pressure. We'd fall over in this case.

          I guess the 2k regions on two servers is a long term goal for us. We could add a memory limit for all memstore opened for replay. If the limit is exceeded, the receiving RS rejects replays. In addition it could also pick a memstore and resign work items for the store and tell the playing RS to reassign the work item(wal file).

          I like your single-WAL memstore flush approach(a special case with number of wal per bucket=1). This way keeps memory management & flush simpler while at cost of more IOs. We could implement this at the beginning. Possibly group more log files for flush depending on how multiple WAL implementation goes.

          Thanks!

          Show
          Jeffrey Zhong added a comment - On option two, if WALs are being replayed without order, couldn't an edit from WAL 1 (an old WAL) overwrite an edit from WAL 3 (a newer WAL) because memstore does not consider sequenceid? You're right. Option2 has to consider original wal sequenceId as well. In Option three, how will you bucket WALs? You will need to pass in the the WAL file name when you do the Put? How will you signal the regionserver the WAL is done? A special edit? I need to pass WAL file name(or its hash) inside each replay batch. Receiving RS can put watcher on the split log file deletion/done ZK events and flush a bucket memstore when all log files of the bucket are recovered. Bucket logic is controlled by receiving RS and configuration. Since all the info are in ZK so receiving RS can determine which files belong to which bucket. Lets say 2k regions on two servers? That means one server will need to take all edits from 1k regions? Lets say there were 256k WALs? At 128M per WAL that is 32G of edits we'd have to keep in memory w/o flushing? If were also taking writes for all 2k regions, that would be extra memory pressure. We'd fall over in this case. I guess the 2k regions on two servers is a long term goal for us. We could add a memory limit for all memstore opened for replay. If the limit is exceeded, the receiving RS rejects replays. In addition it could also pick a memstore and resign work items for the store and tell the playing RS to reassign the work item(wal file). I like your single-WAL memstore flush approach(a special case with number of wal per bucket=1). This way keeps memory management & flush simpler while at cost of more IOs. We could implement this at the beginning. Possibly group more log files for flush depending on how multiple WAL implementation goes. Thanks!
          Hide
          Himanshu Vashishtha added a comment -

          Thanks for the above discussion. I have some follow up questions on Option 2/3:

          1. If I am reading it correct, Option 2/3 are preserving the sequenceId of the old wal file? Does that mean the WAL edit created at the new RS for this entry would have old sequenceId? Or something else?
          When a region is opened, it reads the max sequence Ids from its StoreFiles and sets the FSHlog counter to it (if the counter is at some lower value). If we are keeping the original sequence IDs, a WAL file could have a random distribution of sequenceIds (would not be tightly ascending as we have it now). Could there be any gotcha here? Such as handling chain fail-over.

          2. Another question is, initially we had one recovered.edits file per WAL; now we planning one HFile per WAL.
          Looking at this, saving on number of I/O (and NN ops) is not that much IMHO as it is the same number of files as such? With larger number of small files, it could lead to more compaction. Though stripe compaction could help, but that's a different thing (and I haven't looked at the compaction code). Bucketing WALs is definitely better.

          Show
          Himanshu Vashishtha added a comment - Thanks for the above discussion. I have some follow up questions on Option 2/3: 1. If I am reading it correct, Option 2/3 are preserving the sequenceId of the old wal file? Does that mean the WAL edit created at the new RS for this entry would have old sequenceId? Or something else? When a region is opened, it reads the max sequence Ids from its StoreFiles and sets the FSHlog counter to it (if the counter is at some lower value). If we are keeping the original sequence IDs, a WAL file could have a random distribution of sequenceIds (would not be tightly ascending as we have it now). Could there be any gotcha here? Such as handling chain fail-over. 2. Another question is, initially we had one recovered.edits file per WAL; now we planning one HFile per WAL. Looking at this, saving on number of I/O (and NN ops) is not that much IMHO as it is the same number of files as such? With larger number of small files, it could lead to more compaction. Though stripe compaction could help, but that's a different thing (and I haven't looked at the compaction code). Bucketing WALs is definitely better.
          Hide
          Jeffrey Zhong added a comment -

          I created HBASE-8701 and linked it to this JIRA to address the concerns from Elliot and Himanshu.

          When a region is opened, it reads the max sequence Ids from its StoreFiles and sets the FSHlog counter to it (if the counter is at some lower value). If we are keeping the original sequence IDs, a WAL file could have a random distribution of sequenceIds (would not be tightly ascending as we have it now). Could there be any gotcha here? Such as handling chain fail-over.

          We have to use skip wal option here.

          Another question is, initially we had one recovered.edits file per WAL; now we planning one HFile per WAL

          This is a good question. The benefits are still no recovered.edits related IOs and allow writes during recovery. Currently we already created many hfiles because we flush after each recovered edits replay. I'm planning to use a config to control the new behavior because the issue we're trying to address isn't a common usage scenario. Later we can introduce bucketing for optimization this part.

          Show
          Jeffrey Zhong added a comment - I created HBASE-8701 and linked it to this JIRA to address the concerns from Elliot and Himanshu. When a region is opened, it reads the max sequence Ids from its StoreFiles and sets the FSHlog counter to it (if the counter is at some lower value). If we are keeping the original sequence IDs, a WAL file could have a random distribution of sequenceIds (would not be tightly ascending as we have it now). Could there be any gotcha here? Such as handling chain fail-over. We have to use skip wal option here. Another question is, initially we had one recovered.edits file per WAL; now we planning one HFile per WAL This is a good question. The benefits are still no recovered.edits related IOs and allow writes during recovery. Currently we already created many hfiles because we flush after each recovered edits replay. I'm planning to use a config to control the new behavior because the issue we're trying to address isn't a common usage scenario. Later we can introduce bucketing for optimization this part.
          Hide
          Devaraj Das added a comment -

          I'm planning to use a config to control the new behavior because the issue we're trying to address isn't a common usage scenario.

          Don't think this is a good idea since if this config is disabled, and we have a scenario where the problem surfaces, it'd lead to data consistency issues. If we want to have a config for having some control, I'd vote we instead have a config that would disallow writes during recovery, or disallow operations (with timestamps?) that would lead to this unpredictability of ordering. Thoughts?

          Show
          Devaraj Das added a comment - I'm planning to use a config to control the new behavior because the issue we're trying to address isn't a common usage scenario. Don't think this is a good idea since if this config is disabled, and we have a scenario where the problem surfaces, it'd lead to data consistency issues. If we want to have a config for having some control, I'd vote we instead have a config that would disallow writes during recovery, or disallow operations (with timestamps?) that would lead to this unpredictability of ordering. Thoughts?
          Hide
          Ted Yu added a comment -

          have a config that would disallow writes during recovery, or disallow operations (with timestamps?)

          Both are fine with me.

          Show
          Ted Yu added a comment - have a config that would disallow writes during recovery, or disallow operations (with timestamps?) Both are fine with me.
          Hide
          stack added a comment -

          We have to use skip wal option here.

          I was hoping to avoid our doing skip-wal for reasons argued above, that replaying edits w/ skip WAL enabled introduces more states and will complicate replay but old edits coming into the new server getting new seqids will itself make for some new interesting states (If the server we are playing into crashes before all is flushed, it will have in its WALs edits where the sequenceid for 'B', is > that for 'C', so on its recovery, 'B', will come out when we want 'C', the last edit inserted at a particular coordinate).

          So, if no WAL, what happens when we need to flush a memstore or a background replay memstore (the one-memstore-per-region we discuss above)? What seqid will we write out into the hfile if we have to flush memory? I suppose if this replay backing memstore had the old WAL seqid, it would be legit to use these. The flushed file would sort properly with an old seqid (but then this would be a different kind of flush, one where you dictate the seqid file rather than take what is current in the server – that will be intrusive to change).

          We'd have to use the old ids in case we had to flush midway through a WAL (I suppose we say this already above)

          But thinking more on the per-WAL replay memstore, there are kinks to figure (apart from the one above where we want to have a flush w/ a seqid that is not the servers current max seqid). As hfiles contain sorted kvs but the edits in the old WAL not in sort order, if we sort the edits so we can flush the hfile, then we'll have seqids not-in-order. Do we take the highest seqid in the hfile as the hfiles' seqid? This would be different to how we usually write hfiles. There could be issues in here.

          Another question is, initially we had one recovered.edits file per WAL; now we planning one HFile per WAL.

          This would be only if we had to flush. We'd keep per-WAL replay memstore so if we have to flush, the file written out – this would be at an extreme.

          I'm planning to use a config to control the new behavior because the issue we're trying to address isn't a common usage scenario.

          I'd vote we instead have a config that would disallow writes during recovery

          +1 on disabling writes during recovery for now. It is this that is adding the complication. If we disable writes during recovery, we can turn on distributed log replay now as the default and enjoy the speedup it brings over current log splitting. We can work on being able to take on writes during recovery for later and over in the new issue.

          Show
          stack added a comment - We have to use skip wal option here. I was hoping to avoid our doing skip-wal for reasons argued above, that replaying edits w/ skip WAL enabled introduces more states and will complicate replay but old edits coming into the new server getting new seqids will itself make for some new interesting states (If the server we are playing into crashes before all is flushed, it will have in its WALs edits where the sequenceid for 'B', is > that for 'C', so on its recovery, 'B', will come out when we want 'C', the last edit inserted at a particular coordinate). So, if no WAL, what happens when we need to flush a memstore or a background replay memstore (the one-memstore-per-region we discuss above)? What seqid will we write out into the hfile if we have to flush memory? I suppose if this replay backing memstore had the old WAL seqid, it would be legit to use these. The flushed file would sort properly with an old seqid (but then this would be a different kind of flush, one where you dictate the seqid file rather than take what is current in the server – that will be intrusive to change). We'd have to use the old ids in case we had to flush midway through a WAL (I suppose we say this already above) But thinking more on the per-WAL replay memstore, there are kinks to figure (apart from the one above where we want to have a flush w/ a seqid that is not the servers current max seqid). As hfiles contain sorted kvs but the edits in the old WAL not in sort order, if we sort the edits so we can flush the hfile, then we'll have seqids not-in-order. Do we take the highest seqid in the hfile as the hfiles' seqid? This would be different to how we usually write hfiles. There could be issues in here. Another question is, initially we had one recovered.edits file per WAL; now we planning one HFile per WAL. This would be only if we had to flush. We'd keep per-WAL replay memstore so if we have to flush, the file written out – this would be at an extreme. I'm planning to use a config to control the new behavior because the issue we're trying to address isn't a common usage scenario. I'd vote we instead have a config that would disallow writes during recovery +1 on disabling writes during recovery for now. It is this that is adding the complication. If we disable writes during recovery, we can turn on distributed log replay now as the default and enjoy the speedup it brings over current log splitting. We can work on being able to take on writes during recovery for later and over in the new issue.
          Hide
          Jeffrey Zhong added a comment -

          Do we take the highest seqid in the hfile as the hfiles' seqid?

          Yes. We use highest seqid as hfile seqid and no need to sort by seqids internally.

          "disable writes during recovery" won't solve the issue totally because we could have put(row1,t) in wal1 and put(row1,t) in wal2. Since we replay wal files concurrently, we could replay wal2 before wal1. The issue will come after we replay wal2, region flush and then we replay wal1. I do agree that handling this edge case makes the implementation much more complicated.

          Show
          Jeffrey Zhong added a comment - Do we take the highest seqid in the hfile as the hfiles' seqid? Yes. We use highest seqid as hfile seqid and no need to sort by seqids internally. "disable writes during recovery" won't solve the issue totally because we could have put(row1,t) in wal1 and put(row1,t) in wal2. Since we replay wal files concurrently, we could replay wal2 before wal1. The issue will come after we replay wal2, region flush and then we replay wal1. I do agree that handling this edge case makes the implementation much more complicated.
          Hide
          Enis Soztutar added a comment -

          Here is a proposed scheme that can solve this problem:

          The region will be opened for replaying as in previous. The normal writes go to the memstore, and the memstore is flushed as usual. The region servers who are reading the WAL and sending to the replaying RS will still be the same, except for the fact that the edits are sent with their seq_ids.

          On the replaying RS, for all regions that are in replaying state, there is a single buffer. All edits are appended to this buffer without any sorting. This buffer can be accounted as a memstore, and it will have the memstore flush size as max size. Once this is reached, or due to global memstore pressure, we are asked to flush, we do spill this to disk after sorting. This buffer keeps <kv,seq> pairs, and sorts according to <kv,seq>. If there is not memory pressure, and buffer does not fill up, we don't need to spill to disk.

          Once the replaying is finished, and master asks the region server to open the region for reading, then we do a final merge sort for the in-memory sorted buffer, and all on-disk spilled buffers and create an hfile, discarding kv's that have the same kv, but smaller seq_id. This file will be a single hfile that corresponds to a flush. This hfile will have a seq_id that is obtained from the wal edits. Then we add this hfile to the store, and open the region as usual. This kind of keeping an unsorted buffer, and sorting it with qsort with spills and final on-disk merge sort might even be faster, since otherwise, we would be doing an insertion to the memstore, which becomes an insertion sort.

          The other thing we need to change is that replayed edits will not go into the wal again, so we keep track of recovering state for the region server, and re-do the work if there is a subsequent failure.

          In sort, this will be close to the BigTable's in-memory sort for each WAL file approach, but instead we gather the edits for the region from all WAL files by doing the replay RPC, and do the sort per region. End result, we create a flushed hfile, as if the region just flushed before the crash.

          Show
          Enis Soztutar added a comment - Here is a proposed scheme that can solve this problem: The region will be opened for replaying as in previous. The normal writes go to the memstore, and the memstore is flushed as usual. The region servers who are reading the WAL and sending to the replaying RS will still be the same, except for the fact that the edits are sent with their seq_ids. On the replaying RS, for all regions that are in replaying state, there is a single buffer. All edits are appended to this buffer without any sorting. This buffer can be accounted as a memstore, and it will have the memstore flush size as max size. Once this is reached, or due to global memstore pressure, we are asked to flush, we do spill this to disk after sorting. This buffer keeps <kv,seq> pairs, and sorts according to <kv,seq>. If there is not memory pressure, and buffer does not fill up, we don't need to spill to disk. Once the replaying is finished, and master asks the region server to open the region for reading, then we do a final merge sort for the in-memory sorted buffer, and all on-disk spilled buffers and create an hfile, discarding kv's that have the same kv, but smaller seq_id. This file will be a single hfile that corresponds to a flush. This hfile will have a seq_id that is obtained from the wal edits. Then we add this hfile to the store, and open the region as usual. This kind of keeping an unsorted buffer, and sorting it with qsort with spills and final on-disk merge sort might even be faster, since otherwise, we would be doing an insertion to the memstore, which becomes an insertion sort. The other thing we need to change is that replayed edits will not go into the wal again, so we keep track of recovering state for the region server, and re-do the work if there is a subsequent failure. In sort, this will be close to the BigTable's in-memory sort for each WAL file approach, but instead we gather the edits for the region from all WAL files by doing the replay RPC, and do the sort per region. End result, we create a flushed hfile, as if the region just flushed before the crash.

            People

            • Assignee:
              Jeffrey Zhong
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development