HBase
  1. HBase
  2. HBASE-5599

[hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.6
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: hbck
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The hbck tool can not fix the six scenarios.
      1. Version file does not exist in root dir.
      Fix: I try to create a version file by 'FSUtils.setVersion' method.

      2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any region server.
      Fix: I get region info form the hdfs file, this region info write to '.META.' table.

      3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME]
      Fix: I get region info form the hdfs file, this region info write to '.META.' table.

      4. [REGIONNAME] should not be deployed according to META, but is deployed on [SERVERNAME]
      Fix: Close this region.

      5. First region should start with an empty key. You need to create a new region and regioninfo in HDFS to plug the hole.
      Fix: The region info is not in hdfs and .META., so it create a empty region for this error.

      6. There is a hole in the region chain between [KEY] and [KEY]. You need to create a new regioninfo and region dir in hdfs to plug the hole.
      Fix: The region info is not in hdfs and .META., so it create a empty region for this hole.

      1. license.png
        234 kB
        Jonathan Hsieh
      2. hbase-5599-trunk_v8.patch
        12 kB
        fulin wang
      3. hbase-5599-trunk_v7.patch
        12 kB
        fulin wang
      4. hbase-5599-trunk_v5.patch
        3 kB
        fulin wang
      5. hbase-5599-92-v8.patch
        12 kB
        Jonathan Hsieh
      6. hbase-5599-0.94_v5.patch
        3 kB
        fulin wang
      7. hbase-5599-0.92_v5.patch
        3 kB
        fulin wang
      8. hbase-5599-0.90.patch
        11 kB
        fulin wang
      9. hbase-5599-0.90_v8
        12 kB
        fulin wang
      10. hbase-5599-0.90_v7.patch
        12 kB
        fulin wang
      11. hbase-5599-0.90_v6.patch
        8 kB
        fulin wang
      12. hbase-5599-0.90_v5.patch
        2 kB
        fulin wang
      13. hbase-5599-0.90_v3.patch
        15 kB
        fulin wang
      14. hbase-5599-0.90_v2.patch
        16 kB
        fulin wang
      15. 0.90-surefire-report-hbck.html
        204 kB
        fulin wang

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in HBase-0.92-security #105 (See https://builds.apache.org/job/HBase-0.92-security/105/)
          HBASE-5599 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies (fulin wang) (Revision 1324879)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
          Show
          Hudson added a comment - Integrated in HBase-0.92-security #105 (See https://builds.apache.org/job/HBase-0.92-security/105/ ) HBASE-5599 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies (fulin wang) (Revision 1324879) Result = FAILURE jmhsieh : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #9 (See https://builds.apache.org/job/HBase-0.94-security/9/)
          HBASE-5599 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies (fulin wang) (Revision 1324880)

          Result = SUCCESS
          jmhsieh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #9 (See https://builds.apache.org/job/HBase-0.94-security/9/ ) HBASE-5599 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies (fulin wang) (Revision 1324880) Result = SUCCESS jmhsieh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #168 (See https://builds.apache.org/job/HBase-TRUNK-security/168/)
          HBASE-5599 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies (fulin wang) (Revision 1324881)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #168 (See https://builds.apache.org/job/HBase-TRUNK-security/168/ ) HBASE-5599 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies (fulin wang) (Revision 1324881) Result = FAILURE jmhsieh : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #367 (See https://builds.apache.org/job/HBase-0.92/367/)
          HBASE-5599 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies (fulin wang) (Revision 1324879)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #367 (See https://builds.apache.org/job/HBase-0.92/367/ ) HBASE-5599 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies (fulin wang) (Revision 1324879) Result = FAILURE jmhsieh : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #102 (See https://builds.apache.org/job/HBase-0.94/102/)
          HBASE-5599 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies (fulin wang) (Revision 1324880)

          Result = SUCCESS
          jmhsieh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #102 (See https://builds.apache.org/job/HBase-0.94/102/ ) HBASE-5599 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies (fulin wang) (Revision 1324880) Result = SUCCESS jmhsieh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2743 (See https://builds.apache.org/job/HBase-TRUNK/2743/)
          HBASE-5599 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies (fulin wang) (Revision 1324881)

          Result = SUCCESS
          jmhsieh :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2743 (See https://builds.apache.org/job/HBase-TRUNK/2743/ ) HBASE-5599 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies (fulin wang) (Revision 1324881) Result = SUCCESS jmhsieh : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
          Hide
          Jonathan Hsieh added a comment -

          Attached tweaked patch applied to 0.92

          Show
          Jonathan Hsieh added a comment - Attached tweaked patch applied to 0.92
          Hide
          Jonathan Hsieh added a comment -

          Fulin, thanks for the patch!

          Committed to 0.90/0.92/0.94/0.96-trunk

          Show
          Jonathan Hsieh added a comment - Fulin, thanks for the patch! Committed to 0.90/0.92/0.94/0.96-trunk
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522215/hbase-5599-trunk_v8.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestSplitLogManager

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1476//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1476//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1476//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522215/hbase-5599-trunk_v8.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestSplitLogManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1476//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1476//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1476//console This message is automatically generated.
          Hide
          fulin wang added a comment -

          Thanks for carefully review.
          I have modified the 'falut'.
          The patch re-upload and select ASF item.

          Show
          fulin wang added a comment - Thanks for carefully review. I have modified the 'falut'. The patch re-upload and select ASF item.
          Hide
          Jonathan Hsieh added a comment -

          Attached a screen shot that shows ASF grant being selected.

          Show
          Jonathan Hsieh added a comment - Attached a screen shot that shows ASF grant being selected.
          Hide
          Jonathan Hsieh added a comment -

          I can take care of the tweak for 0.92.

          For me to commit the patches you've submitted, you need to "Grant license to ASF for inclusion in ASF works (as per the Apache License §5)". When you attach files, there is a selector box at the bottom, and you need to select that.

          Show
          Jonathan Hsieh added a comment - I can take care of the tweak for 0.92. For me to commit the patches you've submitted, you need to "Grant license to ASF for inclusion in ASF works (as per the Apache License §5)". When you attach files, there is a selector box at the bottom, and you need to select that.
          Hide
          fulin wang added a comment -

          Thank Jonathan Hsieh for your comment.
          I do not know what the word "Could you re-upload the same v7 patches but give ASF permission to include" mean?
          you can modify the patch for 0.92 branch and other branch.

          Show
          fulin wang added a comment - Thank Jonathan Hsieh for your comment. I do not know what the word "Could you re-upload the same v7 patches but give ASF permission to include" mean? you can modify the patch for 0.92 branch and other branch.
          Hide
          Jonathan Hsieh added a comment -

          Fulin. Great test, and nice patch. On trunk, the failed test passes for me locally. On 0.90, failed tests pass for me locally.

          I plan on committing to 0.90, 0.92, 0.94, trunk.

          Fixed a few spacing and a spelling nit ("falut" -> "fault") that I'll fix before commit.

          Remaining issues:

          • Could you re-upload the same v7 patches but give ASF permission to include?
          • Trunk version of patch applies to 0.94, needs a minor tweak for 0.92. Mind doing this?
          Show
          Jonathan Hsieh added a comment - Fulin. Great test, and nice patch. On trunk, the failed test passes for me locally. On 0.90, failed tests pass for me locally. I plan on committing to 0.90, 0.92, 0.94, trunk. Fixed a few spacing and a spelling nit ("falut" -> "fault") that I'll fix before commit. Remaining issues: Could you re-upload the same v7 patches but give ASF permission to include? Trunk version of patch applies to 0.94, needs a minor tweak for 0.92. Mind doing this?
          Hide
          Jonathan Hsieh added a comment -

          Renaming due to new scope of patch.

          Show
          Jonathan Hsieh added a comment - Renaming due to new scope of patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521930/hbase-5599-trunk_v7.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.regionserver.wal.TestLogRolling

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1453//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1453//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1453//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521930/hbase-5599-trunk_v7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestLogRolling Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1453//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1453//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1453//console This message is automatically generated.
          Hide
          fulin wang added a comment -

          1. I modify the hbck code before the six error and nine skippe,
          the 0.90-surefire-report-hbck.html file is my test result for my changed.

          2. This is a trunk version test result.
          -------------------------------------------------------------------------------
          Test set: org.apache.hadoop.hbase.util.TestHBaseFsck
          -------------------------------------------------------------------------------
          Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 197.475 sec

          Show
          fulin wang added a comment - 1. I modify the hbck code before the six error and nine skippe, the 0.90-surefire-report-hbck.html file is my test result for my changed. 2. This is a trunk version test result. ------------------------------------------------------------------------------- Test set: org.apache.hadoop.hbase.util.TestHBaseFsck ------------------------------------------------------------------------------- Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 197.475 sec
          Hide
          Jonathan Hsieh added a comment -

          Can you share the test code? Extra sets of eyes might help find the problem!

          Show
          Jonathan Hsieh added a comment - Can you share the test code? Extra sets of eyes might help find the problem!
          Hide
          fulin wang added a comment -

          I have writed the test case. I am running all the test case,
          But the test case testShouldCheckMasterFailOverWhenMETAIsInOpenedState always failed.

          Show
          fulin wang added a comment - I have writed the test case. I am running all the test case, But the test case testShouldCheckMasterFailOverWhenMETAIsInOpenedState always failed.
          Hide
          Jonathan Hsieh added a comment -

          Have you encountered this in your cluster before? (I actually have not)

          To test it properly, you would probably disable a table and then do some funny direct calls to a region server to set up a the funky internal state. I actually spent a lot of time getting tests working for hbck but it definitely caught some important cornercases.

          I've suggested this before, but I'd be happy to commit the versionFile part and push off this SHOULD_NOT_BE_DEPLOYED case to another issue. (Also, still waiting for an answer about the Merge.java edits).

          Show
          Jonathan Hsieh added a comment - Have you encountered this in your cluster before? (I actually have not) To test it properly, you would probably disable a table and then do some funny direct calls to a region server to set up a the funky internal state. I actually spent a lot of time getting tests working for hbck but it definitely caught some important cornercases. I've suggested this before, but I'd be happy to commit the versionFile part and push off this SHOULD_NOT_BE_DEPLOYED case to another issue. (Also, still waiting for an answer about the Merge.java edits).
          Hide
          fulin wang added a comment -

          The SHOULD_NOT_BE_DEPLOYED fault scenario, I am thinking about how to write this test case.
          Because the fault scenario appear when the table is disabled, but the region of this table is not closed.

          Show
          fulin wang added a comment - The SHOULD_NOT_BE_DEPLOYED fault scenario, I am thinking about how to write this test case. Because the fault scenario appear when the table is disabled, but the region of this table is not closed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520573/hbase-5599-0.90_v6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1378//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520573/hbase-5599-0.90_v6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1378//console This message is automatically generated.
          Hide
          Jonathan Hsieh added a comment -

          Thanks for the updates, the new patch looks pretty good. A few things to follow up on before commit:

          • If adding a test case is hard, let's file new issue to "add HBCK test case for SHOULD_NOT_BE_DEPLOYED case".
          • Change the text here to be "Try to fix missing hbase.version file in hdfs."
             System.err.println("   -fixVersionFile   Try to fix the hbase.version missing.");
            
          • I noticed the Merge.java changes are still present in the newer patch – is that intended? It don't seem necessary or related to this issue.

          This should be good to commit for 0.90, if you address these issues.

          Could you provide an updated version for trunk?

          Thanks!

          Show
          Jonathan Hsieh added a comment - Thanks for the updates, the new patch looks pretty good. A few things to follow up on before commit: If adding a test case is hard, let's file new issue to "add HBCK test case for SHOULD_NOT_BE_DEPLOYED case". Change the text here to be "Try to fix missing hbase.version file in hdfs." System .err.println( " -fixVersionFile Try to fix the hbase.version missing." ); I noticed the Merge.java changes are still present in the newer patch – is that intended? It don't seem necessary or related to this issue. This should be good to commit for 0.90, if you address these issues. Could you provide an updated version for trunk? Thanks!
          Hide
          fulin wang added a comment -

          I make a hbase-5599-0.90_v6.patch, please review.
          This is test result:
          Tests Errors Failures Skipped Success Rate Time
          14 0 0 0 100% 181.667

          TestHBaseFsck testHBaseFsckClean 4.843
          testDupeStartKey 10.086
          testDupeRegion 10.382
          testDegenerateRegions 10.098
          testContainedRegionOverlap 9.598
          testOverlapAndOrphan 12.687
          testCoveredStartKey 10.836
          testRegionHole 8.652
          testHDFSRegioninfoMissing 11.895
          testNotInMetaOrDeployedHole 8.262
          testNotInMetaHole 8.387
          testNotInHdfs 8.345
          testNoHdfsTable 46.275

          Show
          fulin wang added a comment - I make a hbase-5599-0.90_v6.patch, please review. This is test result: Tests Errors Failures Skipped Success Rate Time 14 0 0 0 100% 181.667 TestHBaseFsck testHBaseFsckClean 4.843 testDupeStartKey 10.086 testDupeRegion 10.382 testDegenerateRegions 10.098 testContainedRegionOverlap 9.598 testOverlapAndOrphan 12.687 testCoveredStartKey 10.836 testRegionHole 8.652 testHDFSRegioninfoMissing 11.895 testNotInMetaOrDeployedHole 8.262 testNotInMetaHole 8.387 testNotInHdfs 8.345 testNoHdfsTable 46.275
          Hide
          fulin wang added a comment -

          Thanks Jonathan Hsieh for review.
          1. the NO_VERSION_FILE, I will add '-fixVersionFile' to fix this error and test case.
          2. the SHOULD_NOT_BE_DEPLOYED fault scensrios, I change disable code that it is not call bulkAssign method, so the region is deployed but the table is disable.I do not know how to write this test case.

          Show
          fulin wang added a comment - Thanks Jonathan Hsieh for review. 1. the NO_VERSION_FILE, I will add '-fixVersionFile' to fix this error and test case. 2. the SHOULD_NOT_BE_DEPLOYED fault scensrios, I change disable code that it is not call bulkAssign method, so the region is deployed but the table is disable.I do not know how to write this test case.
          Hide
          Jonathan Hsieh added a comment -

          Thanks for updating the patches.

          Instead of relying on shouldFixAssignments, could you add a new config flag for fixing the NO_VERSION_FILE? The fix isn't really a region assignment fix! Maybe call it -fixVersionFile, and have it on with the combo '-repair*' options?

          Can you add a test case for SHOULD_NOT_BE_DEPLOYED? You should be able to hack one of the test cases from TestHBaseFsck. I've had problems just using assign before so if tests come back failing, you may want to try HBaseFsckRepair.fixMultiAssignemnt or tryAssignemntRepair.

          The merge addition isn't really related – was its inclusion intended?

          Show
          Jonathan Hsieh added a comment - Thanks for updating the patches. Instead of relying on shouldFixAssignments, could you add a new config flag for fixing the NO_VERSION_FILE? The fix isn't really a region assignment fix! Maybe call it -fixVersionFile, and have it on with the combo '-repair*' options? Can you add a test case for SHOULD_NOT_BE_DEPLOYED? You should be able to hack one of the test cases from TestHBaseFsck. I've had problems just using assign before so if tests come back failing, you may want to try HBaseFsckRepair.fixMultiAssignemnt or tryAssignemntRepair. The merge addition isn't really related – was its inclusion intended?
          Hide
          fulin wang added a comment -

          The NOT_IN_META_OR_DEPLOYED,NOT_IN_META,FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN fault scenarios, Jonathan Hsieh have fixed,
          I have tested this scenarious, not found any issue.
          This patch I will fix NO_VERSION_FILE, SHOULD_NOT_BE_DEPLOYED fault scenarios.

          Show
          fulin wang added a comment - The NOT_IN_META_OR_DEPLOYED,NOT_IN_META,FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN fault scenarios, Jonathan Hsieh have fixed, I have tested this scenarious, not found any issue. This patch I will fix NO_VERSION_FILE, SHOULD_NOT_BE_DEPLOYED fault scenarios.
          Hide
          fulin wang added a comment -

          Today I will make a patch base on 0.90 new codes.
          If I change in 0.90 that it pass the review, this code will merge into 0.92/0.94/trunk versions.

          Show
          fulin wang added a comment - Today I will make a patch base on 0.90 new codes. If I change in 0.90 that it pass the review, this code will merge into 0.92/0.94/trunk versions.
          Hide
          Jonathan Hsieh added a comment -

          Hi Fulin,

          I've committed HBASE-5128 so let's work together to port the functionality you've implemented into it as well.

          • Some of the methods you've added in HBaseFsckRepair are similar in HBASE-5128. Let's consolidate.
          • It seems that you are focused on the 0.90 versions – it is important that whatever changes we make also make it into the newer 0.92/0.94/trunk versions. I can definitely help out there.
          • It is important to add some testing for the new scenarios as well – checkout TestHbckFsck for some example to emulate. I've basically started to have have one test for each error condition.
          Show
          Jonathan Hsieh added a comment - Hi Fulin, I've committed HBASE-5128 so let's work together to port the functionality you've implemented into it as well. Some of the methods you've added in HBaseFsckRepair are similar in HBASE-5128 . Let's consolidate. It seems that you are focused on the 0.90 versions – it is important that whatever changes we make also make it into the newer 0.92/0.94/trunk versions. I can definitely help out there. It is important to add some testing for the new scenarios as well – checkout TestHbckFsck for some example to emulate. I've basically started to have have one test for each error condition.
          Hide
          fulin wang added a comment -

          close region by HMaster.unassign method, It can clear record in HMaster and close region in regionserver.

          Show
          fulin wang added a comment - close region by HMaster.unassign method, It can clear record in HMaster and close region in regionserver.
          Hide
          Jonathan Hsieh added a comment -

          @fulin. Zhihong's suggestion is to merge the changes you have into the updates in HBASE-5128. I'll be posting an updated copy of that for 0.92/0.94/trunk this evening and the 0.90 version will follow shortly after that.

          Show
          Jonathan Hsieh added a comment - @fulin. Zhihong's suggestion is to merge the changes you have into the updates in HBASE-5128 . I'll be posting an updated copy of that for 0.92/0.94/trunk this evening and the 0.90 version will follow shortly after that.
          Hide
          fulin wang added a comment -

          today I will update the patch, I am testing;
          please, it was referring to the patch, I think you should think about this scenarios in my patch.
          I hope that the hbck tool will be strong, thanks.

          Show
          fulin wang added a comment - today I will update the patch, I am testing; please, it was referring to the patch, I think you should think about this scenarios in my patch. I hope that the hbck tool will be strong, thanks.
          Hide
          Jonathan Hsieh added a comment -

          @ram @fulin

          I'm working on getting HBASE-5128 into trunk/0.94/0.92/0.90. I'm focusing on trunk first (0.94/0.92 should fall out) and changes from the trunk versions (max merge and specific table safegurds) should make it into the 0.90 version.

          I believe NOT_IN_META_OR_DEPLOYED, NOT_IN_META, HOLE_IN_REGION_CHAIN are handled by the HBASE-5128. I believe the others are shortcomings.

          Fulin, give me a few days to get HBASE-5128 in – I'm testing a prerequisite of it currently (HBASE-5589).

          Show
          Jonathan Hsieh added a comment - @ram @fulin I'm working on getting HBASE-5128 into trunk/0.94/0.92/0.90. I'm focusing on trunk first (0.94/0.92 should fall out) and changes from the trunk versions (max merge and specific table safegurds) should make it into the 0.90 version. I believe NOT_IN_META_OR_DEPLOYED, NOT_IN_META, HOLE_IN_REGION_CHAIN are handled by the HBASE-5128 . I believe the others are shortcomings. Fulin, give me a few days to get HBASE-5128 in – I'm testing a prerequisite of it currently ( HBASE-5589 ).
          Hide
          ramkrishna.s.vasudevan added a comment -

          @Fulin
          If you feel it is not present in HBASE-5128 pls add a comment in that JIRA.
          Jonathan can take a look at it and we can take your patch there and integrate it to 0.90?
          Is that fine with you?

          Show
          ramkrishna.s.vasudevan added a comment - @Fulin If you feel it is not present in HBASE-5128 pls add a comment in that JIRA. Jonathan can take a look at it and we can take your patch there and integrate it to 0.90? Is that fine with you?
          Hide
          fulin wang added a comment -

          Yes, I referring to HBASE-5128.
          I want to know when Jonathan H will check in 0.90?

          Show
          fulin wang added a comment - Yes, I referring to HBASE-5128 . I want to know when Jonathan H will check in 0.90?
          Hide
          ramkrishna.s.vasudevan added a comment -

          @Fulin
          Yes, pls follow as Ted suggested.

          Show
          ramkrishna.s.vasudevan added a comment - @Fulin Yes, pls follow as Ted suggested.
          Hide
          Ted Yu added a comment -

          @Fulin:
          Ramkrishna was referring to HBASE-5128.
          Jonathan H would prepare a port of HBASE-5128 to 0.90

          Please participate in the review of HBASE-5128 so that your fix becomes part of it.

          Show
          Ted Yu added a comment - @Fulin: Ramkrishna was referring to HBASE-5128 . Jonathan H would prepare a port of HBASE-5128 to 0.90 Please participate in the review of HBASE-5128 so that your fix becomes part of it.
          Hide
          ramkrishna.s.vasudevan added a comment -

          Is uber-hbck doing this?

          Show
          ramkrishna.s.vasudevan added a comment - Is uber-hbck doing this?

            People

            • Assignee:
              fulin wang
              Reporter:
              fulin wang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development