HBase
  1. HBase
  2. HBASE-5894

Table deletion failed but HBaseAdmin#deletetable reports it as success

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.90.7, 0.92.2, 0.94.0
    • Fix Version/s: 0.94.1, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      all versions

    • Hadoop Flags:
      Reviewed

      Description

      Reproduce this issue by following steps:
      For reproduce it I add this code in DeleteTableHandler#handleTableOperation():

            LOG.debug("Deleting region " + region.getRegionNameAsString() +
              " from META and FS");
      	  +if (true) {
      	  +  throw new IOException("ERROR");
      	  +}
            // Remove region from META
            MetaEditor.deleteRegion(this.server.getCatalogTracker(), region);
      

      step1:create a table and disable it.
      step2:delete it by HBaseAdmin#deleteTable() API.

      result:after lone time, The log say the Table has been deleted, but in fact if we do "list" in shell,the table also exists.

      1. HBASE-5894_trunk_patch_v3.patch
        1 kB
        xufeng
      2. HBASE-5894_trunk_patch_v2.patch
        1 kB
        xufeng
      3. HBASE-5894_trunk_patch_v2_surefire-report.html
        289 kB
        xufeng
      4. HBASE-5894_trunk_patch_v1.patch
        1 kB
        xufeng
      5. HBASE-5894_trunk_patch_v1_surefire-report.html
        286 kB
        xufeng
      6. HBASE-5894_94_patch_v2.patch
        1 kB
        xufeng
      7. HBASE-5894_94_patch_v1.patch
        1 kB
        xufeng
      8. HBASE-5894_94_patch_v1_surefire-report.html
        280 kB
        xufeng
      9. HBASE-5894_92_patch_v2.patch
        1 kB
        xufeng
      10. HBASE-5894_92_patch_v1.patch
        1 kB
        xufeng
      11. HBASE-5894_92_patch_v1_surefire-report.html
        281 kB
        xufeng
      12. HBASE-5894_90_patch_v2.patch
        1 kB
        xufeng
      13. HBASE-5894_90_patch_v1.patch
        1 kB
        xufeng
      14. HBASE-5894_90_patch_v1_surefire-report.html
        201 kB
        xufeng

        Activity

        Hide
        xufeng added a comment -

        I review the code and ensure that all versions have this problem.

        Show
        xufeng added a comment - I review the code and ensure that all versions have this problem.
        Hide
        xufeng added a comment -

        Patch for trunk and verified in real cluster.
        I will submit the patch for 90,92 and 94 after verified in real cluster and done the unit test.

        Show
        xufeng added a comment - Patch for trunk and verified in real cluster. I will submit the patch for 90,92 and 94 after verified in real cluster and done the unit test.
        Hide
        xufeng added a comment -

        Please review the patch and give me some suggestion,thanks.

        Show
        xufeng added a comment - Please review the patch and give me some suggestion,thanks.
        Hide
        Ted Yu added a comment -
        +      throw new RegionException("Retries exhausted, it took too long to wait"+
        

        I think IOException should be thrown above.

        If a test can be added to verify the fix, that would be nice.

        Show
        Ted Yu added a comment - + throw new RegionException( "Retries exhausted, it took too long to wait" + I think IOException should be thrown above. If a test can be added to verify the fix, that would be nice.
        Hide
        xufeng added a comment -

        @Ted
        The for loop in HBaseAdmin#deleteTable() cost too long time.
        In fact When I verified this patch,My client code spent 50 min until return.

        In test,If one test cost more than 900s(the value of forkedProcessTimeoutInSeconds defined in pom.xml)
        then this test will be killed and the others also will not run,is it?

        Show
        xufeng added a comment - @Ted The for loop in HBaseAdmin#deleteTable() cost too long time. In fact When I verified this patch,My client code spent 50 min until return. In test,If one test cost more than 900s(the value of forkedProcessTimeoutInSeconds defined in pom.xml) then this test will be killed and the others also will not run,is it?
        Hide
        Ted Yu added a comment -

        You can use the following annotation to limit the duration for a particular test:

          @Test(timeout = 120000)
        
        Show
        Ted Yu added a comment - You can use the following annotation to limit the duration for a particular test: @Test(timeout = 120000)
        Hide
        xufeng added a comment -

        @Ted
        I have updated the trunk patch and created patch of the other versions,please review,thanks.

        All patch verified in real cluster by my reproduce steps.

        Show
        xufeng added a comment - @Ted I have updated the trunk patch and created patch of the other versions,please review,thanks. All patch verified in real cluster by my reproduce steps.
        Hide
        xufeng added a comment -

        @Ted
        >>If a test can be added to verify the fix, that would be nice.

        In fact,I found this issue in TestLogRolling in 90 version.

        One day the test be killed,I tracked it and found that it timeout in deleteTable() becauseof the dfs problem.

        I think that it is hard to add the new test to verify the fix,because it will cost long time, the test will be killed or timeout(if we add @Test(timeout = 120000)).

        Maybe I am wrong,Can you suggest to me?

        Show
        xufeng added a comment - @Ted >>If a test can be added to verify the fix, that would be nice. In fact,I found this issue in TestLogRolling in 90 version. One day the test be killed,I tracked it and found that it timeout in deleteTable() becauseof the dfs problem. I think that it is hard to add the new test to verify the fix,because it will cost long time, the test will be killed or timeout(if we add @Test(timeout = 120000)). Maybe I am wrong,Can you suggest to me?
        Hide
        xufeng added a comment -

        Hi
        Anyone can review it and give me some suggestion?

        Show
        xufeng added a comment - Hi Anyone can review it and give me some suggestion?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525286/HBASE-5894_trunk_patch_v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1759//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1759//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1759//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525286/HBASE-5894_trunk_patch_v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1759//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1759//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1759//console This message is automatically generated.
        Hide
        stack added a comment -

        +1 on patch

        Show
        stack added a comment - +1 on patch
        Hide
        Ted Yu added a comment -

        I got the following when trying to apply the patch:

        Patching file src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java using Plan A...
        Hunk #1 failed at 511.
        Hunk #2 failed at 543.
        Hunk #3 failed at 578.
        3 out of 3 hunks failed--saving rejects to src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java.rej
        

        Maybe due to the ^M's in the patch ?

                }^M
              }^M
        
        Show
        Ted Yu added a comment - I got the following when trying to apply the patch: Patching file src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java using Plan A... Hunk #1 failed at 511. Hunk #2 failed at 543. Hunk #3 failed at 578. 3 out of 3 hunks failed--saving rejects to src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java.rej Maybe due to the ^M's in the patch ? }^M }^M
        Hide
        xufeng added a comment -

        @Ted
        Which patch has this problem?

        Show
        xufeng added a comment - @Ted Which patch has this problem?
        Hide
        Ted Yu added a comment -

        HBASE-5894_trunk_patch_v2.patch

        Show
        Ted Yu added a comment - HBASE-5894 _trunk_patch_v2.patch
        Hide
        xufeng added a comment -

        @Ted
        I updated the trunk patch,can you apply this patch now?
        if ok,I will check the others and update them.

        Show
        xufeng added a comment - @Ted I updated the trunk patch,can you apply this patch now? if ok,I will check the others and update them.
        Hide
        xufeng added a comment -

        @Ted
        I also updated the others.
        Maybe the patch created by eclipse has this problem.
        Now I create the patch by TortoiseSVN.

        Show
        xufeng added a comment - @Ted I also updated the others. Maybe the patch created by eclipse has this problem. Now I create the patch by TortoiseSVN.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525741/HBASE-5894_94_patch_v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1781//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525741/HBASE-5894_94_patch_v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1781//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525738/HBASE-5894_trunk_patch_v3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
        org.apache.hadoop.hbase.master.TestSplitLogManager

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1780//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1780//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1780//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525738/HBASE-5894_trunk_patch_v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol org.apache.hadoop.hbase.master.TestSplitLogManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1780//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1780//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1780//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        I tried the two failed tests and they passed with patch v3.

        Integrated to trunk first.

        Show
        Ted Yu added a comment - I tried the two failed tests and they passed with patch v3. Integrated to trunk first.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2852 (See https://builds.apache.org/job/HBase-TRUNK/2852/)
        HBASE-5894 Table deletion failed but HBaseAdmin#deletetable reports it as success (Xufeng) (Revision 1334464)

        Result = SUCCESS
        tedyu :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2852 (See https://builds.apache.org/job/HBase-TRUNK/2852/ ) HBASE-5894 Table deletion failed but HBaseAdmin#deletetable reports it as success (Xufeng) (Revision 1334464) Result = SUCCESS tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        Hide
        Ted Yu added a comment -

        Integrated to 0.90, 0.92 and 0.94 branches.

        Thanks for the patch Xufeng.

        Thanks for the review Stack.

        Show
        Ted Yu added a comment - Integrated to 0.90, 0.92 and 0.94 branches. Thanks for the patch Xufeng. Thanks for the review Stack.
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #183 (See https://builds.apache.org/job/HBase-0.94/183/)
        HBASE-5894 Table deletion failed but HBaseAdmin#deletetable reports it as success (Xufeng) (Revision 1334475)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #183 (See https://builds.apache.org/job/HBase-0.94/183/ ) HBASE-5894 Table deletion failed but HBaseAdmin#deletetable reports it as success (Xufeng) (Revision 1334475) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #399 (See https://builds.apache.org/job/HBase-0.92/399/)
        HBASE-5894 Table deletion failed but HBaseAdmin#deletetable reports it as success (Xufeng) (Revision 1334476)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #399 (See https://builds.apache.org/job/HBase-0.92/399/ ) HBASE-5894 Table deletion failed but HBaseAdmin#deletetable reports it as success (Xufeng) (Revision 1334476) Result = FAILURE tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        Hide
        xufeng added a comment -

        Ted and Stack
        Thanks for your help.

        Show
        xufeng added a comment - Ted and Stack Thanks for your help.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #193 (See https://builds.apache.org/job/HBase-TRUNK-security/193/)
        HBASE-5894 Table deletion failed but HBaseAdmin#deletetable reports it as success (Xufeng) (Revision 1334464)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #193 (See https://builds.apache.org/job/HBase-TRUNK-security/193/ ) HBASE-5894 Table deletion failed but HBaseAdmin#deletetable reports it as success (Xufeng) (Revision 1334464) Result = FAILURE tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #107 (See https://builds.apache.org/job/HBase-0.92-security/107/)
        HBASE-5894 Table deletion failed but HBaseAdmin#deletetable reports it as success (Xufeng) (Revision 1334476)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #107 (See https://builds.apache.org/job/HBase-0.92-security/107/ ) HBASE-5894 Table deletion failed but HBaseAdmin#deletetable reports it as success (Xufeng) (Revision 1334476) Result = FAILURE tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #27 (See https://builds.apache.org/job/HBase-0.94-security/27/)
        HBASE-5894 Table deletion failed but HBaseAdmin#deletetable reports it as success (Xufeng) (Revision 1334475)

        Result = SUCCESS
        tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #27 (See https://builds.apache.org/job/HBase-0.94-security/27/ ) HBASE-5894 Table deletion failed but HBaseAdmin#deletetable reports it as success (Xufeng) (Revision 1334475) Result = SUCCESS tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java

          People

          • Assignee:
            xufeng
            Reporter:
            xufeng
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development