HBase
  1. HBase
  2. HBASE-9084

HBase admin flush has a data loss risk even after HBASE-7671

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 0.95.1, 0.94.10
    • Fix Version/s: None
    • Component/s: regionserver
    • Labels:
      None
    1. hbase-9084v2.patch
      0.8 kB
      chunhui shen
    2. HBASE-9084.txt
      6 kB
      Liang Xie
    3. HBASE-9084-0.94.txt
      6 kB
      Liang Xie

      Activity

      Hide
      Liang Xie added a comment -

      any more comments? or could it go ahead to trunk?

      Show
      Liang Xie added a comment - any more comments? or could it go ahead to trunk?
      Hide
      Ted Yu added a comment -

      lgtm

      Show
      Ted Yu added a comment - lgtm
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12595330/hbase-9084v2.patch
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

      +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 lineLengths. The patch does not introduce lines longer than 100

      +1 site. The mvn site goal succeeds with this patch.

      +1 core tests. The patch passed unit tests in .

      Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//testReport/
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
      Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595330/hbase-9084v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6552//console This message is automatically generated.
      Hide
      Liang Xie added a comment -

      oh, now i totally understand your idea, no doubt, +1 for me

      Show
      Liang Xie added a comment - oh, now i totally understand your idea, no doubt, +1 for me
      Hide
      chunhui shen added a comment -

      Make the simple solution patch

      Liang Xie
      Do you have any new doubt?

      Show
      chunhui shen added a comment - Make the simple solution patch Liang Xie Do you have any new doubt?
      Hide
      chunhui shen added a comment -

      the second flush(issued by hbaseadmin flush side) won't work by

      if (this.rsServices != null && this.rsServices.isAborted()) {
            // Don't flush when server aborting, it's unsafe
            throw new IOException("Aborting flush because server is abortted...");
          }
      

      or

      synchronized (writestate) {
              if (!writestate.flushing && writestate.writesEnabled) {
                this.writestate.flushing = true;
              } else {
                if (LOG.isDebugEnabled()) {
                  LOG.debug("NOT flushing memstore for region " + this
                      + ", flushing=" + writestate.flushing + ", writesEnabled="
                      + writestate.writesEnabled);
                }
                status.abort("Not flushing since "
                    + (writestate.flushing ? "already flushing"
                        : "writes not enabled"));
                return false;
              }
            }
      
      Show
      chunhui shen added a comment - the second flush(issued by hbaseadmin flush side) won't work by if ( this .rsServices != null && this .rsServices.isAborted()) { // Don't flush when server aborting, it's unsafe throw new IOException( "Aborting flush because server is abortted..." ); } or synchronized (writestate) { if (!writestate.flushing && writestate.writesEnabled) { this .writestate.flushing = true ; } else { if (LOG.isDebugEnabled()) { LOG.debug( "NOT flushing memstore for region " + this + ", flushing=" + writestate.flushing + ", writesEnabled=" + writestate.writesEnabled); } status.abort( "Not flushing since " + (writestate.flushing ? "already flushing" : "writes not enabled" )); return false ; } }
      Hide
      chunhui shen added a comment -

      the second flush(issued by hbaseadmin flush side) was successful

      It won't be successful, since we have aborted the server

      Show
      chunhui shen added a comment - the second flush(issued by hbaseadmin flush side) was successful It won't be successful, since we have aborted the server
      Hide
      Liang Xie added a comment -

      emm, imho, it doesn't work. data loss means the second flush(issued by hbaseadmin flush side) was successful, but didn't flush its own memstore snapshot, just flushed the previous &failed old snapshot, then the second flush's data was lost silently...

      so i thought if just add this catch statement is not enough.

      or maybe i didn't get u.

      Show
      Liang Xie added a comment - emm, imho, it doesn't work. data loss means the second flush(issued by hbaseadmin flush side) was successful, but didn't flush its own memstore snapshot, just flushed the previous &failed old snapshot, then the second flush's data was lost silently... so i thought if just add this catch statement is not enough. or maybe i didn't get u.
      Hide
      chunhui shen added a comment -

      It means move the call abort code to HRegion, thus "due to "this.rsServices != null && this.rsServices.isAborted()" is not be set " should be impossible.

      Correct me if something I miss

      Show
      chunhui shen added a comment - It means move the call abort code to HRegion, thus "due to "this.rsServices != null && this.rsServices.isAborted()" is not be set " should be impossible. Correct me if something I miss
      Hide
      chunhui shen added a comment -

      Seems have such a risk.

      What about using the following fix:

      try {
              boolean result = internalFlushcache(status);
      
              if (coprocessorHost != null) {
                status.setStatus("Running post-flush coprocessor hooks");
                coprocessorHost.postFlush();
              }
      
              status.markComplete("Flush successful");
              return result;
            }catch(DroppedSnapshotException ex){
      if(rsServices!=null){
      rsServices.abort("Replay of HLog required. Forcing server shutdown", ex);
      }
      return false;
      }
      
      

      I think the above code could fix the hole

      Show
      chunhui shen added a comment - Seems have such a risk. What about using the following fix: try { boolean result = internalFlushcache(status); if (coprocessorHost != null ) { status.setStatus( "Running post-flush coprocessor hooks" ); coprocessorHost.postFlush(); } status.markComplete( "Flush successful" ); return result; } catch (DroppedSnapshotException ex){ if (rsServices!= null ){ rsServices.abort( "Replay of HLog required. Forcing server shutdown" , ex); } return false ; } I think the above code could fix the hole
      Hide
      Liang Xie added a comment -

      since after setting closing flag, all the following put requestes will fail immediately, there is no neccessray to verify the put-scan result at all.

      Show
      Liang Xie added a comment - since after setting closing flag, all the following put requestes will fail immediately, there is no neccessray to verify the put-scan result at all.
      Hide
      Ted Yu added a comment -
      -    // Let us try to split and recover
      

      Why removing this chunk of test ?

      Show
      Ted Yu added a comment - - // Let us try to split and recover Why removing this chunk of test ?
      Hide
      Hadoop QA added a comment -

      +1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12594889/HBASE-9084.txt
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      +1 tests included. The patch appears to include 3 new or modified tests.

      +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

      +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 lineLengths. The patch does not introduce lines longer than 100

      +1 site. The mvn site goal succeeds with this patch.

      +1 core tests. The patch passed unit tests in .

      Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//testReport/
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
      Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594889/HBASE-9084.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6519//console This message is automatically generated.
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12594889/HBASE-9084.txt
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      +1 tests included. The patch appears to include 3 new or modified tests.

      +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

      +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 lineLengths. The patch does not introduce lines longer than 100

      -1 site. The patch appears to cause mvn site goal to fail.

      +1 core tests. The patch passed unit tests in .

      Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//testReport/
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
      Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
      Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594889/HBASE-9084.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6518//console This message is automatically generated.

        People

        • Assignee:
          Liang Xie
          Reporter:
          Liang Xie
        • Votes:
          0 Vote for this issue
          Watchers:
          8 Start watching this issue

          Dates

          • Created:
            Updated:

            Development