HBase
  1. HBase
  2. HBASE-10844

Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Promotes an -ea assert to logged FATAL and RS abort when memstore is found to be in an inconsistent state.

      Description

      Observed this in the testing with Phoenix. The test in Phoenix - MutableIndexFailureIT deliberately fails the batchmutation call via the installed coprocessor. But the update is not rolled back. That leaves the memstore inconsistent. In particular, I observed that getFlushableSize is updated before the coprocessor was called but the update is not rolled back. When the region is being closed at some later point, the assert introduced in HBASE-10514 in the HRegion.doClose() causes the RegionServer to shutdown abnormally.

      1. 10844-1.txt
        1 kB
        Devaraj Das
      2. 10844-1-0.98.txt
        1 kB
        Devaraj Das
      3. 10844-v2.patch
        1 kB
        Nick Dimiduk
      4. HBASE-10844.02.patch
        2 kB
        Nick Dimiduk
      5. HBASE-10844.02-branch-1.0.patch
        2 kB
        Nick Dimiduk
      6. HBASE-10844.02-0.98.patch
        2 kB
        Nick Dimiduk

        Activity

        Hide
        Devaraj Das added a comment -

        Straightforward fix.

        Show
        Devaraj Das added a comment - Straightforward fix.
        Hide
        Devaraj Das added a comment -

        I should write a simple unit test. Will do so soon.

        Show
        Devaraj Das added a comment - I should write a simple unit test. Will do so soon.
        Hide
        Andrew Purtell added a comment -

        +1

        Show
        Andrew Purtell added a comment - +1
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12637029/10844-1.txt
        against trunk revision .
        ATTACHMENT ID: 12637029

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 6 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637029/10844-1.txt against trunk revision . ATTACHMENT ID: 12637029 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc . The javadoc tool appears to have generated 6 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9105//console This message is automatically generated.
        Hide
        Anoop Sam John added a comment -

        +1

        Show
        Anoop Sam John added a comment - +1
        Hide
        Anoop Sam John added a comment -

        Need a 94 patch also.

        Show
        Anoop Sam John added a comment - Need a 94 patch also.
        Hide
        ramkrishna.s.vasudevan added a comment -

        +1 on patch.

        Show
        ramkrishna.s.vasudevan added a comment - +1 on patch.
        Hide
        Anoop Sam John added a comment -

        Devaraj Das I guess we can commit this now.

        Show
        Anoop Sam John added a comment - Devaraj Das I guess we can commit this now.
        Hide
        Devaraj Das added a comment -

        Anoop Sam John, thanks for reminding. I'll commit this one today/tomorrow.

        Show
        Devaraj Das added a comment - Anoop Sam John , thanks for reminding. I'll commit this one today/tomorrow.
        Hide
        Lars Hofhansl added a comment -

        I do not think this is right.
        The point of no return is when the WAL is synced. If the region server dies after the sync the edit are replayed and hence the change will be visible. Hence the memstore should not be rolled back after the WAL edit is synced.

        postBatchMutate is "post" the batch mutate, so it cannot undo the mutation, it happened already.

        -1 on this change.

        Show
        Lars Hofhansl added a comment - I do not think this is right. The point of no return is when the WAL is synced. If the region server dies after the sync the edit are replayed and hence the change will be visible. Hence the memstore should not be rolled back after the WAL edit is synced. postBatchMutate is "post" the batch mutate, so it cannot undo the mutation, it happened already. -1 on this change.
        Hide
        Devaraj Das added a comment -

        Thanks Lars Hofhansl for catching this. Am busy since the last few days with some other stuff - will find some time to think about this shortly.

        Show
        Devaraj Das added a comment - Thanks Lars Hofhansl for catching this. Am busy since the last few days with some other stuff - will find some time to think about this shortly.
        Hide
        Lars Hofhansl added a comment -

        We could move the post hook before the syncOrDefer call... But it would not be a "post" hook anymore.
        Or we could explore why the action in question cannot be done in the pre hook.

        Show
        Lars Hofhansl added a comment - We could move the post hook before the syncOrDefer call... But it would not be a "post" hook anymore. Or we could explore why the action in question cannot be done in the pre hook.
        Hide
        Anoop Sam John added a comment -

        No Lars we cannot move before sync. This needs to be a post. Thanks for the catch. I have missed totally that we called sync b4 this cp call

        Show
        Anoop Sam John added a comment - No Lars we cannot move before sync. This needs to be a post. Thanks for the catch. I have missed totally that we called sync b4 this cp call
        Hide
        ramkrishna.s.vasudevan added a comment -

        Yes, remember now that we need cannot revert the memstore as it is synced up.
        sorry for the +1.
        If we set WriteToWal as false then rolling back of the memstore even after post call would make sense, right?

        Show
        ramkrishna.s.vasudevan added a comment - Yes, remember now that we need cannot revert the memstore as it is synced up. sorry for the +1. If we set WriteToWal as false then rolling back of the memstore even after post call would make sense, right?
        Hide
        Lars Hofhansl added a comment -

        If writeToWal is false for all mutations in this batch then the point of no return is the flushing of the memstore.

        Luckily the memstore flusher waits for all priot MVCC transactions to finish. And since in doMiniBatchMutation we call the post hook before mvcc.completeMemstoreInsert(w) we're safe from concurrent flushes.

        Hence in that case it would be OK in theory to fail post hook and then roll back the memstore.

        Show
        Lars Hofhansl added a comment - If writeToWal is false for all mutations in this batch then the point of no return is the flushing of the memstore. Luckily the memstore flusher waits for all priot MVCC transactions to finish. And since in doMiniBatchMutation we call the post hook before mvcc.completeMemstoreInsert(w) we're safe from concurrent flushes. Hence in that case it would be OK in theory to fail post hook and then roll back the memstore.
        Hide
        Anoop Sam John added a comment -

        IMO it will be good to be consistent with the sync wal case. ie even if WriteToWal is false, do not rollback with failures in post hooks. post means the mutation is done.

        Show
        Anoop Sam John added a comment - IMO it will be good to be consistent with the sync wal case. ie even if WriteToWal is false, do not rollback with failures in post hooks. post means the mutation is done.
        Hide
        Lars Hofhansl added a comment -

        Oh, I totally agree. I was just answering Ram's hypothetical question
        A post hook is a post hook, it should not undo anything.

        Show
        Lars Hofhansl added a comment - Oh, I totally agree. I was just answering Ram's hypothetical question A post hook is a post hook, it should not undo anything.
        Hide
        Lars Hofhansl added a comment -

        Should we close this as "invalid"?

        Show
        Lars Hofhansl added a comment - Should we close this as "invalid"?
        Hide
        Devaraj Das added a comment -

        Yes, Lars Hofhansl, I am resolving it as won't-fix.

        Show
        Devaraj Das added a comment - Yes, Lars Hofhansl , I am resolving it as won't-fix.
        Hide
        Devaraj Das added a comment -

        Thinking about it (and in offline discussions with Jeffrey Zhong and Enis Soztutar), it seems we should remove the 'assert' from the code that causes the RegionServer to abort. The reason being that coprocessors could fail. The assert was introduced in HBASE-10514.

                 // close each store in parallel
                 for (final Store store : stores.values()) {
        +          assert abort? true: store.getFlushableSize() == 0;
        

        Thoughts?

        Show
        Devaraj Das added a comment - Thinking about it (and in offline discussions with Jeffrey Zhong and Enis Soztutar ), it seems we should remove the 'assert' from the code that causes the RegionServer to abort. The reason being that coprocessors could fail. The assert was introduced in HBASE-10514 . // close each store in parallel for (final Store store : stores.values()) { + assert abort? true: store.getFlushableSize() == 0; Thoughts?
        Hide
        Devaraj Das added a comment -

        Patch for 0.98.

        Show
        Devaraj Das added a comment - Patch for 0.98.
        Hide
        Andrew Purtell added a comment -

        So with this patch we'd remove the assert and replace it with a warning that memstore datastructures have been only partially updated?

        --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        @@ -1109,7 +1109,13 @@ public class HRegion implements HeapSize { // , Writable{
         
                 // close each store in parallel
                 for (final Store store : stores.values()) {
        -          assert abort? true: store.getFlushableSize() == 0;
        +          if (store.getFlushableSize() != 0) {
        +            LOG.warn("store.getFlushableSize for " + store + " is not zero! It's " 
        +                + store.getFlushableSize() + ". Maybe a coprocessor "
        +                + "operation failed and "
        +                + "left the memstore datastructures in a partially updated state. "
        +                + "Current memstoreSize " + this.getMemstoreSize().get());
        +          }
                   completionService
                       .submit(new Callable<Pair<byte[], Collection<StoreFile>>>() {
                         @Override
        

        Shouldn't we be aborting in that case anyway? Or replace the assert with an abort()?

        Show
        Andrew Purtell added a comment - So with this patch we'd remove the assert and replace it with a warning that memstore datastructures have been only partially updated? --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java @@ -1109,7 +1109,13 @@ public class HRegion implements HeapSize { // , Writable{ // close each store in parallel for ( final Store store : stores.values()) { - assert abort? true : store.getFlushableSize() == 0; + if (store.getFlushableSize() != 0) { + LOG.warn( "store.getFlushableSize for " + store + " is not zero! It's " + + store.getFlushableSize() + ". Maybe a coprocessor " + + "operation failed and " + + "left the memstore datastructures in a partially updated state. " + + "Current memstoreSize " + this .getMemstoreSize().get()); + } completionService .submit( new Callable<Pair< byte [], Collection<StoreFile>>>() { @Override Shouldn't we be aborting in that case anyway? Or replace the assert with an abort()?
        Hide
        Nick Dimiduk added a comment -

        I stumbled into this ticket while helping debug what I suspect to be a misbehaving coprocessor. Updating patch from 0.98 to log the region name in addition to the store name. The assert logic is slightly different on master vs. 0.98, so check me on the writestate portion.

        Show
        Nick Dimiduk added a comment - I stumbled into this ticket while helping debug what I suspect to be a misbehaving coprocessor. Updating patch from 0.98 to log the region name in addition to the store name. The assert logic is slightly different on master vs. 0.98, so check me on the writestate portion.
        Hide
        Nick Dimiduk added a comment -

        I should say – replaces the WARN with an abort(), which is what the assert would do anyway.

        FWIW, we were seeing this WARN triggered during region close.

        Show
        Nick Dimiduk added a comment - I should say – replaces the WARN with an abort(), which is what the assert would do anyway. FWIW, we were seeing this WARN triggered during region close.
        Hide
        Andrew Purtell added a comment -

        +1, the v2 patch addresses my concerns about only warning previously

        Show
        Andrew Purtell added a comment - +1, the v2 patch addresses my concerns about only warning previously
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12749964/10844-v2.patch
        against master branch at commit a78e6e94994aaba2bee7747054ea9a55f1edd421.
        ATTACHMENT ID: 12749964

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javac. The patch appears to cause mvn compile goal to fail with Hadoop version 2.4.0.

        Compilation errors resume:
        [ERROR] COMPILATION ERROR :
        [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:[1450,19] cannot find symbol
        [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:[1451,93] long cannot be dereferenced
        [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project hbase-server: Compilation failure: Compilation failure:
        [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:[1450,19] cannot find symbol
        [ERROR] symbol: method getRegionNameAsString()
        [ERROR] location: class org.apache.hadoop.hbase.regionserver.HRegion
        [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:[1451,93] long cannot be dereferenced
        [ERROR] -> [Help 1]
        [ERROR]
        [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
        [ERROR] Re-run Maven using the -X switch to enable full debug logging.
        [ERROR]
        [ERROR] For more information about the errors and possible solutions, please read the following articles:
        [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
        [ERROR]
        [ERROR] After correcting the problems, you can resume the build with the command
        [ERROR] mvn <goals> -rf :hbase-server

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15057//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12749964/10844-v2.patch against master branch at commit a78e6e94994aaba2bee7747054ea9a55f1edd421. ATTACHMENT ID: 12749964 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause mvn compile goal to fail with Hadoop version 2.4.0. Compilation errors resume: [ERROR] COMPILATION ERROR : [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: [1450,19] cannot find symbol [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: [1451,93] long cannot be dereferenced [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project hbase-server: Compilation failure: Compilation failure: [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: [1450,19] cannot find symbol [ERROR] symbol: method getRegionNameAsString() [ERROR] location: class org.apache.hadoop.hbase.regionserver.HRegion [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: [1451,93] long cannot be dereferenced [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hbase-server Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15057//console This message is automatically generated.
        Hide
        Nick Dimiduk added a comment -

        Cleaned up patch for master. HBASE-10844.02.patch compiles for master, branch-1, branch-1.2, branch-1.1. Patch from master does not cherry-pick cleanly any further back, so branches branch-1.0 and 0.98 get their own patches. Just accounting for minor fuzz in the assert line.

        Show
        Nick Dimiduk added a comment - Cleaned up patch for master. HBASE-10844 .02.patch compiles for master, branch-1, branch-1.2, branch-1.1. Patch from master does not cherry-pick cleanly any further back, so branches branch-1.0 and 0.98 get their own patches. Just accounting for minor fuzz in the assert line.
        Hide
        Andrew Purtell added a comment -

        lgtm

        Show
        Andrew Purtell added a comment - lgtm
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12750403/HBASE-10844.02-0.98.patch
        against 0.98 branch at commit 4dd30ab019cfbf3691fd08f7941d33d8bbc37f05.
        ATTACHMENT ID: 12750403

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 23 warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn post-site goal to fail.

        +1 core tests. The patch passed unit tests in .

        -1 core zombie tests. There are 1 zombie test(s): at org.apache.oozie.test.MiniHCatServer$1.run(MiniHCatServer.java:137)
        at org.apache.oozie.test.XTestCase$MiniClusterShutdownMonitor.run(XTestCase.java:1071)

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/15092//testReport/
        Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15092//artifact/patchprocess/newFindbugsWarnings.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/15092//artifact/patchprocess/checkstyle-aggregate.html

        Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15092//artifact/patchprocess/patchJavadocWarnings.txt
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15092//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12750403/HBASE-10844.02-0.98.patch against 0.98 branch at commit 4dd30ab019cfbf3691fd08f7941d33d8bbc37f05. ATTACHMENT ID: 12750403 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. -1 javadoc . The javadoc tool appears to have generated 23 warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn post-site goal to fail. +1 core tests . The patch passed unit tests in . -1 core zombie tests . There are 1 zombie test(s): at org.apache.oozie.test.MiniHCatServer$1.run(MiniHCatServer.java:137) at org.apache.oozie.test.XTestCase$MiniClusterShutdownMonitor.run(XTestCase.java:1071) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/15092//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15092//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/15092//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15092//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15092//console This message is automatically generated.
        Hide
        Andrew Purtell added a comment -

        Committing shortly

        Show
        Andrew Purtell added a comment - Committing shortly
        Hide
        Andrew Purtell added a comment -

        Pushed to 0.98 and up using above instruction.

        Show
        Andrew Purtell added a comment - Pushed to 0.98 and up using above instruction.
        Hide
        Nick Dimiduk added a comment -
        Show
        Nick Dimiduk added a comment - Thanks Andrew Purtell
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-1.0 #1016 (See https://builds.apache.org/job/HBase-1.0/1016/)
        HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev fbf66b48b3f17c3310ad42b8b274a1ce3020c6b6)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-1.0 #1016 (See https://builds.apache.org/job/HBase-1.0/1016/ ) HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev fbf66b48b3f17c3310ad42b8b274a1ce3020c6b6) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-1.1 #618 (See https://builds.apache.org/job/HBase-1.1/618/)
        HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev e585cd8e18dd8d7f6b738bbad490222c3ab4fcbc)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-1.1 #618 (See https://builds.apache.org/job/HBase-1.1/618/ ) HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev e585cd8e18dd8d7f6b738bbad490222c3ab4fcbc) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-1.3 #111 (See https://builds.apache.org/job/HBase-1.3/111/)
        HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev 0b6d9b25f59f2b5e858d008222084f31b2cee145)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-1.3 #111 (See https://builds.apache.org/job/HBase-1.3/111/ ) HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev 0b6d9b25f59f2b5e858d008222084f31b2cee145) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK #6730 (See https://builds.apache.org/job/HBase-TRUNK/6730/)
        HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev ecba08829da149aca11929b5b88a6df880e0da35)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #6730 (See https://builds.apache.org/job/HBase-TRUNK/6730/ ) HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev ecba08829da149aca11929b5b88a6df880e0da35) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.98 #1083 (See https://builds.apache.org/job/HBase-0.98/1083/)
        HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev a3a2860b16ebaf4f05f9a5eb8b0d2fd22252fe8b)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.98 #1083 (See https://builds.apache.org/job/HBase-0.98/1083/ ) HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev a3a2860b16ebaf4f05f9a5eb8b0d2fd22252fe8b) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-1.2-IT #93 (See https://builds.apache.org/job/HBase-1.2-IT/93/)
        HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev adcb905a0d30bdd27aa7439e336e0cf95f6dbae9)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-1.2-IT #93 (See https://builds.apache.org/job/HBase-1.2-IT/93/ ) HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev adcb905a0d30bdd27aa7439e336e0cf95f6dbae9) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-1.2 #112 (See https://builds.apache.org/job/HBase-1.2/112/)
        HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev adcb905a0d30bdd27aa7439e336e0cf95f6dbae9)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-1.2 #112 (See https://builds.apache.org/job/HBase-1.2/112/ ) HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev adcb905a0d30bdd27aa7439e336e0cf95f6dbae9) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-1.3-IT #93 (See https://builds.apache.org/job/HBase-1.3-IT/93/)
        HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev 0b6d9b25f59f2b5e858d008222084f31b2cee145)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-1.3-IT #93 (See https://builds.apache.org/job/HBase-1.3-IT/93/ ) HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev 0b6d9b25f59f2b5e858d008222084f31b2cee145) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1036 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1036/)
        HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev a3a2860b16ebaf4f05f9a5eb8b0d2fd22252fe8b)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1036 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1036/ ) HBASE-10844 Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state (apurtell: rev a3a2860b16ebaf4f05f9a5eb8b0d2fd22252fe8b) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Nick Dimiduk added a comment -

        I found a good build from this commit, it was down in https://builds.apache.org/job/HBase-1.2/jdk=latest1.7,label=Hadoop/112/. Everything else looks like OOM and hung tests.

        Show
        Nick Dimiduk added a comment - I found a good build from this commit, it was down in https://builds.apache.org/job/HBase-1.2/jdk=latest1.7,label=Hadoop/112/ . Everything else looks like OOM and hung tests.

          People

          • Assignee:
            Nick Dimiduk
            Reporter:
            Devaraj Das
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development