Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0
    • Component/s: Client, regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
      Similar to what we do in doMiniBatchPut, we should create the log entry with the lock held, but only sync the HLog after the lock is released, along with rollback logic in case the sync'ing fails.

      1. 5541.txt
        5 kB
        Lars Hofhansl
      2. 5541-v2.txt
        5 kB
        Lars Hofhansl

        Issue Links

          Activity

          Hide
          Ted Yu added a comment -

          after the log is released

          I guess you meant 'after the lock is released'

          Show
          Ted Yu added a comment - after the log is released I guess you meant 'after the lock is released'
          Hide
          Lars Hofhansl added a comment -

          Indeed. Fixed. Thanks for pointing out Ted.

          Show
          Lars Hofhansl added a comment - Indeed. Fixed. Thanks for pointing out Ted.
          Hide
          Lars Hofhansl added a comment -

          Here's a patch.
          TestAtomicOperation still passes fine.

          Show
          Lars Hofhansl added a comment - Here's a patch. TestAtomicOperation still passes fine.
          Hide
          Ted Yu added a comment -

          There're two step 11 now:

          +        if (acquiredLocks != null) {
          +          // 11. release the row lock
          
          Show
          Ted Yu added a comment - There're two step 11 now: + if (acquiredLocks != null ) { + // 11. release the row lock
          Hide
          Lars Hofhansl added a comment -

          Whoops. Yes, that comment should've been removed.

          Show
          Lars Hofhansl added a comment - Whoops. Yes, that comment should've been removed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12517600/5541.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated -129 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.coprocessor.TestMasterObserver
          org.apache.hadoop.hbase.mapreduce.TestImportTsv
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1136//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1136//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1136//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517600/5541.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -129 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1136//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1136//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1136//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          Looks good to me. Any objections to committing this?

          Show
          Lars Hofhansl added a comment - Looks good to me. Any objections to committing this?
          Hide
          Ted Yu added a comment -

          Redundant step 11 comment would be removed, right ?

          Show
          Ted Yu added a comment - Redundant step 11 comment would be removed, right ?
          Hide
          Lars Hofhansl added a comment -

          Yep

          Show
          Lars Hofhansl added a comment - Yep
          Hide
          Lars Hofhansl added a comment -

          I'll wait for an official +1.

          Show
          Lars Hofhansl added a comment - I'll wait for an official +1.
          Hide
          Ted Yu added a comment -

          +1 from me.

          Show
          Ted Yu added a comment - +1 from me.
          Hide
          Lars Hofhansl added a comment -

          Just comment changes

          Show
          Lars Hofhansl added a comment - Just comment changes
          Hide
          Lars Hofhansl added a comment -

          Committed 0.94 and trunk

          Show
          Lars Hofhansl added a comment - Committed 0.94 and trunk
          Hide
          stack added a comment -

          Why we have another variable for saying if locked? Can't we ask the locked object if locked?

                 this.updatesLock.readLock().lock();
          +      locked = true;
          

          this.updatesLock.readLock().lock(); returns a lock. Use that? Set it to null when unlocked? If non-null in finally, unlock it?

          Hmm... you are trying to make it same as doMiniBatchPut... ok. Ignore above.

          Could this patch not make it so doMiniBatchPut and the update to this method shared code? They seem pretty similar?

          This was done in a finally block:

          -        mvcc.completeMemstoreInsert(w);
          

          Its ok that its no longer in a finally block? Could we skip out during the sync w/o updating this? I suppose thats ok... then these new edits won't be seen by a running scanner? If exception up in cp calling postdelete, postput, we'll rollback edits from memstore but mvcc has been advanced and won't go backward? I see you update mvcc if w != null... but maybe the update of mvcc should happen after call to cp both here and in the doMiniBatchPut?

          Otherwise +1 on commit

          Show
          stack added a comment - Why we have another variable for saying if locked? Can't we ask the locked object if locked? this .updatesLock.readLock().lock(); + locked = true ; this.updatesLock.readLock().lock(); returns a lock. Use that? Set it to null when unlocked? If non-null in finally, unlock it? Hmm... you are trying to make it same as doMiniBatchPut... ok. Ignore above. Could this patch not make it so doMiniBatchPut and the update to this method shared code? They seem pretty similar? This was done in a finally block: - mvcc.completeMemstoreInsert(w); Its ok that its no longer in a finally block? Could we skip out during the sync w/o updating this? I suppose thats ok... then these new edits won't be seen by a running scanner? If exception up in cp calling postdelete, postput, we'll rollback edits from memstore but mvcc has been advanced and won't go backward? I see you update mvcc if w != null... but maybe the update of mvcc should happen after call to cp both here and in the doMiniBatchPut? Otherwise +1 on commit
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12517655/5541-v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated -123 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 159 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestImportTsv

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1142//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1142//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1142//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517655/5541-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -123 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 159 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1142//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1142//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1142//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #22 (See https://builds.apache.org/job/HBase-0.94/22/)
          HBASE-5541 Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks (Revision 1298680)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #22 (See https://builds.apache.org/job/HBase-0.94/22/ ) HBASE-5541 Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks (Revision 1298680) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          Lars Hofhansl added a comment -

          @Stack: Sorry should've waited with the commit a bit.

          The mvcc.completeMemstoreInsert(w); is still done in the finally block. It's attempted once in the try block and then if w is still not-null in the finally block, it is redone (that means the try block didn't get to that statement, because the WAL sync failed).

          Show
          Lars Hofhansl added a comment - @Stack: Sorry should've waited with the commit a bit. The mvcc.completeMemstoreInsert(w); is still done in the finally block. It's attempted once in the try block and then if w is still not-null in the finally block, it is redone (that means the try block didn't get to that statement, because the WAL sync failed).
          Hide
          Lars Hofhansl added a comment -

          Re: Duplicate code. Yeah there is a bunch of that.
          Between mutateRowsWithLocks, doMiniBatchPut, and processRow there is lot of duplicated code.
          Scott started to make it better in HBASE-5542. Need to go through and see how we can share code from doMiniBatchPut too.

          They are slightly different, though. doMiniBatchPut does a best effort write. If some puts fail it continues with the ones that succeed. For mutateRowsWithLocks and processRow all ops need to succeed.

          Show
          Lars Hofhansl added a comment - Re: Duplicate code. Yeah there is a bunch of that. Between mutateRowsWithLocks, doMiniBatchPut, and processRow there is lot of duplicated code. Scott started to make it better in HBASE-5542 . Need to go through and see how we can share code from doMiniBatchPut too. They are slightly different, though. doMiniBatchPut does a best effort write. If some puts fail it continues with the ones that succeed. For mutateRowsWithLocks and processRow all ops need to succeed.
          Hide
          Lars Hofhansl added a comment -

          @Stack: you OK with the explanation above?

          Show
          Lars Hofhansl added a comment - @Stack: you OK with the explanation above?
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #132 (See https://builds.apache.org/job/HBase-TRUNK-security/132/)
          HBASE-5541 Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks (Revision 1298678)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #132 (See https://builds.apache.org/job/HBase-TRUNK-security/132/ ) HBASE-5541 Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks (Revision 1298678) Result = FAILURE larsh : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          stack added a comment -

          Commit is fine.

          What about case where the edits are applied, sync succeeds, but the cp postdelete and postput fail? Then mvcc will have been updated but we have removed the edits from memstore.

          i suppose its ok? The read point is advanced but if we take this edit in isolation, no mutations made it through?

          Show
          stack added a comment - Commit is fine. What about case where the edits are applied, sync succeeds, but the cp postdelete and postput fail? Then mvcc will have been updated but we have removed the edits from memstore. i suppose its ok? The read point is advanced but if we take this edit in isolation, no mutations made it through?
          Hide
          Lars Hofhansl added a comment -

          When we get to the CP post hooks all locks are released, the WAL is synched, and the mvcc is rolled forward. No edit will be removed from the memstore (that only happens when the wal sync fails).
          What that means is that a post hook cannot undo the operation (the same as standalone Put and Delete).

          I spent some time trying to poke holes in the design, I think it is tight.

          Show
          Lars Hofhansl added a comment - When we get to the CP post hooks all locks are released, the WAL is synched, and the mvcc is rolled forward. No edit will be removed from the memstore (that only happens when the wal sync fails). What that means is that a post hook cannot undo the operation (the same as standalone Put and Delete). I spent some time trying to poke holes in the design, I think it is tight.
          Hide
          stack added a comment -

          Grand

          Show
          stack added a comment - Grand
          Hide
          Lars Hofhansl added a comment -

          That is not to say that there are no holes.

          Show
          Lars Hofhansl added a comment - That is not to say that there are no holes.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2675 (See https://builds.apache.org/job/HBase-TRUNK/2675/)
          HBASE-5541 Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks (Revision 1298678)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2675 (See https://builds.apache.org/job/HBase-TRUNK/2675/ ) HBASE-5541 Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks (Revision 1298678) Result = SUCCESS larsh : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java

            People

            • Assignee:
              Lars Hofhansl
              Reporter:
              Lars Hofhansl
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development