[HBASE-20975] Lock may not be taken or released while rolling back procedure

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.0.1
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.1.1, 2.0.2
• Component/s: amv2
• Assignee: Allan Yang
• Reporter: Allan Yang
• Created: 30/Jul/18 07:56
• Updated: 14/Aug/18 11:44
• Resolved: 13/Aug/18 12:24

Description

Find this one when investigating HBASE-20921, too.

Here is some code from executeRollback in ProcedureExecutor.java.

    boolean reuseLock = false;
    while (stackTail --> 0) {
      final Procedure proc = subprocStack.get(stackTail);

      LockState lockState;
      //If reuseLock, then don't acquire the lock
      if (!reuseLock && (lockState = acquireLock(proc)) != LockState.LOCK_ACQUIRED) {
        return lockState;
      }

      lockState = executeRollback(proc);
      boolean abortRollback = lockState != LockState.LOCK_ACQUIRED;
      abortRollback |= !isRunning() || !store.isRunning();

      //If the next procedure in the stack is the current one, then reuseLock = true
      reuseLock = stackTail > 0 && (subprocStack.get(stackTail - 1) == proc) && !abortRollback;
      //If reuseLock, don't releaseLock
      if (!reuseLock) {
        releaseLock(proc, false);
      }

      if (abortRollback) {
        return lockState;
      }

      subprocStack.remove(stackTail);

      if (proc.isYieldAfterExecutionStep(getEnvironment())) {
        return LockState.LOCK_YIELD_WAIT;
      }

      //But, here, lock is released no matter reuseLock is true or false
      if (proc != rootProc) {
        execCompletionCleanup(proc);
      }
    }

You can see my comments in the code above, reuseLock can cause the procedure executing(rollback) without a lock. Though I haven't found any bugs introduced by this issue, it is indeed a potential bug need to fix.
I think we can just remove the reuseLock logic. Acquire and release lock every time.

Find another case that during rolling back, the procedure's lock may not be released properly:
see comment: https://issues.apache.org/jira/browse/HBASE-20975?focusedCommentId=16565123&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16565123