Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22486

Fix flaky test TestLockManager

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.3.0, 2.1.6
    • Component/s: None
    • Labels:
      None

      Description

      When the LockProcedure was scheduled. It will called TimeoutExecutorThread#add method.

      public void add(Procedure<TEnvironment> procedure) {
        assert procedure.getState() == ProcedureState.WAITING_TIMEOUT;
        LOG.info("ADDED {}; timeout={}, timestamp={}", procedure, procedure.getTimeout(),
          procedure.getTimeoutTimestamp());
        queue.add(new DelayedProcedure<>(procedure));
      }
      

      But LockProcedure's unlock method may called before it was scheduled. And it will change the state to RUNNABLE. This will make the assert error.

      // Can be called before procedure gets scheduled, in which case, the execute() will finish
      // immediately and release the underlying locks.
      public void unlock(final MasterProcedureEnv env) {
        unlock.set(true);
        locked.set(false);
        // Maybe timeout already awakened the event and the procedure has finished.
        synchronized (event) {
          if (!event.isReady()) {
            LOG.info("Update procedure {} state to RUNNABLE", this);
            setState(ProcedureProtos.ProcedureState.RUNNABLE);
            event.wake(env.getProcedureScheduler());
          }
        }
      }
      

       

      I added log in these methods.

      2019-05-28 15:34:16,179 INFO [Time-limited test] locking.LockProcedure(223): Update procedure pid=21, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=namespace:table, type=EXCLUSIVE state to RUNNABLE
      2019-05-28 15:34:16,179 DEBUG [Time-limited test] procedure.MasterProcedureScheduler(352): Add TableQueue(namespace:table, xlock=true (21) sharedLock=0 size=1) to run queue because: pid=21, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=namespace:table, type=EXCLUSIVE has lock
      2019-05-28 15:34:16,179 INFO [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1967): ASSERT pid=21
      java.lang.AssertionError
      at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.add(TimeoutExecutorThread.java:82)
      at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1690)
      at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
      at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
      at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1965)

        Attachments

          Activity

            People

            • Assignee:
              zghao Guanghao Zhang
              Reporter:
              zghao Guanghao Zhang

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment