Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4466

Remove the deadlock from AbstractDelegationTokenSecretManager

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: namenode, security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In HDFS-3374, new synchronization in AbstractDelegationTokenSecretManager.ExpiredTokenRemover was added to make sure the ExpiredTokenRemover thread can be interrupted in time. Otherwise TestDelegation fails intermittently because the MiniDFScluster thread could be shut down before tokenRemover thread.
      However, as Todd pointed out in HDFS-3374, a potential deadlock was introduced by its patch:

      • FSNamesystem.saveNamespace (holding FSN lock) calls DTSM.saveSecretManagerState (which takes DTSM lock)
      • ExpiredTokenRemover.run (holding DTSM lock) calls rollMasterKey calls updateCurrentKey calls logUpdateMasterKey which takes FSN lock
        So if there is a concurrent saveNamespace at the same tie as the expired token remover runs, it might make the NN deadlock.

      This JIRA is to track the change of removing the possible deadlock from AbstractDelegationTokenSecretManager.

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to branch-1. Thank you Brandon!

          Show
          Suresh Srinivas added a comment - I committed the patch to branch-1. Thank you Brandon!
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch.

          Show
          Suresh Srinivas added a comment - +1 for the patch.
          Hide
          Brandon Li added a comment -

          Test-patch result:

          -1 overall.  
              +1 @author.  The patch does not contain any @author tags.
              +1 tests included.  The patch appears to include 3 new or modified tests.
              +1 javadoc.  The javadoc tool did not generate any warning messages.
              +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
              -1 findbugs.  The patch appears to introduce 208 new Findbugs (version 2.0.0) warnings.
          

          The patch doesn't introduce new findbugs warning.

          Show
          Brandon Li added a comment - Test-patch result: -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 208 new Findbugs (version 2.0.0) warnings. The patch doesn't introduce new findbugs warning.
          Hide
          Brandon Li added a comment -

          Uploaded a patch for branch-1. As it is in trunk, the patch uses an noInterruptsLock object instead of AbstractDelegationTokenSecretManager itself to synchronize interrupting ExpiredTokenRemover thread and editlog update.

          Show
          Brandon Li added a comment - Uploaded a patch for branch-1. As it is in trunk, the patch uses an noInterruptsLock object instead of AbstractDelegationTokenSecretManager itself to synchronize interrupting ExpiredTokenRemover thread and editlog update.

            People

            • Assignee:
              Brandon Li
              Reporter:
              Brandon Li
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development