Hadoop Common
  1. Hadoop Common
  2. HADOOP-6939

Inconsistent lock ordering in AbstractDelegationTokenSecretManager

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      AbstractDelegationTokenSecretManager.startThreads() is synchronized, which calls updateCurrentKey(), which calls logUpdateMasterKey. logUpdateMasterKey's implementation for HDFS's manager calls namesystem.logUpdateMasterKey() which is synchronized. Thus the lock order is ADTSM -> FSN. In FSN.saveNamespace, though, it calls DTSM.saveSecretManagerState(), so the lock order is FSN -> ADTSM.

      I don't think this deadlock occurs in practice since saveNamespace won't occur until after the ADTSM has started its threads, but should be fixed anyway.

      1. HADOOP-6939.patch
        0.9 kB
        Tom White
      2. hadoop-6939.txt
        1.0 kB
        Todd Lipcon
      3. lockorder.png
        51 kB
        Todd Lipcon

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #535 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/535/)
        HADOOP-6939. Inconsistent lock ordering in AbstractDelegationTokenSecretManager. Contributed by Todd Lipcon.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #535 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/535/ ) HADOOP-6939 . Inconsistent lock ordering in AbstractDelegationTokenSecretManager. Contributed by Todd Lipcon.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #451 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/451/)
        HADOOP-6939. Inconsistent lock ordering in AbstractDelegationTokenSecretManager. Contributed by Todd Lipcon.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #451 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/451/ ) HADOOP-6939 . Inconsistent lock ordering in AbstractDelegationTokenSecretManager. Contributed by Todd Lipcon.
        Tom White made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.23.0 [ 12315569 ]
        Resolution Fixed [ 1 ]
        Hide
        Tom White added a comment -

        I've just committed this. Thanks Todd!

        (I re-ran the tests that failed for hudson and they all passed for me.)

        Show
        Tom White added a comment - I've just committed this. Thanks Todd! (I re-ran the tests that failed for hudson and they all passed for me.)
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12464925/HADOOP-6939.patch
        against trunk revision 1039959.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.fs.TestLocalFSFileContextMainOperations
        org.apache.hadoop.fs.TestLocalFSFileContextSymlink
        org.apache.hadoop.fs.TestTrash
        org.apache.hadoop.io.TestSequenceFile

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/126//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/126//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/126//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12464925/HADOOP-6939.patch against trunk revision 1039959. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.fs.TestLocalFSFileContextMainOperations org.apache.hadoop.fs.TestLocalFSFileContextSymlink org.apache.hadoop.fs.TestTrash org.apache.hadoop.io.TestSequenceFile +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/126//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/126//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/126//console This message is automatically generated.
        Tom White made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tom White made changes -
        Attachment HADOOP-6939.patch [ 12464925 ]
        Hide
        Tom White added a comment -

        Fixed patch.

        Show
        Tom White added a comment - Fixed patch.
        Tom White made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Tom White added a comment -

        This patch doesn't compile, can you regenerate it please?

        Show
        Tom White added a comment - This patch doesn't compile, can you regenerate it please?
        Tom White made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags [Reviewed]
        Hide
        Tom White added a comment -

        Re-running through hudson.

        Show
        Tom White added a comment - Re-running through hudson.
        Tom White made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Tom White added a comment -

        +1

        Show
        Tom White added a comment - +1
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12453840/hadoop-6939.txt
        against trunk revision 1031422.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The patch appears to cause tar ant target to fail.

        -1 findbugs. The patch appears to cause Findbugs to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        -1 system test framework. The patch failed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/19//testReport/
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/19//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12453840/hadoop-6939.txt against trunk revision 1031422. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/19//testReport/ Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/19//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12453840/hadoop-6939.txt
        against trunk revision 995285.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The patch appears to cause tar ant target to fail.

        -1 findbugs. The patch appears to cause Findbugs to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        -1 system tests framework. The patch failed system tests framework compile.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/45/testReport/
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/45/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/45/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12453840/hadoop-6939.txt against trunk revision 995285. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. -1 system tests framework. The patch failed system tests framework compile. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/45/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/45/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/45/console This message is automatically generated.
        Todd Lipcon made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Todd Lipcon made changes -
        Assignee Todd Lipcon [ tlipcon ]
        Todd Lipcon made changes -
        Attachment hadoop-6939.txt [ 12453840 ]
        Hide
        Todd Lipcon added a comment -

        Attached patch fixes the issue (verified before/after with a jcarder run of TestCheckPointForSecurityTokens)

        Show
        Todd Lipcon added a comment - Attached patch fixes the issue (verified before/after with a jcarder run of TestCheckPointForSecurityTokens)
        Todd Lipcon made changes -
        Field Original Value New Value
        Attachment lockorder.png [ 12453837 ]
        Hide
        Todd Lipcon added a comment -

        jcarder output from 0.20+security

        Show
        Todd Lipcon added a comment - jcarder output from 0.20+security
        Todd Lipcon created issue -

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development