Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-984

Delegation Tokens should be persisted in Namenode

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Layout version is set to -24 reflecting changes in edits log and fsimage format related to persisting delegation tokens.

      Description

      The Delegation tokens should be persisted in the FsImage and EditLogs so that they are valid to be used after namenode shutdown and restart.

      1. HDFS-984-0_20.4.patch
        61 kB
        Jitendra Nath Pandey
      2. HDFS-984.7.patch
        40 kB
        Jitendra Nath Pandey
      3. HDFS-984.14.patch
        42 kB
        Jitendra Nath Pandey
      4. HDFS-984.12.patch
        42 kB
        Jitendra Nath Pandey
      5. HDFS-984.11.patch
        43 kB
        Jitendra Nath Pandey
      6. HDFS-984.10.patch
        41 kB
        Jitendra Nath Pandey

        Issue Links

          Activity

          Eugene Koontz made changes -
          Link This issue is related to HDFS-3627 [ HDFS-3627 ]
          Eli Collins made changes -
          Link This issue is related to HDFS-3486 [ HDFS-3486 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314046 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #252 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/252/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #252 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/252/ )
          Jitendra Nath Pandey made changes -
          Attachment HDFS-984-0_20.4.patch [ 12437292 ]
          Hide
          Jitendra Nath Pandey added a comment -

          20 patch uploaded.
          Combines HADOOP-6573 and MR-1537 too.

          Show
          Jitendra Nath Pandey added a comment - 20 patch uploaded. Combines HADOOP-6573 and MR-1537 too.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #204 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/204/)
          . Persistent delegation tokens. Contributed by Jitendra Pandey.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #204 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/204/ ) . Persistent delegation tokens. Contributed by Jitendra Pandey.
          Konstantin Shvachko made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags [Incompatible change, Reviewed]
          Release Note Layout version is set to -24 reflecting changes in edits log and fsimage format related to persisting delegation tokens.
          Fix Version/s 0.22.0 [ 12314241 ]
          Resolution Fixed [ 1 ]
          Hide
          Konstantin Shvachko added a comment -

          I just committed this. Thank you Jitendra.

          Show
          Konstantin Shvachko added a comment - I just committed this. Thank you Jitendra.
          Hide
          Jitendra Nath Pandey added a comment -

          Other tests also run fine.

          Show
          Jitendra Nath Pandey added a comment - Other tests also run fine.
          Hide
          Konstantin Shvachko added a comment -

          Please run ant test-core. run-test-hdfs target does not cover some tests like fi tests.

          Show
          Konstantin Shvachko added a comment - Please run ant test-core. run-test-hdfs target does not cover some tests like fi tests.
          Hide
          Jitendra Nath Pandey added a comment -

          ant run-test-hdfs ran successfully.

          Show
          Jitendra Nath Pandey added a comment - ant run-test-hdfs ran successfully.
          Hide
          Jitendra Nath Pandey added a comment -

          test patch results-

          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 15 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          Show
          Jitendra Nath Pandey added a comment - test patch results- [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 15 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Jitendra Nath Pandey made changes -
          Attachment HDFS-984.14.patch [ 12436941 ]
          Hide
          Jitendra Nath Pandey added a comment -

          New patch addressing the comments

          Show
          Jitendra Nath Pandey added a comment - New patch addressing the comments
          Hide
          Konstantin Shvachko added a comment -
          • AbstractDelegationTokenSecretManager.renewToken() should be synchronized
          • Same with cancelToken()
          • imports in DelegationTokenSecretManager and Tests

          +1 other than that.

          Show
          Konstantin Shvachko added a comment - AbstractDelegationTokenSecretManager.renewToken() should be synchronized Same with cancelToken() imports in DelegationTokenSecretManager and Tests +1 other than that.
          Jitendra Nath Pandey made changes -
          Attachment HDFS-984.12.patch [ 12436813 ]
          Jitendra Nath Pandey made changes -
          Attachment HDFS-984.11.patch [ 12436777 ]
          Hide
          Jitendra Nath Pandey added a comment -

          New patch addressing the review comments.

          Show
          Jitendra Nath Pandey added a comment - New patch addressing the review comments.
          Hide
          Konstantin Shvachko added a comment -

          Including comments for both issue this one and HADOOP-6573.

          1. Interchanging names logGetDelegationToken() and logAddDelegationToken() are confusing - should be consistent either all Get or all Add. Should OP_GET_DELEGATION_TOKEN be OP_ADD_DELEGATION_TOKEN?
          2. Reuse the instance of DelegationTokenIdentifier in loadFSEdits()
          3. loadFSEdits() should log new token ops in debug mode.
          4. I think we should throw an exception if the same token is added (not renewed) twice.
          5. AbstractDelegationTokenSecretManager uses 2 locks (this and currentTokens) - probably an attempt to do a read/write lock. You don't use them consistently.
            currentTokens.get() returns a reference to the value, not a copy. So when you access the value fields outside the lock they may be changing by other threads. It would be better to have just one lock (this).
          6. No need to make isInSafeMode() public.
          7. Import warnings in FSNamesystem, DelegationTokenSecretManager
          8. @Override // <which class it overrides>
          9. public methods and classes must have JavaDoc.
          Show
          Konstantin Shvachko added a comment - Including comments for both issue this one and HADOOP-6573 . Interchanging names logGetDelegationToken() and logAddDelegationToken() are confusing - should be consistent either all Get or all Add. Should OP_GET_DELEGATION_TOKEN be OP_ADD_DELEGATION_TOKEN? Reuse the instance of DelegationTokenIdentifier in loadFSEdits() loadFSEdits() should log new token ops in debug mode. I think we should throw an exception if the same token is added (not renewed) twice. AbstractDelegationTokenSecretManager uses 2 locks (this and currentTokens) - probably an attempt to do a read/write lock. You don't use them consistently. currentTokens.get() returns a reference to the value, not a copy. So when you access the value fields outside the lock they may be changing by other threads. It would be better to have just one lock (this). No need to make isInSafeMode() public. Import warnings in FSNamesystem, DelegationTokenSecretManager @Override // <which class it overrides> public methods and classes must have JavaDoc.
          Jitendra Nath Pandey made changes -
          Attachment HDFS-984.10.patch [ 12436417 ]
          Hide
          Jitendra Nath Pandey added a comment -

          > Can you please explain the use-case why these need to be persisted?
          Delegation tokens are used by tasks to authenticate with namenode. If namenode restarts and looses information for previously issued delegation tokens, the running tasks will not be able to connect to the namenode.
          The scope of this jira is limited to delegation tokens issued by namenode. The delgation tokens in MR issued by JT are not being persisted.

          Show
          Jitendra Nath Pandey added a comment - > Can you please explain the use-case why these need to be persisted? Delegation tokens are used by tasks to authenticate with namenode. If namenode restarts and looses information for previously issued delegation tokens, the running tasks will not be able to connect to the namenode. The scope of this jira is limited to delegation tokens issued by namenode. The delgation tokens in MR issued by JT are not being persisted.
          Jitendra Nath Pandey made changes -
          Attachment HDFS-984.7.patch [ 12436298 ]
          Hide
          Jitendra Nath Pandey added a comment -

          This patch is dependent on HADOOP-6573 patch to compile successfully.

          Show
          Jitendra Nath Pandey added a comment - This patch is dependent on HADOOP-6573 patch to compile successfully.
          Hide
          dhruba borthakur added a comment -

          Can you please explain the use-case why these need to be persisted?

          Show
          dhruba borthakur added a comment - Can you please explain the use-case why these need to be persisted?
          Jitendra Nath Pandey made changes -
          Field Original Value New Value
          Link This issue is blocked by HADOOP-6573 [ HADOOP-6573 ]
          Jitendra Nath Pandey created issue -

            People

            • Assignee:
              Jitendra Nath Pandey
              Reporter:
              Jitendra Nath Pandey
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development