Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-992

Re-factor block access token implementation to conform to the generic Token interface in Common

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This makes it possible to use block access token as shared key for client-to-datanode authentication over RPC. However, access authorization is still based on block access token semantics.

      1. h992-BK-0.20-07.patch
        177 kB
        Kan Zhang
      2. h992-BK-0.20-07.1.patch
        2 kB
        Jitendra Nath Pandey
      3. h992-29.patch
        185 kB
        Jitendra Nath Pandey
      4. h992-28.patch
        185 kB
        Jitendra Nath Pandey
      5. h992-27.patch
        185 kB
        Jitendra Nath Pandey
      6. h992-26.patch
        184 kB
        Kan Zhang
      7. h992-23.patch
        179 kB
        Kan Zhang
      8. h992-21.patch
        179 kB
        Kan Zhang
      9. h992-20.patch
        179 kB
        Kan Zhang
      10. h992-18.patch
        176 kB
        Kan Zhang
      11. h992-12.patch
        175 kB
        Kan Zhang

        Issue Links

          Activity

          Kan Zhang created issue -
          Kan Zhang made changes -
          Field Original Value New Value
          Link This issue is blocked by HADOOP-6581 [ HADOOP-6581 ]
          Kan Zhang made changes -
          Link This issue is related to HADOOP-4487 [ HADOOP-4487 ]
          Kan Zhang made changes -
          Link This issue is related to HADOOP-4359 [ HADOOP-4359 ]
          Hide
          Kan Zhang added a comment -

          Adding a patch that
          1. added BlockTokenIdentifier, removed BlockAccessToken, the new block token (used to be called access token) will just be Token<BlockTokenIdentifier>.
          2. Refactored AccessTokenHandler to be BlockTokenSecretManager and added BlockTokenSelector so that a block token can be used for authentication over RPC.
          3. Enabled ClientDatanodeProtocol to use block token for authentication.
          4. Added authorization checking based on the authenticated BlockTokenIdentifier at Datanode.

          Show
          Kan Zhang added a comment - Adding a patch that 1. added BlockTokenIdentifier, removed BlockAccessToken, the new block token (used to be called access token) will just be Token<BlockTokenIdentifier>. 2. Refactored AccessTokenHandler to be BlockTokenSecretManager and added BlockTokenSelector so that a block token can be used for authentication over RPC. 3. Enabled ClientDatanodeProtocol to use block token for authentication. 4. Added authorization checking based on the authenticated BlockTokenIdentifier at Datanode.
          Kan Zhang made changes -
          Attachment h992-12.patch [ 12436475 ]
          Hide
          Kan Zhang added a comment -

          ant run-test-hdfs, all passed, but one TestFiRename.java didn't compile.

          compile-fault-inject:
               [echo] Start weaving aspects in place
               [iajc] error at public class TestRename {
               [iajc]              ^^^^^^^^^
               [iajc] /export/crawlspace/kan/6666/hdfs/trunk/src/test/aop/org/apache/hadoop/fs/TestFiRename.java:50:0::0 The public type TestRename must be defined in its own file
               [iajc] MessageHolder:  (220 info)  (1 error) 
               [iajc] [error   0]: error at public class TestRename {
               [iajc]              ^^^^^^^^^
               [iajc] /export/crawlspace/kan/6666/hdfs/trunk/src/test/aop/org/apache/hadoop/fs/TestFiRename.java:50:0::0 The public type TestRename must be defined in its own file
          
          Show
          Kan Zhang added a comment - ant run-test-hdfs, all passed, but one TestFiRename.java didn't compile. compile-fault-inject: [echo] Start weaving aspects in place [iajc] error at public class TestRename { [iajc] ^^^^^^^^^ [iajc] /export/crawlspace/kan/6666/hdfs/trunk/src/test/aop/org/apache/hadoop/fs/TestFiRename.java:50:0::0 The public type TestRename must be defined in its own file [iajc] MessageHolder: (220 info) (1 error) [iajc] [error 0]: error at public class TestRename { [iajc] ^^^^^^^^^ [iajc] /export/crawlspace/kan/6666/hdfs/trunk/src/test/aop/org/apache/hadoop/fs/TestFiRename.java:50:0::0 The public type TestRename must be defined in its own file
          Hide
          Kan Zhang added a comment -

          new patch that fixed the datanode to use BlockTokenSecretManager to authenticate RPC clients. Ran "ant test" manually and passed.

          Show
          Kan Zhang added a comment - new patch that fixed the datanode to use BlockTokenSecretManager to authenticate RPC clients. Ran "ant test" manually and passed.
          Kan Zhang made changes -
          Attachment h992-17.patch [ 12436801 ]
          Kan Zhang made changes -
          Attachment h992-17.patch [ 12436801 ]
          Kan Zhang made changes -
          Attachment h992-18.patch [ 12436805 ]
          Hide
          Kan Zhang added a comment -

          ran "ant run-test-hdfs" and passed. But hdfsproxy test fails with the following message.

          /Users/kan/6666/hdfs/trunk/build.xml:588: The following error occurred while executing this line:
          /Users/kan/6666/hdfs/trunk/build.xml:569: The following error occurred while executing this line:
          /Users/kan/6666/hdfs/trunk/src/contrib/build.xml:48: The following error occurred while executing this line:
          /Users/kan/6666/hdfs/trunk/src/contrib/hdfsproxy/build.xml:292: org.codehaus.cargo.container.ContainerException: Failed to download [http://apache.osuosl.org/tomcat/tomcat-6/v6.0.18/bin/apache-tomcat-6.0.18.zip]
          
          Show
          Kan Zhang added a comment - ran "ant run-test-hdfs" and passed. But hdfsproxy test fails with the following message. /Users/kan/6666/hdfs/trunk/build.xml:588: The following error occurred while executing this line: /Users/kan/6666/hdfs/trunk/build.xml:569: The following error occurred while executing this line: /Users/kan/6666/hdfs/trunk/src/contrib/build.xml:48: The following error occurred while executing this line: /Users/kan/6666/hdfs/trunk/src/contrib/hdfsproxy/build.xml:292: org.codehaus.cargo.container.ContainerException: Failed to download [http: //apache.osuosl.org/tomcat/tomcat-6/v6.0.18/bin/apache-tomcat-6.0.18.zip]
          Kan Zhang made changes -
          Attachment h992-20.patch [ 12437073 ]
          Hide
          Kan Zhang added a comment -

          the latest patch (h992-20.patch) changes checkAccess() from return boolean to return void and throws exception when check fails.

          Show
          Kan Zhang added a comment - the latest patch (h992-20.patch) changes checkAccess() from return boolean to return void and throws exception when check fails.
          Hide
          Kan Zhang added a comment -

          fixed a findbugs warning.

          Show
          Kan Zhang added a comment - fixed a findbugs warning.
          Kan Zhang made changes -
          Attachment h992-21.patch [ 12437085 ]
          Hide
          Kan Zhang added a comment -

          update to the latest trunk.

          Show
          Kan Zhang added a comment - update to the latest trunk.
          Kan Zhang made changes -
          Attachment h992-23.patch [ 12437101 ]
          Kan Zhang made changes -
          Attachment h992-26.patch [ 12437303 ]
          Hide
          Kan Zhang added a comment -

          patch for Yahoo 0.20s branch.

          Show
          Kan Zhang added a comment - patch for Yahoo 0.20s branch.
          Kan Zhang made changes -
          Attachment h992-BK-0.20-07.patch [ 12437340 ]
          Hide
          Jitendra Nath Pandey added a comment -

          This patch is on top of the previous backport.

          Show
          Jitendra Nath Pandey added a comment - This patch is on top of the previous backport.
          Jitendra Nath Pandey made changes -
          Attachment h992-BK-0.20-07.1.patch [ 12438371 ]
          Hide
          Jitendra Nath Pandey added a comment -

          Updated patch against the latest trunk.

          Tests were run manually. All passed except TestHDFSTrash, which also fails in the latest trunk without this patch.

          Show
          Jitendra Nath Pandey added a comment - Updated patch against the latest trunk. Tests were run manually. All passed except TestHDFSTrash, which also fails in the latest trunk without this patch.
          Jitendra Nath Pandey made changes -
          Attachment h992-27.patch [ 12444974 ]
          Hide
          Jitendra Nath Pandey added a comment -

          Updated avro version to 1.3.2.

          Show
          Jitendra Nath Pandey added a comment - Updated avro version to 1.3.2.
          Jitendra Nath Pandey made changes -
          Attachment h992-28.patch [ 12445203 ]
          Hide
          Jakob Homan added a comment -

          Triggering Hudson

          Show
          Jakob Homan added a comment - Triggering Hudson
          Jakob Homan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12445203/h992-28.patch
          against trunk revision 948260.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 36 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/377/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/377/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/377/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445203/h992-28.patch against trunk revision 948260. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 36 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/377/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/377/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/377/console This message is automatically generated.
          Hide
          Jakob Homan added a comment -

          +1 pending test-patch results. This really should have been split into two parts: one the automatic refactoring done by Eclipse and the other the logic refactoring, to make reviewing easier and the history cleaner.

          Show
          Jakob Homan added a comment - +1 pending test-patch results. This really should have been split into two parts: one the automatic refactoring done by Eclipse and the other the logic refactoring, to make reviewing easier and the history cleaner.
          Jakob Homan made changes -
          Hadoop Flags [Reviewed]
          Hide
          Jitendra Nath Pandey added a comment -

          +1 for the patch.

          Show
          Jitendra Nath Pandey added a comment - +1 for the patch.
          Hide
          Jitendra Nath Pandey added a comment -

          Please ignore my previous "+1" comment, it was intended for a different jira.

          Show
          Jitendra Nath Pandey added a comment - Please ignore my previous "+1" comment, it was intended for a different jira.
          Jitendra Nath Pandey made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Jitendra Nath Pandey added a comment -

          Updated patch fixes a findbugs warning.

          Show
          Jitendra Nath Pandey added a comment - Updated patch fixes a findbugs warning.
          Jitendra Nath Pandey made changes -
          Attachment h992-29.patch [ 12445573 ]
          Hide
          Jitendra Nath Pandey added a comment -

          findbugs and javac were run manually.

          Show
          Jitendra Nath Pandey added a comment - findbugs and javac were run manually.
          Hide
          Jakob Homan added a comment -

          +1 for updated patch.

          Show
          Jakob Homan added a comment - +1 for updated patch.
          Hide
          Jakob Homan added a comment -

          I've committed this to trunk. Resolving as fixed. Thanks Jitendra and Kan.

          Show
          Jakob Homan added a comment - I've committed this to trunk. Resolving as fixed. Thanks Jitendra and Kan.
          Jakob Homan made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #283 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/283/)
          HDFS-992. Re-factor block access token implementation to conform to the generic Token interface in Common (Kan Zhang and Jitendra Pandey via jghoman)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #283 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/283/ ) HDFS-992 . Re-factor block access token implementation to conform to the generic Token interface in Common (Kan Zhang and Jitendra Pandey via jghoman)
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Kan Zhang
              Reporter:
              Kan Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development