Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13316

Enforce Kerberos authentication for required ops in DelegationTokenAuthenticator

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: kms, security
    • Labels:
      None
    • Target Version/s:

      Description

      Delegation tokens are supposed to be exchanged in a secure authentication, for security concerns.
      For example, HDFS only distribute or renew a delegation token under kerberos authentication

      DelegationTokenAuthenticationHandler used by KMS + HTTPFS doesn't follow this now, and poses security concerns. Details in comments.

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #10023 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10023/)
        HADOOP-13316. Enforce Kerberos authentication for required ops in (wang: rev 7d2070493e07198896bc49135bc84ef00499a375)

        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestDelegationTokenAuthenticationHandlerWithMocks.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10023 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10023/ ) HADOOP-13316 . Enforce Kerberos authentication for required ops in (wang: rev 7d2070493e07198896bc49135bc84ef00499a375) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestDelegationTokenAuthenticationHandlerWithMocks.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed back through branch-2.8, thanks for finding and fixing this blocker issue Xiao!

        Show
        andrew.wang Andrew Wang added a comment - Committed back through branch-2.8, thanks for finding and fixing this blocker issue Xiao!
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM +1, thanks Xiao. Will commit shortly.

        Show
        andrew.wang Andrew Wang added a comment - LGTM +1, thanks Xiao. Will commit shortly.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 27s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 41s trunk passed
        +1 compile 7m 5s trunk passed
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 1m 4s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 24s trunk passed
        +1 javadoc 0m 46s trunk passed
        +1 mvninstall 0m 41s the patch passed
        +1 compile 7m 10s the patch passed
        +1 javac 7m 10s the patch passed
        -0 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 2 new + 74 unchanged - 1 fixed = 76 total (was 75)
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 30s the patch passed
        +1 javadoc 0m 53s the patch passed
        -1 unit 8m 5s hadoop-common in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        39m 37s



        Reason Tests
        Failed junit tests hadoop.net.TestClusterTopology



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:85209cc
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812989/HADOOP-13316.01.patch
        JIRA Issue HADOOP-13316
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e3d56a3170d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0b9edf6
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9868/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9868/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9868/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9868/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 41s trunk passed +1 compile 7m 5s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 24s trunk passed +1 javadoc 0m 46s trunk passed +1 mvninstall 0m 41s the patch passed +1 compile 7m 10s the patch passed +1 javac 7m 10s the patch passed -0 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 2 new + 74 unchanged - 1 fixed = 76 total (was 75) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 30s the patch passed +1 javadoc 0m 53s the patch passed -1 unit 8m 5s hadoop-common in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 39m 37s Reason Tests Failed junit tests hadoop.net.TestClusterTopology Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812989/HADOOP-13316.01.patch JIRA Issue HADOOP-13316 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e3d56a3170d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0b9edf6 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9868/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9868/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9868/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9868/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment - - edited

        Patch 1 to fix this, with test cases that fail-before, pass-after.
        Manually verified #3 fails in the above scenario with this change.

        Thanks Aaron T. Myers for helping me understand the concept and Andrew Wang for suggesting to create a blocker.

        Show
        xiaochen Xiao Chen added a comment - - edited Patch 1 to fix this, with test cases that fail-before, pass-after. Manually verified #3 fails in the above scenario with this change. Thanks Aaron T. Myers for helping me understand the concept and Andrew Wang for suggesting to create a blocker.
        Hide
        xiaochen Xiao Chen added a comment -

        Example from HTTPFS:
        1. Innocent user gets a token:

        curl -i -L --negotiate -u: "http://xiaog-1.gce.cloudera.com:14000/webhdfs/v1/?op=GETDELEGATIONTOKEN"
        HTTP/1.1 401 Unauthorized
        Server: Apache-Coyote/1.1
        WWW-Authenticate: Negotiate
        Content-Length: 0
        Date: Fri, 24 Jun 2016 03:30:47 GMT
        
        HTTP/1.1 200 OK
        Server: Apache-Coyote/1.1
        Content-Type: application/json
        Content-Length: 125
        Date: Fri, 24 Jun 2016 03:30:47 GMT
        
        {"Token":{"urlString":"IAAGaW1wYWxhBmltcGFsYQCKAVWAdb1aigFVpIJBWgECFLd6Bb7yTckDnGgC1e6FWQ0WlmirEldFQkhERlMgZGVsZWdhdGlvbgA"}}
        

        2. Malicious user who used to have no auth:

        [root@xiaog-1 ~]# curl -i -L --negotiate -u: "http://xiaog-1.gce.cloudera.com:14000/webhdfs/v1/?op=GETDELEGATIONTOKEN"
        HTTP/1.1 401 Unauthorized
        Server: Apache-Coyote/1.1
        WWW-Authenticate: Negotiate
        Content-Length: 0
        Date: Fri, 24 Jun 2016 03:36:19 GMT
        

        3. Malicious user intercepts the token from innocent user, and happily gets its own:

        [root@xiaog-1 ~]# curl -i -L --negotiate -u: "http://xiaog-1.gce.cloudera.com:14000/webhdfs/v1/?op=GETDELEGATIONTOKEN&delegation=IAAGaW1wYWxhBmltcGFsYQCKAVWAdb1aigFVpIJBWgECFLd6Bb7yTckDnGgC1e6FWQ0WlmirEldFQkhERlMgZGVsZWdhdGlvbgA"
        HTTP/1.1 200 OK
        Server: Apache-Coyote/1.1
        Content-Type: application/json
        Content-Length: 125
        Date: Fri, 24 Jun 2016 03:36:46 GMT
        
        {"Token":{"urlString":"IAAGaW1wYWxhBmltcGFsYQCKAVWAezXcigFVpIe53AICFGoUkUqrWVq4n1aCuv3lpVihQrevEldFQkhERlMgZGVsZWdhdGlvbgA"}}
        
        Show
        xiaochen Xiao Chen added a comment - Example from HTTPFS: 1. Innocent user gets a token: curl -i -L --negotiate -u: "http://xiaog-1.gce.cloudera.com:14000/webhdfs/v1/?op=GETDELEGATIONTOKEN" HTTP/1.1 401 Unauthorized Server: Apache-Coyote/1.1 WWW-Authenticate: Negotiate Content-Length: 0 Date: Fri, 24 Jun 2016 03:30:47 GMT HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: application/json Content-Length: 125 Date: Fri, 24 Jun 2016 03:30:47 GMT {"Token":{"urlString":"IAAGaW1wYWxhBmltcGFsYQCKAVWAdb1aigFVpIJBWgECFLd6Bb7yTckDnGgC1e6FWQ0WlmirEldFQkhERlMgZGVsZWdhdGlvbgA"}} 2. Malicious user who used to have no auth: [root@xiaog-1 ~]# curl -i -L --negotiate -u: "http://xiaog-1.gce.cloudera.com:14000/webhdfs/v1/?op=GETDELEGATIONTOKEN" HTTP/1.1 401 Unauthorized Server: Apache-Coyote/1.1 WWW-Authenticate: Negotiate Content-Length: 0 Date: Fri, 24 Jun 2016 03:36:19 GMT 3. Malicious user intercepts the token from innocent user, and happily gets its own: [root@xiaog-1 ~]# curl -i -L --negotiate -u: "http://xiaog-1.gce.cloudera.com:14000/webhdfs/v1/?op=GETDELEGATIONTOKEN&delegation=IAAGaW1wYWxhBmltcGFsYQCKAVWAdb1aigFVpIJBWgECFLd6Bb7yTckDnGgC1e6FWQ0WlmirEldFQkhERlMgZGVsZWdhdGlvbgA" HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: application/json Content-Length: 125 Date: Fri, 24 Jun 2016 03:36:46 GMT {"Token":{"urlString":"IAAGaW1wYWxhBmltcGFsYQCKAVWAezXcigFVpIe53AICFGoUkUqrWVq4n1aCuv3lpVihQrevEldFQkhERlMgZGVsZWdhdGlvbgA"}}

          People

          • Assignee:
            xiaochen Xiao Chen
            Reporter:
            xiaochen Xiao Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development