Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11482

Use correct UGI when KMSClientProvider is called by a proxy user

    Details

      Description

      Long Living clients of HDFS (For eg. OOZIE) use cached DFSClients which in turn use a cached KMSClientProvider to talk to KMS.

      Before an MR Job is run, the job client calls the DFClient.addDelegationTokens() method which calls addDelegationTokens() on the KMSClientProvider to get any delegation token associated to the user.

      Unfortunately, this call uses a cached DelegationTokenAuthenticationURL.Token instance which can cause the SignerSecretProvider implementation of the AuthenticationFilter at the KMS Server end to fail validation. Which results in the MR job itself failing.

      1. HADOOP-11482.2.patch
        4 kB
        Arun Suresh
      2. HADOOP-11482.1.patch
        4 kB
        Arun Suresh

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Pulled this into 2.6.1 after Akira Ajisaka verified that the patch applies cleanly. Ran compilation and TestKMS before the push.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.6.1 after Akira Ajisaka verified that the patch applies cleanly. Ran compilation and TestKMS before the push.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2034 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2034/)
        HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2034 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2034/ ) HADOOP-11482 . Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #84 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/84/)
        HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
        • hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #84 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/84/ ) HADOOP-11482 . Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2015 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2015/)
        HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2015 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2015/ ) HADOOP-11482 . Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #80 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/80/)
        HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
        • hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #80 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/80/ ) HADOOP-11482 . Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #817 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/817/)
        HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
        • hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #817 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/817/ ) HADOOP-11482 . Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #83 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/83/)
        HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #83 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/83/ ) HADOOP-11482 . Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #6919 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6919/)
        HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e)

        • hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6919 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6919/ ) HADOOP-11482 . Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh. (wang: rev 4b00935643f6c3656ccbd7eeb54884738bc12c2e) hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        andrew.wang Andrew Wang added a comment -

        TYs again Arun, committed to trunk and branch-2.

        Show
        andrew.wang Andrew Wang added a comment - TYs again Arun, committed to trunk and branch-2.
        Hide
        andrew.wang Andrew Wang added a comment -

        +1 LGTM, I guess it's hard to decrease the test time further. Will commit shortly.

        Show
        andrew.wang Andrew Wang added a comment - +1 LGTM, I guess it's hard to decrease the test time further. Will commit shortly.
        Hide
        asuresh Arun Suresh added a comment -

        The test failure looks unrelated...

        Show
        asuresh Arun Suresh added a comment - The test failure looks unrelated...
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12694087/HADOOP-11482.2.patch
        against trunk revision 5f124ef.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms:

        org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5463//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5463//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12694087/HADOOP-11482.2.patch against trunk revision 5f124ef. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms: org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5463//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5463//console This message is automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Thanks for the review Andrew Wang, Updating patch :

        • added more comments
        • cut down total sleep time from 20 to 8 secs (We unfortunately need atleast 4 secs to ensure the rollover signer rolls over atleast twice... and I put in another 4 to be on the safe side)
        Show
        asuresh Arun Suresh added a comment - Thanks for the review Andrew Wang , Updating patch : added more comments cut down total sleep time from 20 to 8 secs (We unfortunately need atleast 4 secs to ensure the rollover signer rolls over atleast twice... and I put in another 4 to be on the safe side)
        Hide
        andrew.wang Andrew Wang added a comment -

        Hey Arun, fix looks good, just a few comments:

        • Can we expand the comment a bit? Not necessarily obvious to the reader why the UGI would change.
        • The test additions end up sleeping for an additional 20seconds, which is quite substantial. Is there a way we can test this without such long sleeps?

        Thanks again!

        Show
        andrew.wang Andrew Wang added a comment - Hey Arun, fix looks good, just a few comments: Can we expand the comment a bit? Not necessarily obvious to the reader why the UGI would change. The test additions end up sleeping for an additional 20seconds, which is quite substantial. Is there a way we can test this without such long sleeps? Thanks again!
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12692475/HADOOP-11482.1.patch
        against trunk revision 5805dc0.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5409//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5409//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692475/HADOOP-11482.1.patch against trunk revision 5805dc0. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5409//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5409//console This message is automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Attaching patch to fix this..

        Show
        asuresh Arun Suresh added a comment - Attaching patch to fix this..

          People

          • Assignee:
            asuresh Arun Suresh
            Reporter:
            asuresh Arun Suresh
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development