Hadoop Common
  1. Hadoop Common
  2. HADOOP-10326

M/R jobs can not access S3 if Kerberos is enabled

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.4.0
    • Component/s: security
    • Labels:
    • Environment:

      hadoop-1.0.0;MIT kerberos;java 1.6.0_26
      CDH4.3.0(hadoop 2.0.0-alpha);MIT kerberos;java 1.6.0_26

    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      With Kerberos enabled, any job that is taking as input or output s3 files fails.

      It can be easily reproduced with wordcount shipped in hadoop-examples.jar and a public S3 file:

      /opt/hadoop/bin/hadoop --config /opt/hadoop/conf/ jar /opt/hadoop/hadoop-examples-1.0.0.jar wordcount s3n://ubikodpublic/test out01
      

      returns:

      12/08/10 12:40:19 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 192 for hadoop on 10.85.151.233:9000
      12/08/10 12:40:19 INFO security.TokenCache: Got dt for hdfs://aws04.machine.com:9000/mapred/staging/hadoop/.staging/job_201208101229_0004;uri=10.85.151.233:9000;t.service=10.85.151.233:9000
      12/08/10 12:40:19 INFO mapred.JobClient: Cleaning up the staging area hdfs://aws04.machine.com:9000/mapred/staging/hadoop/.staging/job_201208101229_0004
      java.lang.IllegalArgumentException: java.net.UnknownHostException: ubikodpublic
              at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:293)
              at org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:317)
              at org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:189)
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:92)
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
              at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:197)
              at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
      <SNIP>
      

        Activity

        Hide
        Anoop Sam John added a comment -

        I also met with this issue today. Need to check with the patch any way. Manuel DE FERRAN you want create and attach patch?

        Show
        Anoop Sam John added a comment - I also met with this issue today. Need to check with the patch any way. Manuel DE FERRAN you want create and attach patch?
        Hide
        Anoop Sam John added a comment -

        With the above mentioned change I am able to disctp data to S3 when security is enabled also.

        Show
        Anoop Sam John added a comment - With the above mentioned change I am able to disctp data to S3 when security is enabled also.
        Hide
        Mohamed Abdul Rasheed added a comment -

        I encountered this bug while using DistCp to S3. The patch fixed the problem. Thanks!!!

        Show
        Mohamed Abdul Rasheed added a comment - I encountered this bug while using DistCp to S3. The patch fixed the problem. Thanks!!!
        Hide
        Karthik Kambatla added a comment -

        Manuel DE FERRAN - want to take a stab at this, post a patch?

        Show
        Karthik Kambatla added a comment - Manuel DE FERRAN - want to take a stab at this, post a patch?
        Hide
        Karthik Kambatla added a comment -

        The fix would work, but silently ignores the fact that we won't be using security. May be, we should add a config that users should enable to allow this, just so the users know what they are doing?

        Show
        Karthik Kambatla added a comment - The fix would work, but silently ignores the fact that we won't be using security. May be, we should add a config that users should enable to allow this, just so the users know what they are doing?
        Hide
        Karthik Kambatla added a comment -

        Patch from Manuel:

        This patch seems to fix it.

        Index: core/org/apache/hadoop/security/SecurityUtil.java
        ===================================================================
        --- core/org/apache/hadoop/security/SecurityUtil.java   (révision 1305278)
        +++ core/org/apache/hadoop/security/SecurityUtil.java   (copie de travail)
        @@ -313,6 +313,9 @@
             if (authority == null || authority.isEmpty()) {
               return null;
             }
        +    if (uri.getScheme().equals("s3n") || uri.getScheme().equals("s3")) {
        +      return null;
        +    }
             InetSocketAddress addr = NetUtils.createSocketAddr(authority, defPort);
             return buildTokenService(addr).toString();
            }
        
        Show
        Karthik Kambatla added a comment - Patch from Manuel: This patch seems to fix it. Index: core/org/apache/hadoop/security/SecurityUtil.java =================================================================== --- core/org/apache/hadoop/security/SecurityUtil.java (révision 1305278) +++ core/org/apache/hadoop/security/SecurityUtil.java (copie de travail) @@ -313,6 +313,9 @@ if (authority == null || authority.isEmpty()) { return null ; } + if (uri.getScheme().equals( "s3n" ) || uri.getScheme().equals( "s3" )) { + return null ; + } InetSocketAddress addr = NetUtils.createSocketAddr(authority, defPort); return buildTokenService(addr).toString(); }
        Hide
        Karthik Kambatla added a comment -

        Spoke to Alejandro Abdelnur and Aaron T. Myers offline. Both believe, YARN/MR and SecurityUtil should be agnostic to how the underlying filesystem handles tokens. The S3 client should handle ignoring these tokens. Moved to a Common JIRA to address that.

        Show
        Karthik Kambatla added a comment - Spoke to Alejandro Abdelnur and Aaron T. Myers offline. Both believe, YARN/MR and SecurityUtil should be agnostic to how the underlying filesystem handles tokens. The S3 client should handle ignoring these tokens. Moved to a Common JIRA to address that.
        Hide
        bc Wong added a comment -

        Adding patch "0001-HADOOP-10326.-s3-s3n-does-not-support-tokens.patch". Tested on secure cluster, running distcp and wordcount against s3 data.

        Show
        bc Wong added a comment - Adding patch "0001- HADOOP-10326 .-s3-s3n-does-not-support-tokens.patch". Tested on secure cluster, running distcp and wordcount against s3 data.
        Hide
        Aaron T. Myers added a comment -

        Patch looks good to me. +1 pending Jenkins.

        For some reason I can't seem to add bc as a Hadoop contributor at the moment. Alejandro Abdelnur - would you mind taking care of that in the JIRA admin console?

        Show
        Aaron T. Myers added a comment - Patch looks good to me. +1 pending Jenkins. For some reason I can't seem to add bc as a Hadoop contributor at the moment. Alejandro Abdelnur - would you mind taking care of that in the JIRA admin console?
        Hide
        Alejandro Abdelnur added a comment -

        JIRA console funny, getting "The JIRA server could not be contacted. This may be a temporary glitch or the server may be down." pop up when trying to add him

        Show
        Alejandro Abdelnur added a comment - JIRA console funny, getting "The JIRA server could not be contacted. This may be a temporary glitch or the server may be down." pop up when trying to add him
        Hide
        Aaron T. Myers added a comment -

        Yea, me too. Curious, but not a big deal. Can fix that up later.

        Show
        Aaron T. Myers added a comment - Yea, me too. Curious, but not a big deal. Can fix that up later.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12627825/0001-HADOOP-10326.-s3-s3n-does-not-support-tokens.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3554//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3554//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12627825/0001-HADOOP-10326.-s3-s3n-does-not-support-tokens.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3554//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3554//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        I've just committed this to trunk and branch-2.

        Thanks a lot for the contribution, bc.

        Show
        Aaron T. Myers added a comment - I've just committed this to trunk and branch-2. Thanks a lot for the contribution, bc.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #5144 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5144/)
        HADOOP-10326. M/R jobs can not access S3 if Kerberos is enabled. Contributed by bc Wong. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566965)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3/S3FileSystemContractBaseTest.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3native/NativeS3FileSystemContractBaseTest.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5144 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5144/ ) HADOOP-10326 . M/R jobs can not access S3 if Kerberos is enabled. Contributed by bc Wong. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566965 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3/S3FileSystemContractBaseTest.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3native/NativeS3FileSystemContractBaseTest.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #478 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/478/)
        HADOOP-10326. M/R jobs can not access S3 if Kerberos is enabled. Contributed by bc Wong. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566965)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3/S3FileSystemContractBaseTest.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3native/NativeS3FileSystemContractBaseTest.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #478 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/478/ ) HADOOP-10326 . M/R jobs can not access S3 if Kerberos is enabled. Contributed by bc Wong. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566965 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3/S3FileSystemContractBaseTest.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3native/NativeS3FileSystemContractBaseTest.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1670 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1670/)
        HADOOP-10326. M/R jobs can not access S3 if Kerberos is enabled. Contributed by bc Wong. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566965)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3/S3FileSystemContractBaseTest.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3native/NativeS3FileSystemContractBaseTest.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1670 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1670/ ) HADOOP-10326 . M/R jobs can not access S3 if Kerberos is enabled. Contributed by bc Wong. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566965 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3/S3FileSystemContractBaseTest.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3native/NativeS3FileSystemContractBaseTest.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1695 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1695/)
        HADOOP-10326. M/R jobs can not access S3 if Kerberos is enabled. Contributed by bc Wong. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566965)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3/S3FileSystemContractBaseTest.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3native/NativeS3FileSystemContractBaseTest.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1695 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1695/ ) HADOOP-10326 . M/R jobs can not access S3 if Kerberos is enabled. Contributed by bc Wong. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566965 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3/S3FileSystemContractBaseTest.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/s3native/NativeS3FileSystemContractBaseTest.java

          People

          • Assignee:
            bc Wong
            Reporter:
            Manuel DE FERRAN
          • Votes:
            1 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development