[HADOOP-10326] M/R jobs can not access S3 if Kerberos is enabled

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.4.0
• Component/s: security
• Labels: s3

Environment

hadoop-1.0.0;MIT kerberos;java 1.6.0_26
CDH4.3.0(hadoop 2.0.0-alpha);MIT kerberos;java 1.6.0_26

• Target Version/s: 2.4.0
• Hadoop Flags: Reviewed
• Assignee: bc Wong
• Reporter: Manuel DE FERRAN
• Created: 10/Aug/12 13:49
• Updated: 10/Apr/14 13:11
• Resolved: 11/Feb/14 02:49

Description

With Kerberos enabled, any job that is taking as input or output s3 files fails.

It can be easily reproduced with wordcount shipped in hadoop-examples.jar and a public S3 file:

/opt/hadoop/bin/hadoop --config /opt/hadoop/conf/ jar /opt/hadoop/hadoop-examples-1.0.0.jar wordcount s3n://ubikodpublic/test out01

returns:

12/08/10 12:40:19 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 192 for hadoop on 10.85.151.233:9000
12/08/10 12:40:19 INFO security.TokenCache: Got dt for hdfs://aws04.machine.com:9000/mapred/staging/hadoop/.staging/job_201208101229_0004;uri=10.85.151.233:9000;t.service=10.85.151.233:9000
12/08/10 12:40:19 INFO mapred.JobClient: Cleaning up the staging area hdfs://aws04.machine.com:9000/mapred/staging/hadoop/.staging/job_201208101229_0004
java.lang.IllegalArgumentException: java.net.UnknownHostException: ubikodpublic
        at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:293)
        at org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:317)
        at org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:189)
        at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:92)
        at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:197)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
<SNIP>