[HDFS-8337] Accessing httpfs via webhdfs doesn't work from a jar with kerberos

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: security, webhdfs
• Hadoop Flags: Reviewed
• Assignee: Yongjun Zhang
• Reporter: Yongjun Zhang
• Created: 07/May/15 06:58
• Updated: 30/Aug/16 01:34
• Resolved: 20/Jun/15 07:32

Description

In a secure cluster, running a simple program:

import org.apache.hadoop.conf.*; 
import org.apache.hadoop.fs.*; 
import org.apache.hadoop.security.*;
class Foo { 
  public static void main(String args[]) throws Exception { 
    FileSystem fs = FileSystem.get(new java.net.URI("webhdfs://<host>:14000/"), new Configuration()); 
    System.out.println(fs.listStatus(new Path("/"))[0]); 
    java.io.OutputStream os = fs.create(new Path("/tmp/foo")); 
    os.write('a'); 
    os.close(); 
  } 
} 

Basically to access httpfs via webhdfs, the following exception is thrown:

[systest@yj52s ~]$ /usr/java/jdk1.7.0_67-cloudera/bin/java -cp $(hadoop classpath):. Foo
15/05/06 23:51:38 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded
Exception in thread "main" org.apache.hadoop.ipc.RemoteException(com.sun.jersey.api.ParamException$QueryParamException): java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.http.client.HttpFSFileSystem.Operation.GETDELEGATIONTOKEN
	at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:163)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:354)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1299)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:237)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:423)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:444)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:691)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:603)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1277)
	at Foo.main(Foo.java:7)

Thanks qwertymaniac and caseyjbrotherton for reporting the issue.

Issue Links

• depends upon: HADOOP-12103 Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing
• is related to: HDFS-7983 HTTPFS proxy server needs pluggable-auth support