Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8337

Accessing httpfs via webhdfs doesn't work from a jar with kerberos

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: security, webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In a secure cluster, running a simple program:

      import org.apache.hadoop.conf.*; 
      import org.apache.hadoop.fs.*; 
      import org.apache.hadoop.security.*;
      class Foo { 
        public static void main(String args[]) throws Exception { 
          FileSystem fs = FileSystem.get(new java.net.URI("webhdfs://<host>:14000/"), new Configuration()); 
          System.out.println(fs.listStatus(new Path("/"))[0]); 
          java.io.OutputStream os = fs.create(new Path("/tmp/foo")); 
          os.write('a'); 
          os.close(); 
        } 
      } 
      

      Basically to access httpfs via webhdfs, the following exception is thrown:

      [systest@yj52s ~]$ /usr/java/jdk1.7.0_67-cloudera/bin/java -cp $(hadoop classpath):. Foo
      15/05/06 23:51:38 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded
      Exception in thread "main" org.apache.hadoop.ipc.RemoteException(com.sun.jersey.api.ParamException$QueryParamException): java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.http.client.HttpFSFileSystem.Operation.GETDELEGATIONTOKEN
      	at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:163)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:354)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1299)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:237)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:423)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:444)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:691)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:603)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1277)
      	at Foo.main(Foo.java:7)
      

      Thanks Harsh J and Casey Brotherton for reporting the issue.

        Attachments

        1. HDFS-8337.001.patch
          3 kB
          Yongjun Zhang
        2. HDFS-8337.002.patch
          3 kB
          Yongjun Zhang
        3. HDFS-8337.003.patch
          3 kB
          Yongjun Zhang
        4. HDFS-8337.004.patch
          1 kB
          Yongjun Zhang

          Issue Links

            Activity

              People

              • Assignee:
                yzhangal Yongjun Zhang
                Reporter:
                yzhangal Yongjun Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: