Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8337

Accessing httpfs via webhdfs doesn't work from a jar with kerberos

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: security, webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In a secure cluster, running a simple program:

      import org.apache.hadoop.conf.*; 
      import org.apache.hadoop.fs.*; 
      import org.apache.hadoop.security.*;
      class Foo { 
        public static void main(String args[]) throws Exception { 
          FileSystem fs = FileSystem.get(new java.net.URI("webhdfs://<host>:14000/"), new Configuration()); 
          System.out.println(fs.listStatus(new Path("/"))[0]); 
          java.io.OutputStream os = fs.create(new Path("/tmp/foo")); 
          os.write('a'); 
          os.close(); 
        } 
      } 
      

      Basically to access httpfs via webhdfs, the following exception is thrown:

      [systest@yj52s ~]$ /usr/java/jdk1.7.0_67-cloudera/bin/java -cp $(hadoop classpath):. Foo
      15/05/06 23:51:38 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded
      Exception in thread "main" org.apache.hadoop.ipc.RemoteException(com.sun.jersey.api.ParamException$QueryParamException): java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.http.client.HttpFSFileSystem.Operation.GETDELEGATIONTOKEN
      	at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:163)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:354)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1299)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:237)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:423)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:444)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:691)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:603)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
      	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1277)
      	at Foo.main(Foo.java:7)
      

      Thanks Harsh J and Casey Brotherton for reporting the issue.

      1. HDFS-8337.001.patch
        3 kB
        Yongjun Zhang
      2. HDFS-8337.002.patch
        3 kB
        Yongjun Zhang
      3. HDFS-8337.003.patch
        3 kB
        Yongjun Zhang
      4. HDFS-8337.004.patch
        1 kB
        Yongjun Zhang

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2180 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2180/)
          HDFS-8337. Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java
            Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2180 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2180/ ) HDFS-8337 . Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #232 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/232/)
          HDFS-8337. Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #232 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/232/ ) HDFS-8337 . Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #223 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/223/)
          HDFS-8337. Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #223 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/223/ ) HDFS-8337 . Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2162 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2162/)
          HDFS-8337. Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2162 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2162/ ) HDFS-8337 . Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #234 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/234/)
          HDFS-8337. Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #234 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/234/ ) HDFS-8337 . Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #964 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/964/)
          HDFS-8337. Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #964 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/964/ ) HDFS-8337 . Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8040 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8040/)
          HDFS-8337. Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8040 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8040/ ) HDFS-8337 . Accessing httpfs via webhdfs doesn't work from a jar with kerberos. Contributed by Yongjun Zhang. (yzhang: rev 971dc838ecf8bf55a9bd64128ce2447e4613ea76) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Move HDFS-8337 to branch 2.8 in CHANGES.txt. (yzhang: rev 20c03c96066752f0a7158846de4d7bc4253e83f6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks Alejandro Abdelnur and Arun Suresh for the help and review. I committed to trunk and branch-2.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks Alejandro Abdelnur and Arun Suresh for the help and review. I committed to trunk and branch-2.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 52s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 50s There were no new javac warning messages.
          +1 javadoc 10m 21s There were no new javadoc warning messages.
          +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 32s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse.
          +1 findbugs 0m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 32s Pre-build of native portion
          +1 hdfs tests 3m 38s Tests passed in hadoop-hdfs-httpfs.
              45m 21s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740739/HDFS-8337.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 311a417
          hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11419/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11419/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11419/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 52s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 50s There were no new javac warning messages. +1 javadoc 10m 21s There were no new javadoc warning messages. +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 32s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse. +1 findbugs 0m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 32s Pre-build of native portion +1 hdfs tests 3m 38s Tests passed in hadoop-hdfs-httpfs.     45m 21s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740739/HDFS-8337.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 311a417 hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11419/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11419/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11419/console This message was automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Created/Committed HADOOP-12103 for the hadoop-common portion of the change.

          Submitted rev004 for the hdfs portion here.

          Show
          yzhangal Yongjun Zhang added a comment - Created/Committed HADOOP-12103 for the hadoop-common portion of the change. Submitted rev004 for the hdfs portion here.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Many thanks for looking into Arun Suresh!

          I tested in real cluster with latest HttpFSAuthenticationFilter to see the exception stack reported here. And I tested that the fix works. (with the Foo.java provided in the jira description.)

          I also tested plain httpfs with curl command, to see that it works before and after the fix.

          Show
          yzhangal Yongjun Zhang added a comment - Many thanks for looking into Arun Suresh ! I tested in real cluster with latest HttpFSAuthenticationFilter to see the exception stack reported here. And I tested that the fix works. (with the Foo.java provided in the jira description.) I also tested plain httpfs with curl command, to see that it works before and after the fix.
          Hide
          asuresh Arun Suresh added a comment -

          Thanks for the patch Yongjun Zhang

          The patch looks good to me.
          I understand unit testing this might be non-trivial. Can you please describe the manual testing you've done to confirm things are fine.
          +1 pending that

          Show
          asuresh Arun Suresh added a comment - Thanks for the patch Yongjun Zhang The patch looks good to me. I understand unit testing this might be non-trivial. Can you please describe the manual testing you've done to confirm things are fine. +1 pending that
          Hide
          yzhangal Yongjun Zhang added a comment -

          HI Alejandro Abdelnur,

          Thanks for your help earlier, would you please help doing a review here? thanks.

          Show
          yzhangal Yongjun Zhang added a comment - HI Alejandro Abdelnur , Thanks for your help earlier, would you please help doing a review here? thanks.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Many thanks Alejandro Abdelnur!

          Good to know this workaround, and hope we can commit the patch soon!

          Show
          yzhangal Yongjun Zhang added a comment - Many thanks Alejandro Abdelnur ! Good to know this workaround, and hope we can commit the patch soon!
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          I'll look. BTW, a workaround until there is a release with this fix is adding the following to the httpfs-site.xml:

            <property>
              <name>httpfs.authentication.type</name>
              <value>org.apache.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticationHandler</value>
            </property>
          
            <property>
              <name>httpfs.authentication.delegation-token.token-kind</name>
              <value>WEBHDFS delegation</value>
            </property>
          
          Show
          tucu00 Alejandro Abdelnur added a comment - I'll look. BTW, a workaround until there is a release with this fix is adding the following to the httpfs-site.xml : <property> <name>httpfs.authentication.type</name> <value>org.apache.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticationHandler</value> </property> <property> <name>httpfs.authentication.delegation-token.token-kind</name> <value>WEBHDFS delegation</value> </property>
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Alejandro Abdelnur,

          Thanks for looking into.

          If you look at rev 003, you can see that I exactly did that, except I I introduced some code sharing. Would you please take a look? Thanks a lot.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Alejandro Abdelnur , Thanks for looking into. If you look at rev 003, you can see that I exactly did that, except I I introduced some code sharing. Would you please take a look? Thanks a lot.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          Yongjun Zhang, thanks for looking into this. After digging a bit in the code, I think the intended way for this to work is that HTTPFS authentication filter should do the following in its getConfiguration() method (KMS does exactly this):

              String authType = props.getProperty(AUTH_TYPE);
              if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
                props.setProperty(AUTH_TYPE, PseudoDelegationTokenAuthenticationHandler.class.getName());
              } else if (authType.equals(KerberosAuthenticationHandler.TYPE)) {
                props.setProperty(AUTH_TYPE, KerberosDelegationTokenAuthenticationHandler.class.getName());
              }
              props.setProperty(DelegationTokenAuthenticationHandler.TOKEN_KIND, WebHdfsConstants.WEBHDFS_TOKEN_KIND.toString());
          
          Show
          tucu00 Alejandro Abdelnur added a comment - Yongjun Zhang , thanks for looking into this. After digging a bit in the code, I think the intended way for this to work is that HTTPFS authentication filter should do the following in its getConfiguration() method (KMS does exactly this): String authType = props.getProperty(AUTH_TYPE); if (authType.equals(PseudoAuthenticationHandler.TYPE)) { props.setProperty(AUTH_TYPE, PseudoDelegationTokenAuthenticationHandler.class.getName()); } else if (authType.equals(KerberosAuthenticationHandler.TYPE)) { props.setProperty(AUTH_TYPE, KerberosDelegationTokenAuthenticationHandler.class.getName()); } props.setProperty(DelegationTokenAuthenticationHandler.TOKEN_KIND, WebHdfsConstants.WEBHDFS_TOKEN_KIND.toString());
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Alejandro Abdelnur,

          Thanks again for your earlier help. Would you please help taking a look at rev 003? Thanks much!

          Show
          yzhangal Yongjun Zhang added a comment - Hi Alejandro Abdelnur , Thanks again for your earlier help. Would you please help taking a look at rev 003? Thanks much!
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 57s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 31s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 10s Tests passed in hadoop-common.
          +1 hdfs tests 3m 38s Tests passed in hadoop-hdfs-httpfs.
              67m 44s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738279/HDFS-8337.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a6cb489
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11263/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11263/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11263/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11263/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 31s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 10s Tests passed in hadoop-common. +1 hdfs tests 3m 38s Tests passed in hadoop-hdfs-httpfs.     67m 44s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738279/HDFS-8337.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a6cb489 hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11263/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11263/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11263/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11263/console This message was automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Upload rev 003 to address checkstyle failure.

          Show
          yzhangal Yongjun Zhang added a comment - Upload rev 003 to address checkstyle failure.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 54s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 21s The applied patch generated 1 new checkstyle issues (total was 5, now 6).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 24s Tests passed in hadoop-common.
          +1 hdfs tests 3m 36s Tests passed in hadoop-hdfs-httpfs.
              67m 51s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738256/HDFS-8337.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / b61b489
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11262/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11262/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11262/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11262/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11262/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 21s The applied patch generated 1 new checkstyle issues (total was 5, now 6). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 24s Tests passed in hadoop-common. +1 hdfs tests 3m 36s Tests passed in hadoop-hdfs-httpfs.     67m 51s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738256/HDFS-8337.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b61b489 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11262/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11262/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11262/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11262/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11262/console This message was automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Uploaded rev 002 to address test failures.

          Show
          yzhangal Yongjun Zhang added a comment - Uploaded rev 002 to address test failures.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 29s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 42s There were no new javac warning messages.
          +1 javadoc 9m 48s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 22s The applied patch generated 1 new checkstyle issues (total was 5, now 6).
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 23m 17s Tests failed in hadoop-common.
          -1 hdfs tests 2m 25s Tests failed in hadoop-hdfs-httpfs.
              67m 23s  



          Reason Tests
          Failed unit tests hadoop.ha.TestZKFailoverController
            hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem
            hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem
            hadoop.fs.http.server.TestHttpFSServerNoACLs
            hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem
            hadoop.fs.http.server.TestHttpFSServer
            hadoop.fs.http.server.TestHttpFSServerNoXAttrs
            hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738131/HDFS-8337.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / e46cb80
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11255/artifact/patchprocess/diffcheckstylehadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11255/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11255/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11255/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11255/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11255/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 29s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 42s There were no new javac warning messages. +1 javadoc 9m 48s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 22s The applied patch generated 1 new checkstyle issues (total was 5, now 6). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 23m 17s Tests failed in hadoop-common. -1 hdfs tests 2m 25s Tests failed in hadoop-hdfs-httpfs.     67m 23s   Reason Tests Failed unit tests hadoop.ha.TestZKFailoverController   hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem   hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem   hadoop.fs.http.server.TestHttpFSServerNoACLs   hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem   hadoop.fs.http.server.TestHttpFSServer   hadoop.fs.http.server.TestHttpFSServerNoXAttrs   hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738131/HDFS-8337.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e46cb80 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11255/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11255/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11255/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11255/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11255/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11255/console This message was automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Found two issues:

          1. HttpFSAuthenticationFilter#getConfiguration doesn't set the AUTH_TYPE property to the name of the corresponding auth handler class;
          2. We need to set the TOKEN_KIND in HttpFSAuthenticationFilter , I choose to do it in HttpFSAuthenticationFilter#getConfiguration.

          Submitted patch 001 to address them, and verified that it works in a real cluster.

          Hi Alejandro Abdelnur and Arun Suresh,

          Many thanks for the help when I investigated this issue. Would you please taking a look at the patch I just submitted? Thanks!

          Show
          yzhangal Yongjun Zhang added a comment - Found two issues: 1. HttpFSAuthenticationFilter#getConfiguration doesn't set the AUTH_TYPE property to the name of the corresponding auth handler class; 2. We need to set the TOKEN_KIND in HttpFSAuthenticationFilter , I choose to do it in HttpFSAuthenticationFilter#getConfiguration. Submitted patch 001 to address them, and verified that it works in a real cluster. Hi Alejandro Abdelnur and Arun Suresh , Many thanks for the help when I investigated this issue. Would you please taking a look at the patch I just submitted? Thanks!

            People

            • Assignee:
              yzhangal Yongjun Zhang
              Reporter:
              yzhangal Yongjun Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development