Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5842

Cannot create hftp filesystem when using a proxy user ugi and a doAs on a secure cluster

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Noticed this while debugging issues in another application. We saw an error when trying to do a FileSystem.get using an hftp file system on a secure cluster using a proxy user ugi.

      This is a small snippet used

       FileSystem testFS = ugi.doAs(new PrivilegedExceptionAction<FileSystem>() {
                  @Override
                  public FileSystem run() throws IOException {
                      return FileSystem.get(hadoopConf);
                  }
              });
      

      The same code worked for hdfs and webhdfs but not for hftp when the ugi used was UserGroupInformation.createProxyUser

      1. HADOOP-10215.000.patch
        1 kB
        Jing Zhao
      2. HADOOP-10215.001.patch
        4 kB
        Jing Zhao
      3. HADOOP-10215.002.patch
        7 kB
        Jing Zhao
      4. HADOOP-10215.002.patch
        7 kB
        Jing Zhao

        Activity

        Hide
        arpitgupta Arpit Gupta added a comment -

        Here is the stack trace from a simple test i wrote

        ava.io.IOException: Unable to obtain remote token
                at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:233)
                at org.apache.hadoop.hdfs.HftpFileSystem$2.run(HftpFileSystem.java:265)
                at org.apache.hadoop.hdfs.HftpFileSystem$2.run(HftpFileSystem.java:259)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:396)
                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
                at org.apache.hadoop.hdfs.HftpFileSystem.getDelegationToken(HftpFileSystem.java:259)
                at org.apache.hadoop.hdfs.HftpFileSystem.initDelegationToken(HftpFileSystem.java:205)
                at org.apache.hadoop.hdfs.HftpFileSystem.initialize(HftpFileSystem.java:194)
                at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433)
                at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
                at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
                at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
                at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
                at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:166)
                at org.hw.tests.ProxyUserTests$1.run(ProxyUserTests.java:122)
                at org.hw.tests.ProxyUserTests$1.run(ProxyUserTests.java:119)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:396)
                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
                at org.hw.tests.ProxyUserTests.getFileSystem(ProxyUserTests.java:119)
                at org.hw.tests.ProxyUserTests.testProxyUserFileSystems(ProxyUserTests.java:78)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
                at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
                at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
                at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
                at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
                at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
                at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
                at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
                at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
                at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
                at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
                at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
                at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
                at org.junit.runners.Suite.runChild(Suite.java:128)
                at org.junit.runners.Suite.runChild(Suite.java:24)
                at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
                at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
                at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
                at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
                at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
                at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
                at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
                at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
                at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
                at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
                at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
                at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
                at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
                at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
        Caused by: java.io.IOException: Exception trying to open authenticated connection to http:/NN_HOST:50070/getDelegationToken
                at org.apache.hadoop.security.SecurityUtil.openSecureHttpConnection(SecurityUtil.java:514)
                at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:222)
                ... 59 more
        Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
                at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:300)
                at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
                at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:232)
                at org.apache.hadoop.security.SecurityUtil.openSecureHttpConnection(SecurityUtil.java:512)
                ... 60 more
        Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
                at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
                at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
                at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
                at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
                at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
                at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
                at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:279)
                at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:255)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:396)
                at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:255)
                ... 63 more
        

        The same test created a hdfs, webhdfs and hftp file system and only the hftp file system failed.

        Show
        arpitgupta Arpit Gupta added a comment - Here is the stack trace from a simple test i wrote ava.io.IOException: Unable to obtain remote token at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:233) at org.apache.hadoop.hdfs.HftpFileSystem$2.run(HftpFileSystem.java:265) at org.apache.hadoop.hdfs.HftpFileSystem$2.run(HftpFileSystem.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.hdfs.HftpFileSystem.getDelegationToken(HftpFileSystem.java:259) at org.apache.hadoop.hdfs.HftpFileSystem.initDelegationToken(HftpFileSystem.java:205) at org.apache.hadoop.hdfs.HftpFileSystem.initialize(HftpFileSystem.java:194) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:166) at org.hw.tests.ProxyUserTests$1.run(ProxyUserTests.java:122) at org.hw.tests.ProxyUserTests$1.run(ProxyUserTests.java:119) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.hw.tests.ProxyUserTests.getFileSystem(ProxyUserTests.java:119) at org.hw.tests.ProxyUserTests.testProxyUserFileSystems(ProxyUserTests.java:78) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:24) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) Caused by: java.io.IOException: Exception trying to open authenticated connection to http:/NN_HOST:50070/getDelegationToken at org.apache.hadoop.security.SecurityUtil.openSecureHttpConnection(SecurityUtil.java:514) at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:222) ... 59 more Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:300) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:232) at org.apache.hadoop.security.SecurityUtil.openSecureHttpConnection(SecurityUtil.java:512) ... 60 more Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:279) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:255) ... 63 more The same test created a hdfs, webhdfs and hftp file system and only the hftp file system failed.
        Hide
        jingzhao Jing Zhao added a comment -

        Looks like we only need to check if the current user is a proxy user before getting a delegation token. If yes, we should use the real user. Upload a simple patch to fix.

        Show
        jingzhao Jing Zhao added a comment - Looks like we only need to check if the current user is a proxy user before getting a delegation token. If yes, we should use the real user. Upload a simple patch to fix.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12622279/HADOOP-10215.000.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3417//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3417//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12622279/HADOOP-10215.000.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3417//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3417//console This message is automatically generated.
        Hide
        jingzhao Jing Zhao added a comment -

        Update the patch. The new patch makes sure that the DT is under the proxy user's name.

        I've tested the patch in my local security setup and the patch works fine. The testing code:

        public class TestHftpFSWithProxyUser {
          public static void main(String[] argv) throws Exception {
            if (argv.length <= 1) {
              System.err.println("Usage: TestHftpFSWithProxyUser fs-uri proxyUser");
              return;
            }
            
            String fsUri = argv[0];
            String proxyUserName = argv[1];
            
            UserGroupInformation real = UserGroupInformation.getCurrentUser();
            System.out.println("Get real ugi: " + real.getShortUserName());
            
            UserGroupInformation proxy = UserGroupInformation.createProxyUser(
                proxyUserName, real);
            System.out.println("Create proxy ugi: " + proxy.getShortUserName());
            
            final Configuration conf = new Configuration();
            conf.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, fsUri);
            
            FileStatus[] status = proxy.doAs(new PrivilegedExceptionAction<FileStatus[]>() {
              @Override
              public FileStatus[] run() throws Exception {
                FileSystem fs = FileSystem.get(conf);
                return fs.listStatus(new Path("/"));
              }
            });
            System.out.println("ls results: " + Arrays.asList(status).toString());
          }
        }
        
        Show
        jingzhao Jing Zhao added a comment - Update the patch. The new patch makes sure that the DT is under the proxy user's name. I've tested the patch in my local security setup and the patch works fine. The testing code: public class TestHftpFSWithProxyUser { public static void main( String [] argv) throws Exception { if (argv.length <= 1) { System .err.println( "Usage: TestHftpFSWithProxyUser fs-uri proxyUser" ); return ; } String fsUri = argv[0]; String proxyUserName = argv[1]; UserGroupInformation real = UserGroupInformation.getCurrentUser(); System .out.println( "Get real ugi: " + real.getShortUserName()); UserGroupInformation proxy = UserGroupInformation.createProxyUser( proxyUserName, real); System .out.println( "Create proxy ugi: " + proxy.getShortUserName()); final Configuration conf = new Configuration(); conf.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, fsUri); FileStatus[] status = proxy.doAs( new PrivilegedExceptionAction<FileStatus[]>() { @Override public FileStatus[] run() throws Exception { FileSystem fs = FileSystem.get(conf); return fs.listStatus( new Path( "/" )); } }); System .out.println( "ls results: " + Arrays.asList(status).toString()); } }
        Hide
        jingzhao Jing Zhao added a comment -

        Looks like HftpFileSystem#renewDelegationToken and HftpFileSystem#cancelDelegationToken need the similar fix. Update the patch to fix these two methods.

        Show
        jingzhao Jing Zhao added a comment - Looks like HftpFileSystem#renewDelegationToken and HftpFileSystem#cancelDelegationToken need the similar fix. Update the patch to fix these two methods.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12624407/HADOOP-10215.001.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3460//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3460//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12624407/HADOOP-10215.001.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3460//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3460//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12625416/HADOOP-10215.002.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3480//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3480//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12625416/HADOOP-10215.002.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3480//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3480//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12625416/HADOOP-10215.002.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.web.TestHttpsFileSystem
        org.apache.hadoop.hdfs.server.namenode.TestNameNodeHttpServer

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5955//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5955//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12625416/HADOOP-10215.002.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.web.TestHttpsFileSystem org.apache.hadoop.hdfs.server.namenode.TestNameNodeHttpServer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5955//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5955//console This message is automatically generated.
        Hide
        jingzhao Jing Zhao added a comment -

        The failed test has been reported in HDFS-5718 and should be unrelated.

        Show
        jingzhao Jing Zhao added a comment - The failed test has been reported in HDFS-5718 and should be unrelated.
        Hide
        jnp Jitendra Nath Pandey added a comment -

        checkTGTAndReloginFromKeytab is removed, it will cause issues once TGT expires.

        Show
        jnp Jitendra Nath Pandey added a comment - checkTGTAndReloginFromKeytab is removed, it will cause issues once TGT expires.
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks for the review, Jitendra. So checkTGTAndReloginFromKeytab is always called in URLConnectionFactory#openConnection, which is called by getDT/renewDT/cancelDT. Thus I think we do not need to call checkTGTAndReloginFromKeytab multiple times here.

        Show
        jingzhao Jing Zhao added a comment - Thanks for the review, Jitendra. So checkTGTAndReloginFromKeytab is always called in URLConnectionFactory#openConnection, which is called by getDT/renewDT/cancelDT. Thus I think we do not need to call checkTGTAndReloginFromKeytab multiple times here.
        Hide
        jnp Jitendra Nath Pandey added a comment -

        URLConnectionFactory#openConnection, which is called by getDT/renewDT/cancelDT. Thus I think we do not need to call checkTGTAndReloginFromKeytab multiple times here.

        Okay, sounds good. +1 for the patch.

        Show
        jnp Jitendra Nath Pandey added a comment - URLConnectionFactory#openConnection, which is called by getDT/renewDT/cancelDT. Thus I think we do not need to call checkTGTAndReloginFromKeytab multiple times here. Okay, sounds good. +1 for the patch.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #5061 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5061/)
        HDFS-5842. Cannot create hftp filesystem when using a proxy user ugi and a doAs on a secure cluster. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562603)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/HftpFileSystem.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5061 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5061/ ) HDFS-5842 . Cannot create hftp filesystem when using a proxy user ugi and a doAs on a secure cluster. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562603 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/HftpFileSystem.java
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks for the review, Jitendra! I've committed this to trunk and branch-2.

        Show
        jingzhao Jing Zhao added a comment - Thanks for the review, Jitendra! I've committed this to trunk and branch-2.
        Hide
        andrew.wang Andrew Wang added a comment -

        Should this be included in branch-2.3 as well?

        Show
        andrew.wang Andrew Wang added a comment - Should this be included in branch-2.3 as well?
        Hide
        jingzhao Jing Zhao added a comment -

        Yeah, that will be great. Thanks Andrew!

        Show
        jingzhao Jing Zhao added a comment - Yeah, that will be great. Thanks Andrew!
        Hide
        andrew.wang Andrew Wang added a comment -

        No prob, merged to branch-2.3. Thanks Jing!

        Show
        andrew.wang Andrew Wang added a comment - No prob, merged to branch-2.3. Thanks Jing!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #5065 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5065/)
        Update CHANGES.txt to move HDFS-5842 to 2.3.0 (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562656)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5065 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5065/ ) Update CHANGES.txt to move HDFS-5842 to 2.3.0 (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562656 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #466 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/466/)
        Update CHANGES.txt to move HDFS-5842 to 2.3.0 (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562656)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          HDFS-5842. Cannot create hftp filesystem when using a proxy user ugi and a doAs on a secure cluster. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562603)
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/HftpFileSystem.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #466 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/466/ ) Update CHANGES.txt to move HDFS-5842 to 2.3.0 (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562656 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS-5842 . Cannot create hftp filesystem when using a proxy user ugi and a doAs on a secure cluster. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562603 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/HftpFileSystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1683 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1683/)
        Update CHANGES.txt to move HDFS-5842 to 2.3.0 (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562656)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          HDFS-5842. Cannot create hftp filesystem when using a proxy user ugi and a doAs on a secure cluster. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562603)
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/HftpFileSystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1683 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1683/ ) Update CHANGES.txt to move HDFS-5842 to 2.3.0 (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562656 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS-5842 . Cannot create hftp filesystem when using a proxy user ugi and a doAs on a secure cluster. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562603 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/HftpFileSystem.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1658 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1658/)
        Update CHANGES.txt to move HDFS-5842 to 2.3.0 (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562656)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          HDFS-5842. Cannot create hftp filesystem when using a proxy user ugi and a doAs on a secure cluster. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562603)
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/HftpFileSystem.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1658 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1658/ ) Update CHANGES.txt to move HDFS-5842 to 2.3.0 (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562656 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS-5842 . Cannot create hftp filesystem when using a proxy user ugi and a doAs on a secure cluster. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1562603 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/HftpFileSystem.java

          People

          • Assignee:
            jingzhao Jing Zhao
            Reporter:
            arpitgupta Arpit Gupta
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development