Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11934

Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I was attempting to use the LdapGroupsMapping code and the JavaKeyStoreProvider at the same time, and hit a really interesting, yet fatal, issue. The code goes into what ought to have been an infinite loop, were it not for it overflowing the stack and Java ending the loop. Here is a snippet of the stack; my annotations are at the bottom.

      	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
      	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
      	at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:88)
      	at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:65)
      	at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:291)
      	at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:58)
      	at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1863)
      	at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1843)
      	at org.apache.hadoop.security.LdapGroupsMapping.getPassword(LdapGroupsMapping.java:386)
      	at org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:349)
      	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
      	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
      	at org.apache.hadoop.security.Groups.<init>(Groups.java:70)
      	at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
      	at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280)
      	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
      	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
      	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:804)
      	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:774)
      	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
      	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2753)
      	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2745)
      	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
      	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
      	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
      	at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:88)
      	at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:65)
      	at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:291)
      	at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:58)
      	at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1863)
      	at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1843)
      	at org.apache.hadoop.security.LdapGroupsMapping.getPassword(LdapGroupsMapping.java:386)
      	at org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:349)
      	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
      	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
      	at org.apache.hadoop.security.Groups.<init>(Groups.java:70)
      	at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
      	at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280)
      	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
      	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
      	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:804)
      	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:774)
      	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
      	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2753)
      	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2745)
      	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
      	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
      	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)

      Here's my annotation, going from bottom to top.

      • Somehow we enter Path.getFileSystem()
      • This goes to FileSystem cache stuff, and then it wants the current user
      • So we get to UserGroupInformation.getCurrentUser(), which as you can imagine gets to
      • getUserToGroupsMappingService and thence to LdapGroupsMapping.setConf().
      • That code gets the needed passwords, and we're using the CredentialProvider, so unsurprisingly we get to
      • getPasswordFromCredentialProviders() - which chooses the JavaKeyStoreProvider like I told it to.
      • The JavaKeyStoreProvider, in its constructor, does "fs = path.getFileSystem(conf);"
      • And guess what, we're back in Path.getFileSystem, where we started at the beginning.

      Please let me know if I've somehow configured something incorrectly, but if I have I can't figure out what it is...

      1. HADOOP-11934.001.patch
        33 kB
        Larry McCay
      2. HADOOP-11934.002.patch
        35 kB
        Larry McCay
      3. HADOOP-11934.003.patch
        35 kB
        Larry McCay
      4. HADOOP-11934.004.patch
        35 kB
        Larry McCay
      5. HADOOP-11934.005.patch
        35 kB
        Larry McCay
      6. HADOOP-11934.006.patch
        33 kB
        Larry McCay
      7. HADOOP-11934.007.patch
        33 kB
        Larry McCay
      8. HADOOP-11934.008.patch
        33 kB
        Larry McCay
      9. HADOOP-11934.009.patch
        33 kB
        Larry McCay
      10. HADOOP-11934.010.patch
        33 kB
        Larry McCay
      11. HADOOP-11934.012.patch
        32 kB
        Larry McCay
      12. HADOOP-11934.013.patch
        32 kB
        Larry McCay
      13. HADOOP-11934-11.patch
        31 kB
        Larry McCay
      14. HADOOP-11934-branch.2.6.1.txt
        33 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Hide
          yoderme Mike Yoder added a comment -

          Larry McCay Brandon Li - mentioning you guys since your names are on HADOOP-10905. Thanks for having a peek at this issue.

          Show
          yoderme Mike Yoder added a comment - Larry McCay Brandon Li - mentioning you guys since your names are on HADOOP-10905 . Thanks for having a peek at this issue.
          Hide
          lmccay Larry McCay added a comment -

          Hey Mike Yoder - can you show me your configuration for hadoop.security.credential.provider.path?

          Show
          lmccay Larry McCay added a comment - Hey Mike Yoder - can you show me your configuration for hadoop.security.credential.provider.path?
          Hide
          yoderme Mike Yoder added a comment -

          It looks like

          jceks://file/full/path/to/creds.jceks

          Show
          yoderme Mike Yoder added a comment - It looks like jceks://file/full/path/to/creds.jceks
          Hide
          lmccay Larry McCay added a comment -

          I will have a look at this tomorrow.
          It seems that we have a recursive call happening here.

          I assume that the same credential provider path config works fine using the credential hadoop command.

          Show
          lmccay Larry McCay added a comment - I will have a look at this tomorrow. It seems that we have a recursive call happening here. I assume that the same credential provider path config works fine using the credential hadoop command.
          Hide
          yoderme Mike Yoder added a comment -

          Yeah, the same sort of provider path works fine for the HS2 keystore password and the hadoop_ssl_server_keystore_(key)password.

          Show
          yoderme Mike Yoder added a comment - Yeah, the same sort of provider path works fine for the HS2 keystore password and the hadoop_ssl_server_keystore_(key)password.
          Hide
          lmccay Larry McCay added a comment -

          It seems that this has nothing to do with the provider path. I believe that the get of the FS within the JKS provider itself is triggering another look up of user groups - I guess to check whether the user has access to the FS - which is then calling the provider again and so on...

          Show
          lmccay Larry McCay added a comment - It seems that this has nothing to do with the provider path. I believe that the get of the FS within the JKS provider itself is triggering another look up of user groups - I guess to check whether the user has access to the FS - which is then calling the provider again and so on...
          Hide
          lmccay Larry McCay added a comment -

          Mike Yoder - would you happen to have access to the beginning of that stack trace?
          I need to know the ultimate client of the credential provider API in this scenario - as there may be more than one solution to this issue based on those details.

          Show
          lmccay Larry McCay added a comment - Mike Yoder - would you happen to have access to the beginning of that stack trace? I need to know the ultimate client of the credential provider API in this scenario - as there may be more than one solution to this issue based on those details.
          Hide
          yoderme Mike Yoder added a comment -

          Sorry, it's not in the log. The log shows

          STARTUP_MSG:   java = 1.7.0_67
          ************************************************************/
          2015-05-06 17:00:26,732 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
          2015-05-06 17:00:26,742 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode []
          2015-05-06 17:00:27,157 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
          2015-05-06 17:00:27,343 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
          2015-05-06 17:00:27,343 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
          2015-05-06 17:00:27,348 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is hdfs://mey-may-4.vpc.cloudera.com:8020
          2015-05-06 17:00:27,348 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use mey-may-4.vpc.cloudera.com:8020 to access this namenode/service.
          2015-05-06 17:00:32,144 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
          java.lang.StackOverflowError
                  at java.lang.String.indexOf(String.java:1698)
                  at java.net.URLStreamHandler.parseURL(URLStreamHandler.java:272)
                  at sun.net.www.protocol.file.Handler.parseURL(Handler.java:67)
                  at java.net.URL.<init>(URL.java:614)
                  at java.net.URL.<init>(URL.java:482)
                  at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1057)
                  at sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:1047)
                  at sun.misc.URLClassPath.findResource(URLClassPath.java:176)
                  at java.net.URLClassLoader$2.run(URLClassLoader.java:551)
                  at java.net.URLClassLoader$2.run(URLClassLoader.java:549)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at java.net.URLClassLoader.findResource(URLClassLoader.java:548)
                  at java.lang.ClassLoader.getResource(ClassLoader.java:1147)
                  at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:227)
                  at javax.xml.parsers.SecuritySupport$4.run(SecuritySupport.java:94)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.xml.parsers.SecuritySupport.getResourceAsStream(SecuritySupport.java:87)
                  at javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:283)
                  at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
                  at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
                  at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2425)
                  at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2402)
                  at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2319)
                  at org.apache.hadoop.conf.Configuration.get(Configuration.java:1146)
                  at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:605)
                  at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:272)
                  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
                  at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:804)
                  at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:774)
                  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
                  at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2753)
                  at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2745)
                  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
                  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
                  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
                  at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:88)
          

          .... a lot of repetition ....

                  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
                  at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:88)
                  at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:65)
                  at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:291)
                  at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:58)
                  at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1863)
                  at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1843)
                  at org.apache.hadoop.security.LdapGroupsMapping.getPassword(LdapGroupsMapping.java:386)
                  at org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:349)
                  at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
                  at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
                  at org.apache.hadoop.security.Groups.<init>(Groups.java:70)
                  at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
                  at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280)
                  at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
                  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
                  at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:804)
                  at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:774)
                  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
                  at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2753)
                  at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2745)
                  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
                  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
                  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
          2015-05-06 17:00:32,183 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
          2015-05-06 17:00:32,184 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
          
          Show
          yoderme Mike Yoder added a comment - Sorry, it's not in the log. The log shows STARTUP_MSG: java = 1.7.0_67 ************************************************************/ 2015-05-06 17:00:26,732 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 2015-05-06 17:00:26,742 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode [] 2015-05-06 17:00:27,157 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2015-05-06 17:00:27,343 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2015-05-06 17:00:27,343 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started 2015-05-06 17:00:27,348 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is hdfs://mey-may-4.vpc.cloudera.com:8020 2015-05-06 17:00:27,348 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use mey-may-4.vpc.cloudera.com:8020 to access this namenode/service. 2015-05-06 17:00:32,144 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode. java.lang.StackOverflowError at java.lang.String.indexOf(String.java:1698) at java.net.URLStreamHandler.parseURL(URLStreamHandler.java:272) at sun.net.www.protocol.file.Handler.parseURL(Handler.java:67) at java.net.URL.<init>(URL.java:614) at java.net.URL.<init>(URL.java:482) at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1057) at sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:1047) at sun.misc.URLClassPath.findResource(URLClassPath.java:176) at java.net.URLClassLoader$2.run(URLClassLoader.java:551) at java.net.URLClassLoader$2.run(URLClassLoader.java:549) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findResource(URLClassLoader.java:548) at java.lang.ClassLoader.getResource(ClassLoader.java:1147) at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:227) at javax.xml.parsers.SecuritySupport$4.run(SecuritySupport.java:94) at java.security.AccessController.doPrivileged(Native Method) at javax.xml.parsers.SecuritySupport.getResourceAsStream(SecuritySupport.java:87) at javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:283) at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255) at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2425) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2402) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2319) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1146) at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:605) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:272) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:804) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:774) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2753) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2745) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:88) .... a lot of repetition .... at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:88) at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:65) at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:291) at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:58) at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1863) at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1843) at org.apache.hadoop.security.LdapGroupsMapping.getPassword(LdapGroupsMapping.java:386) at org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:349) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.security.Groups.<init>(Groups.java:70) at org.apache.hadoop.security.Groups.<init>(Groups.java:66) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:804) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:774) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2753) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2745) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) 2015-05-06 17:00:32,183 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1 2015-05-06 17:00:32,184 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
          Hide
          lmccay Larry McCay added a comment -

          AFAICT - we have the following options:

          1. remove credential provider support from LdapGroupMappings - this would be unfortunate
          2. add a new provider type that wouldn't be dependent on the Hadoop FileSystem abstraction and its permission checks
          a. this could easily be done with a provider that speaks directly to the java file API and only supports "local" keystores
          b. a credential server provider that doesn't rely on file permissions on keystores for authorization but kerberos and ACLs of its own

          2.a. would work fine for non-mapreduce clients but unfortunately would require permissions for mapreduce clients that would make the passwords available to endusers.

          2.b. Is actually the only really secure option to begin with and would address this issue properly.

          Perhaps a wrapper around the KMSClientProvider would work for a quick implementation of 2.b. but that will require some investigation and potentially adding support for arbitrary keys rather than only known key types and length checking.

          Mike Yoder - have you been able to proceed by removing the use of the credential provider for LdapGroupsMapping?

          Show
          lmccay Larry McCay added a comment - AFAICT - we have the following options: 1. remove credential provider support from LdapGroupMappings - this would be unfortunate 2. add a new provider type that wouldn't be dependent on the Hadoop FileSystem abstraction and its permission checks a. this could easily be done with a provider that speaks directly to the java file API and only supports "local" keystores b. a credential server provider that doesn't rely on file permissions on keystores for authorization but kerberos and ACLs of its own 2.a. would work fine for non-mapreduce clients but unfortunately would require permissions for mapreduce clients that would make the passwords available to endusers. 2.b. Is actually the only really secure option to begin with and would address this issue properly. Perhaps a wrapper around the KMSClientProvider would work for a quick implementation of 2.b. but that will require some investigation and potentially adding support for arbitrary keys rather than only known key types and length checking. Mike Yoder - have you been able to proceed by removing the use of the credential provider for LdapGroupsMapping?
          Hide
          lmccay Larry McCay added a comment -

          Based on your log it seems to be running inside the NameNode.
          This issue can be addressed by a new localkeystore provider that doesn't use the filesystem abstraction.

          The only downside is that it won't be able to get to keystores that are stored inside of HDFS.

          I can try and crank out a provider for this in short order.

          Thanks for the log info that is very helpful!

          Show
          lmccay Larry McCay added a comment - Based on your log it seems to be running inside the NameNode. This issue can be addressed by a new localkeystore provider that doesn't use the filesystem abstraction. The only downside is that it won't be able to get to keystores that are stored inside of HDFS. I can try and crank out a provider for this in short order. Thanks for the log info that is very helpful!
          Hide
          yoderme Mike Yoder added a comment -

          Yeah, if I remove the credential provider from the LdapGroupsMapping everything is fine.

          A local keystore provider that's basically identical to the JavaKeyStoreProvider but only looks on the local file system fits my use case just fine.

          Might there be a way, inside JavaKeyStoreProvider, to look at the URI before calling path.getFilesystem() and do something different if it's not in HDFS?

          Show
          yoderme Mike Yoder added a comment - Yeah, if I remove the credential provider from the LdapGroupsMapping everything is fine. A local keystore provider that's basically identical to the JavaKeyStoreProvider but only looks on the local file system fits my use case just fine. Might there be a way, inside JavaKeyStoreProvider, to look at the URI before calling path.getFilesystem() and do something different if it's not in HDFS?
          Hide
          lmccay Larry McCay added a comment -

          Cool.

          Well, I don't think we would want to do that. This specific usecase is problematic because the very mechanism that we use to authorize access is the client asking for access. Essentially, LdapGroupsMapping is part of the Hadoop permission checking machinery. So having it protected passwords in a store that is protected by itself is a recursive problem.

          At the same time, there are other instances where this permission check is being used for local keystores that are unrelated to the LdapGroupsMapping - like SSL credentials for instance. There is no recursive issue there.

          I think the local keystore provider will work well.

          In fact, I'm not sure that there are mapreduce client scenarios that we need to worry about - as long as NN is the one always calling LdapGroupsMapping.

          Show
          lmccay Larry McCay added a comment - Cool. Well, I don't think we would want to do that. This specific usecase is problematic because the very mechanism that we use to authorize access is the client asking for access. Essentially, LdapGroupsMapping is part of the Hadoop permission checking machinery. So having it protected passwords in a store that is protected by itself is a recursive problem. At the same time, there are other instances where this permission check is being used for local keystores that are unrelated to the LdapGroupsMapping - like SSL credentials for instance. There is no recursive issue there. I think the local keystore provider will work well. In fact, I'm not sure that there are mapreduce client scenarios that we need to worry about - as long as NN is the one always calling LdapGroupsMapping.
          Hide
          yoderme Mike Yoder added a comment -

          When above I said "do something different" wasn't implying that we should ignore permission checks. I see what you're saying about this very use case being problematic. What I was attempting to say above what that you could put some of the "local keystore" logic into the existing JavaKeystoreProvider if the file is local. But either way works.

          Show
          yoderme Mike Yoder added a comment - When above I said "do something different" wasn't implying that we should ignore permission checks. I see what you're saying about this very use case being problematic. What I was attempting to say above what that you could put some of the "local keystore" logic into the existing JavaKeystoreProvider if the file is local. But either way works.
          Hide
          lmccay Larry McCay added a comment -

          Yeah, I understood what you were saying but that would change it for all usages not just for the group lookup.
          I wouldn't want to put any knowledge of when to do that into the provider and I think we want to maintain the use of the Hadoop checks in the other usecases. I will take a look at it with fresh eyes though - just to make sure.

          Show
          lmccay Larry McCay added a comment - Yeah, I understood what you were saying but that would change it for all usages not just for the group lookup. I wouldn't want to put any knowledge of when to do that into the provider and I think we want to maintain the use of the Hadoop checks in the other usecases. I will take a look at it with fresh eyes though - just to make sure.
          Hide
          lmccay Larry McCay added a comment -

          Attaching patch with refactoring of JavaKeyStoreProvider into a base and two derived classes. LocalJavaKeyStoreProvider will be for use when the Hadoop FileSystem cannot be used - such as with LdapGroupsMapping.

          Show
          lmccay Larry McCay added a comment - Attaching patch with refactoring of JavaKeyStoreProvider into a base and two derived classes. LocalJavaKeyStoreProvider will be for use when the Hadoop FileSystem cannot be used - such as with LdapGroupsMapping.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 32s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 27s There were no new javac warning messages.
          +1 javadoc 9m 32s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 7s The applied patch generated 82 new checkstyle issues (total was 19, now 99).
          -1 whitespace 0m 5s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 39s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 1m 43s The patch appears to introduce 4 new Findbugs (version 2.0.3) warnings.
          +1 common tests 22m 31s Tests passed in hadoop-common.
              59m 35s  



          Reason Tests
          FindBugs module:hadoop-common
            Found reliance on default encoding in org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.bytesToChars(byte[]):in org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.bytesToChars(byte[]): new String(byte[]) At AbstractJavaKeyStoreProvider.java:[line 176]
            Switch statement found in org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.addGroupPermissons(char, Set) where default case is missing At LocalJavaKeyStoreProvider.java:Set) where default case is missing At LocalJavaKeyStoreProvider.java:[lines 147-168]
            Switch statement found in org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.addOthersPermissons(char, Set) where default case is missing At LocalJavaKeyStoreProvider.java:Set) where default case is missing At LocalJavaKeyStoreProvider.java:[lines 120-141]
            Switch statement found in org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.addOwnerPermissions(char, Set) where default case is missing At LocalJavaKeyStoreProvider.java:Set) where default case is missing At LocalJavaKeyStoreProvider.java:[lines 174-195]



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731769/HADOOP-11934.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a60f78e
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6574/artifact/patchprocess/diffcheckstylehadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6574/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/6574/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6574/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6574/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6574/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 32s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 27s There were no new javac warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 7s The applied patch generated 82 new checkstyle issues (total was 19, now 99). -1 whitespace 0m 5s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 1m 43s The patch appears to introduce 4 new Findbugs (version 2.0.3) warnings. +1 common tests 22m 31s Tests passed in hadoop-common.     59m 35s   Reason Tests FindBugs module:hadoop-common   Found reliance on default encoding in org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.bytesToChars(byte[]):in org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.bytesToChars(byte[]): new String(byte[]) At AbstractJavaKeyStoreProvider.java: [line 176]   Switch statement found in org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.addGroupPermissons(char, Set) where default case is missing At LocalJavaKeyStoreProvider.java:Set) where default case is missing At LocalJavaKeyStoreProvider.java: [lines 147-168]   Switch statement found in org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.addOthersPermissons(char, Set) where default case is missing At LocalJavaKeyStoreProvider.java:Set) where default case is missing At LocalJavaKeyStoreProvider.java: [lines 120-141]   Switch statement found in org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.addOwnerPermissions(char, Set) where default case is missing At LocalJavaKeyStoreProvider.java:Set) where default case is missing At LocalJavaKeyStoreProvider.java: [lines 174-195] Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731769/HADOOP-11934.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a60f78e checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6574/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6574/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/6574/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6574/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6574/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6574/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Addressed formatting, findbugs, etc

          Show
          lmccay Larry McCay added a comment - Addressed formatting, findbugs, etc
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 35s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 27s There were no new javac warning messages.
          +1 javadoc 9m 31s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 4s The applied patch generated 20 new checkstyle issues (total was 15, now 21).
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 22m 13s Tests passed in hadoop-common.
              59m 4s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731804/HADOOP-11934.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4536399
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6578/artifact/patchprocess/diffcheckstylehadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6578/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6578/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6578/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6578/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 27s There were no new javac warning messages. +1 javadoc 9m 31s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 4s The applied patch generated 20 new checkstyle issues (total was 15, now 21). -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 22m 13s Tests passed in hadoop-common.     59m 4s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731804/HADOOP-11934.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4536399 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6578/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6578/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6578/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6578/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6578/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          More checkstyle issues...

          Show
          lmccay Larry McCay added a comment - More checkstyle issues...
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 38s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 29s There were no new javac warning messages.
          +1 javadoc 9m 42s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 4s The applied patch generated 8 new checkstyle issues (total was 16, now 10).
          -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 22m 45s Tests passed in hadoop-common.
              59m 53s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731809/HADOOP-11934.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4536399
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6579/artifact/patchprocess/diffcheckstylehadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6579/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6579/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6579/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6579/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 42s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 4s The applied patch generated 8 new checkstyle issues (total was 16, now 10). -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 22m 45s Tests passed in hadoop-common.     59m 53s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731809/HADOOP-11934.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4536399 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6579/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6579/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6579/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6579/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6579/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Okay - this one should handle everything left.

          Show
          lmccay Larry McCay added a comment - Okay - this one should handle everything left.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 33s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 31s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 4s The applied patch generated 8 new checkstyle issues (total was 15, now 9).
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 23m 42s Tests passed in hadoop-common.
              60m 33s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731824/HADOOP-11934.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4536399
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6581/artifact/patchprocess/diffcheckstylehadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6581/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6581/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6581/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6581/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 33s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 31s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 4s The applied patch generated 8 new checkstyle issues (total was 15, now 9). -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 23m 42s Tests passed in hadoop-common.     60m 33s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731824/HADOOP-11934.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4536399 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6581/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6581/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6581/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6581/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6581/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Patch creation issue

          Show
          lmccay Larry McCay added a comment - Patch creation issue
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 34s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 13s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 22m 23s Tests passed in hadoop-common.
              59m 31s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731831/HADOOP-11934.005.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4536399
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6582/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6582/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6582/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 34s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 13s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 22m 23s Tests passed in hadoop-common.     59m 31s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731831/HADOOP-11934.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4536399 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6582/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6582/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6582/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Hi Mike Yoder - if you would like to apply this patch and give it a whirl you will want to change your config to look like:

          localjceks://file/full/path/to/creds.jceks

          This leaves out the Hadoop FileSystem abstraction and therefore the recursive call to get passwords for LdapGroupsMapping.

          Show
          lmccay Larry McCay added a comment - Hi Mike Yoder - if you would like to apply this patch and give it a whirl you will want to change your config to look like: localjceks://file/full/path/to/creds.jceks This leaves out the Hadoop FileSystem abstraction and therefore the recursive call to get passwords for LdapGroupsMapping.
          Hide
          yoderme Mike Yoder added a comment -

          Thanks - I'll give this a try. Stay tuned...

          Show
          yoderme Mike Yoder added a comment - Thanks - I'll give this a try. Stay tuned...
          Hide
          lmccay Larry McCay added a comment -

          Hi Mike Yoder - have you had a chance to try this out yet?
          I'd rather make sure it meets your needs before I ask for reviews of it.

          Show
          lmccay Larry McCay added a comment - Hi Mike Yoder - have you had a chance to try this out yet? I'd rather make sure it meets your needs before I ask for reviews of it.
          Hide
          yoderme Mike Yoder added a comment -

          Sorry for the delay on my side. Had some unrelated cluster troubles and got distracted. Will get back to this soon.

          Show
          yoderme Mike Yoder added a comment - Sorry for the delay on my side. Had some unrelated cluster troubles and got distracted. Will get back to this soon.
          Hide
          lmccay Larry McCay added a comment -

          No worries, just want to get a fix in and you hopefully are in a good position to reproduce/test it.

          Show
          lmccay Larry McCay added a comment - No worries, just want to get a fix in and you hopefully are in a good position to reproduce/test it.
          Hide
          yoderme Mike Yoder added a comment -

          Well, it works for the hadoop ssl.server.keystore.password. So far, so good. Now for the more involved ldap case...

          Show
          yoderme Mike Yoder added a comment - Well, it works for the hadoop ssl.server.keystore.password. So far, so good. Now for the more involved ldap case...
          Hide
          lmccay Larry McCay added a comment -

          Sounds good - the ldap case is what I need though.
          I tried the easy one.

          Show
          lmccay Larry McCay added a comment - Sounds good - the ldap case is what I need though. I tried the easy one.
          Hide
          yoderme Mike Yoder added a comment -

          No more infinite loop on the namenode with the ldap bind user password set. Looking good. Although I would not consider what I did in any way an exhaustive test - I started the namenode and saw lots of messages saying that groups were all weird, as I expected that they would be. Didn't see any "ldap is screwed up" exceptions. But please don't rely on me alone for testing.

          Had a look at the code. Just two comments

          • createPermissions() seems to violate the principle of least surprise when it silently converts input longer than three chars to "700". I would've expected it to throw an error of some form. (And why 700 instead of 600?). And beyond that, all sorts of invalid input is silently ignored.
          • There's a LOT of code to convert three characters (9 bits of information!) into a set of PosixFilePermissions. Can't you convert the three chars to one int and do some bit manipulation?

          Thanks for addressing this bug so quickly!

          Show
          yoderme Mike Yoder added a comment - No more infinite loop on the namenode with the ldap bind user password set. Looking good. Although I would not consider what I did in any way an exhaustive test - I started the namenode and saw lots of messages saying that groups were all weird, as I expected that they would be. Didn't see any "ldap is screwed up" exceptions. But please don't rely on me alone for testing. Had a look at the code. Just two comments createPermissions() seems to violate the principle of least surprise when it silently converts input longer than three chars to "700". I would've expected it to throw an error of some form. (And why 700 instead of 600?). And beyond that, all sorts of invalid input is silently ignored. There's a LOT of code to convert three characters (9 bits of information!) into a set of PosixFilePermissions. Can't you convert the three chars to one int and do some bit manipulation? Thanks for addressing this bug so quickly!
          Hide
          lmccay Larry McCay added a comment -

          Great to hear about the infinite loop - that is all we really need to ensure!
          Actual group lookup won't be affected by this feature and the provider is generally tested in the TestCredentialProviderFactory test.
          There was no real way to unit test the infinite look issue.

          Your two comments are reasonable.
          The default of 700 matches the default for credential store creation from the CLI.
          I don't recall why it required execute permissions but I believe that it did.
          Given that this isn't a configurable argument, I think it is fine the way it is.
          I'll look at changing it though.

          I agree - all that code for getting to the PosixFilePermissions is annoying.
          I'll see what I can do there as well.

          Thanks for the testing and the review, Mike!
          Much appreciated.

          Show
          lmccay Larry McCay added a comment - Great to hear about the infinite loop - that is all we really need to ensure! Actual group lookup won't be affected by this feature and the provider is generally tested in the TestCredentialProviderFactory test. There was no real way to unit test the infinite look issue. Your two comments are reasonable. The default of 700 matches the default for credential store creation from the CLI. I don't recall why it required execute permissions but I believe that it did. Given that this isn't a configurable argument, I think it is fine the way it is. I'll look at changing it though. I agree - all that code for getting to the PosixFilePermissions is annoying. I'll see what I can do there as well. Thanks for the testing and the review, Mike! Much appreciated.
          Hide
          lmccay Larry McCay added a comment -

          Attaching new patch revision to address Mike Yoder's review comments.

          Throwing an exception with invalid permission set attempts. This is called by the base class and is unlikely to occur in the wild but this is better.

          Also, removed unnecessary code for permissions string -> PosixFilePermissions conversion - as requested.

          Show
          lmccay Larry McCay added a comment - Attaching new patch revision to address Mike Yoder 's review comments. Throwing an exception with invalid permission set attempts. This is called by the base class and is unlikely to occur in the wild but this is better. Also, removed unnecessary code for permissions string -> PosixFilePermissions conversion - as requested.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 41s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 33s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 5s The applied patch generated 12 new checkstyle issues (total was 15, now 13).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 23m 40s Tests passed in hadoop-common.
              60m 44s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732928/HADOOP-11934.006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 15ccd96
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6692/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6692/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6692/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6692/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 41s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 5s The applied patch generated 12 new checkstyle issues (total was 15, now 13). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 23m 40s Tests passed in hadoop-common.     60m 44s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732928/HADOOP-11934.006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 15ccd96 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6692/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6692/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6692/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6692/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Removed unnecessary import.

          Show
          lmccay Larry McCay added a comment - Removed unnecessary import.
          Hide
          lmccay Larry McCay added a comment -

          Checkstyle fixes

          Show
          lmccay Larry McCay added a comment - Checkstyle fixes
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 35s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 28s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 6s The applied patch generated 1 new checkstyle issues (total was 15, now 2).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 22m 48s Tests passed in hadoop-common.
              59m 49s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732950/HADOOP-11934.008.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 15ccd96
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6695/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6695/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6695/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6695/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 6s The applied patch generated 1 new checkstyle issues (total was 15, now 2). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 22m 48s Tests passed in hadoop-common.     59m 49s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732950/HADOOP-11934.008.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 15ccd96 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6695/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6695/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6695/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6695/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Removed tabs...

          Show
          lmccay Larry McCay added a comment - Removed tabs...
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 45s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 5s The applied patch generated 2 new checkstyle issues (total was 15, now 3).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 23m 3s Tests passed in hadoop-common.
              60m 11s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733057/HADOOP-11934.009.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / cbc01ed
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6700/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6700/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6700/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6700/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 45s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 5s The applied patch generated 2 new checkstyle issues (total was 15, now 3). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 23m 3s Tests passed in hadoop-common.     60m 11s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733057/HADOOP-11934.009.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cbc01ed checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6700/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6700/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6700/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6700/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Checkstyle - line length...

          Show
          lmccay Larry McCay added a comment - Checkstyle - line length...
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 28s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 27s There were no new javac warning messages.
          +1 javadoc 9m 31s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 4s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 23m 0s Tests passed in hadoop-common.
              59m 39s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733063/HADOOP-11934.010.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ee7beda
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6701/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6701/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6701/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 28s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 27s There were no new javac warning messages. +1 javadoc 9m 31s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 4s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 23m 0s Tests passed in hadoop-common.     59m 39s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733063/HADOOP-11934.010.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ee7beda hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6701/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6701/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6701/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Hi Larry McCay. This looks great overall! Here are a few comments, mostly minor.

          1. Both AbstractJavaKeyStoreProvider and LocalJavaKeyStoreProvider have copied some class-level JavaDocs from JavaKeyStoreProvider. This isn't completely accurate, because those comments talk about pointing to different FileSystem implementations. Could you please revise this?
          2. AbstractJavaKeyStoreProvider constructor: The trunk version of the following code would trim the password. Do we need to keep that?
                      try (InputStream is = pwdFile.openStream()) {
                        password = IOUtils.toCharArray(is);
                      }
            
                      try (InputStream is = pwdFile.openStream()) {
                        password = IOUtils.toString(is).trim().toCharArray();
                      }
            
          3. AbstractJavaKeyStoreProvider#bytesToChars: The existing trunk code used Charsets#UTF_8 to avoid the need to handle UnsupportedEncodingException. Shall we keep it the same, or was this an intentional change?
          4. AbstractJavaKeyStoreProvider#getPathAsString: This has the same implementation in both subclasses. Would it make sense to refactor that up to the base class as a protected final method?
          5. JavaKeyStoreProvider#getOutputStreamForKeystore: This isn't a new thing with your patch, but I wanted to mention that this overload of the FileSystem.create method is not atomic. First it creates the file with default permissions (usually 644), and then setting the requested permissions is done separately. In the case of HDFS, this is 2 separate RPCs. That means there is a brief window in which the file has default permissions. If the process dies after the first RPC but before the second, then the permissions will never be changed. To do this atomically, we'd need to switch to one of the other (much uglier) overloads of FileSystem#create. If you think changing this would be a good improvement, then I recommend queuing up a separate jira for that change, since we already have a mid-sized patch going here.
          6. JavaKeyStoreProvider and LocalJavaKeyStoreProvider: Please add the @Override annotation on all applicable methods.
          7. TestCredentialProviderFactory: After this patch, the tests fail on Windows, due to invalid string concatenation of a test directory that contains '\' characters, which are not valid URI characters. (See below.) There have been similar patches in the past to fix these tests on Windows, so you could look back at those for inspiration on how to fix this. It will probably involve some kind of usage of Path#toUri, which results in all '/' characters, which is valid URI syntax.
          java.io.IOException: Bad configuration of hadoop.security.credential.provider.path at jceks://fileC:\hdc\hadoop-common-project\hadoop-common\target\test\data\creds/test.jks
                  at java.net.URI$Parser.fail(URI.java:2829)
                  at java.net.URI$Parser.parseAuthority(URI.java:3167)
                  at java.net.URI$Parser.parseHierarchical(URI.java:3078)
                  at java.net.URI$Parser.parse(URI.java:3034)
                  at java.net.URI.<init>(URI.java:595)
                  at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:55)
                  at org.apache.hadoop.security.alias.TestCredentialProviderFactory.testFactory(TestCredentialProviderFactory.java:58)
          
          Show
          cnauroth Chris Nauroth added a comment - Hi Larry McCay . This looks great overall! Here are a few comments, mostly minor. Both AbstractJavaKeyStoreProvider and LocalJavaKeyStoreProvider have copied some class-level JavaDocs from JavaKeyStoreProvider . This isn't completely accurate, because those comments talk about pointing to different FileSystem implementations. Could you please revise this? AbstractJavaKeyStoreProvider constructor: The trunk version of the following code would trim the password. Do we need to keep that? try (InputStream is = pwdFile.openStream()) { password = IOUtils.toCharArray(is); } try (InputStream is = pwdFile.openStream()) { password = IOUtils.toString(is).trim().toCharArray(); } AbstractJavaKeyStoreProvider#bytesToChars : The existing trunk code used Charsets#UTF_8 to avoid the need to handle UnsupportedEncodingException . Shall we keep it the same, or was this an intentional change? AbstractJavaKeyStoreProvider#getPathAsString : This has the same implementation in both subclasses. Would it make sense to refactor that up to the base class as a protected final method? JavaKeyStoreProvider#getOutputStreamForKeystore : This isn't a new thing with your patch, but I wanted to mention that this overload of the FileSystem.create method is not atomic. First it creates the file with default permissions (usually 644), and then setting the requested permissions is done separately. In the case of HDFS, this is 2 separate RPCs. That means there is a brief window in which the file has default permissions. If the process dies after the first RPC but before the second, then the permissions will never be changed. To do this atomically, we'd need to switch to one of the other (much uglier) overloads of FileSystem#create . If you think changing this would be a good improvement, then I recommend queuing up a separate jira for that change, since we already have a mid-sized patch going here. JavaKeyStoreProvider and LocalJavaKeyStoreProvider : Please add the @Override annotation on all applicable methods. TestCredentialProviderFactory : After this patch, the tests fail on Windows, due to invalid string concatenation of a test directory that contains '\' characters, which are not valid URI characters. (See below.) There have been similar patches in the past to fix these tests on Windows, so you could look back at those for inspiration on how to fix this. It will probably involve some kind of usage of Path#toUri , which results in all '/' characters, which is valid URI syntax. java.io.IOException: Bad configuration of hadoop.security.credential.provider.path at jceks: //fileC:\hdc\hadoop-common-project\hadoop-common\target\test\data\creds/test.jks at java.net.URI$Parser.fail(URI.java:2829) at java.net.URI$Parser.parseAuthority(URI.java:3167) at java.net.URI$Parser.parseHierarchical(URI.java:3078) at java.net.URI$Parser.parse(URI.java:3034) at java.net.URI.<init>(URI.java:595) at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:55) at org.apache.hadoop.security.alias.TestCredentialProviderFactory.testFactory(TestCredentialProviderFactory.java:58)
          Hide
          lmccay Larry McCay added a comment -

          Hi Chris Nauroth - thank you for the detailed review!
          I will get right on it.

          Show
          lmccay Larry McCay added a comment - Hi Chris Nauroth - thank you for the detailed review! I will get right on it.
          Hide
          lmccay Larry McCay added a comment -

          Addresses Chris Nauroth's review comments.

          I will file a separate for issue #5 - as suggested.

          Show
          lmccay Larry McCay added a comment - Addresses Chris Nauroth 's review comments. I will file a separate for issue #5 - as suggested.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 12s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 34s There were no new javac warning messages.
          +1 javadoc 9m 48s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 15s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 59s Tests passed in hadoop-common.
              64m 6s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735519/HADOOP-11934-11.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / cdbd66b
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6837/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6837/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6837/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 12s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 34s There were no new javac warning messages. +1 javadoc 9m 48s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 15s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 59s Tests passed in hadoop-common.     64m 6s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735519/HADOOP-11934-11.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cdbd66b hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6837/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6837/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6837/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Ignore those last results - incorrectly run test-path.sh messed up the source and I regenerated the patch.

          Show
          lmccay Larry McCay added a comment - Ignore those last results - incorrectly run test-path.sh messed up the source and I regenerated the patch.
          Hide
          lmccay Larry McCay added a comment -

          Addressed review comments.

          Show
          lmccay Larry McCay added a comment - Addressed review comments.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thanks for addressing the feedback, Larry. A few more notes:

          1. AbstractJavaKeyStoreProvider#bytesToChars: This is a minor nit. The declaration and initialization of pass can be condensed to one line, i.e. String pass = ....
          2. JavaKeyStoreProvider#initFileSystem: Please add the @Override annotation.
          3. LocalJavaKeyStoreProvider: The class JavaDoc mentions the "jceks" scheme. Should that be changed to "localjceks"?
          4. LocalJavaKeyStoreProvider#flush: I'm sorry I didn't spot this earlier, but unfortunately, the JDK does not implement a mapping of POSIX permissions to NTFS ACLs for its Files#setPosixFilePermissions and Files#getPosixFilePermissions methods. It just throws an UnsupportedOperationException if we try to run these methods on Windows. (See test failure below.) Fortunately, we do implement that mapping in Hadoop! To make this Windows-compatible, I think we're going to need to explore using org.apache.hadoop.fs.FileUtil#setPermission for the set operation. The get operation unfortunately is more awkward, involving a combination of org.apache.hadoop.fs.Stat, org.apache.hadoop.fs.FileUtil#execCommand and org.apache.hadoop.util.Shell#getGetPermissionCommand. The high level flow for this is in org.apache.hadoop.fs.RawLocalFileSystem. Alternatively, maybe you can think of a simpler way to do a special case for Windows. Let me know.
          Running org.apache.hadoop.security.alias.TestCredentialProviderFactory
          Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.031 sec <<< FAILURE! - in org.apache.hadoop.security.alias.TestCredentialProviderFactory
          testLocalJksProvider(org.apache.hadoop.security.alias.TestCredentialProviderFactory)  Time elapsed: 0.031 sec  <<< ERROR!
          java.lang.UnsupportedOperationException: null
                  at java.nio.file.Files.setPosixFilePermissions(Files.java:1991)
                  at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.flush(LocalJavaKeyStoreProvider.java:149)
                  at org.apache.hadoop.security.alias.TestCredentialProviderFactory.checkSpecificProvider(TestCredentialProviderFactory.java:148)
                  at org.apache.hadoop.security.alias.TestCredentialProviderFactory.testLocalJksProvider(TestCredentialProviderFactory.java:220)
          
          Show
          cnauroth Chris Nauroth added a comment - Thanks for addressing the feedback, Larry. A few more notes: AbstractJavaKeyStoreProvider#bytesToChars : This is a minor nit. The declaration and initialization of pass can be condensed to one line, i.e. String pass = ... . JavaKeyStoreProvider#initFileSystem : Please add the @Override annotation. LocalJavaKeyStoreProvider : The class JavaDoc mentions the "jceks" scheme. Should that be changed to "localjceks"? LocalJavaKeyStoreProvider#flush : I'm sorry I didn't spot this earlier, but unfortunately, the JDK does not implement a mapping of POSIX permissions to NTFS ACLs for its Files#setPosixFilePermissions and Files#getPosixFilePermissions methods. It just throws an UnsupportedOperationException if we try to run these methods on Windows. (See test failure below.) Fortunately, we do implement that mapping in Hadoop! To make this Windows-compatible, I think we're going to need to explore using org.apache.hadoop.fs.FileUtil#setPermission for the set operation. The get operation unfortunately is more awkward, involving a combination of org.apache.hadoop.fs.Stat , org.apache.hadoop.fs.FileUtil#execCommand and org.apache.hadoop.util.Shell#getGetPermissionCommand . The high level flow for this is in org.apache.hadoop.fs.RawLocalFileSystem . Alternatively, maybe you can think of a simpler way to do a special case for Windows. Let me know. Running org.apache.hadoop.security.alias.TestCredentialProviderFactory Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.031 sec <<< FAILURE! - in org.apache.hadoop.security.alias.TestCredentialProviderFactory testLocalJksProvider(org.apache.hadoop.security.alias.TestCredentialProviderFactory) Time elapsed: 0.031 sec <<< ERROR! java.lang.UnsupportedOperationException: null at java.nio.file.Files.setPosixFilePermissions(Files.java:1991) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.flush(LocalJavaKeyStoreProvider.java:149) at org.apache.hadoop.security.alias.TestCredentialProviderFactory.checkSpecificProvider(TestCredentialProviderFactory.java:148) at org.apache.hadoop.security.alias.TestCredentialProviderFactory.testLocalJksProvider(TestCredentialProviderFactory.java:220)
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 13s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 47s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 2s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 35s Tests passed in hadoop-common.
              60m 24s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735523/HADOOP-11934-11.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / cdbd66b
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6841/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6841/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6841/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 47s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 2s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 35s Tests passed in hadoop-common.     60m 24s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735523/HADOOP-11934-11.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cdbd66b hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6841/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6841/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6841/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Chris Nauroth - thanks again for the review!
          #4 above - ugh - wouldn't leveraging FileUtil for this introduce the lookup of groups through LdapGroupsMapping again - putting us back at square one?

          The reason for adding the LocalJavaKeyStoreProvider was to avoid the recursive dependency on LdapGroupsMapping that we get when using LDAP based group lookup with the credential provider.

          Perhaps, we can use: http://docs.oracle.com/javase/7/docs/api/java/nio/file/attribute/AclFileAttributeView.html for this....

          Show
          lmccay Larry McCay added a comment - Chris Nauroth - thanks again for the review! #4 above - ugh - wouldn't leveraging FileUtil for this introduce the lookup of groups through LdapGroupsMapping again - putting us back at square one? The reason for adding the LocalJavaKeyStoreProvider was to avoid the recursive dependency on LdapGroupsMapping that we get when using LDAP based group lookup with the credential provider. Perhaps, we can use: http://docs.oracle.com/javase/7/docs/api/java/nio/file/attribute/AclFileAttributeView.html for this....
          Hide
          cnauroth Chris Nauroth added a comment -

          FileUtil#setPermission is a static method that's implemented entirely in terms of JDK classes like java.io.File. It doesn't interact with a Hadoop FileSystem, so I don't expect it to trigger the Hadoop group lookup machinery.

          I think using AclFileAttributeView implies that we'd need to reimplement the mapping of POSIX permissions onto NTFS ACLs. As per the following quote, the special OWNER, GROUP and EVERYONE users that would map directly to POSIX permissions are only applicable if the file system also supports PosixFileAttributeView, which Windows doesn't.

          When both the AclFileAttributeView and the PosixFileAttributeView are supported then these special user identities may be included in ACL entries that are read or written.

          This mapping is a non-trivial piece of logic. We already have that logic down in the JNI layer inside libwinutils.c, functions GetWindowsDACLs and ChangeFileModeByMask. I'm going to play around with this a bit more and come back with a recommendation for the simplest way this code can call into that logic.

          Show
          cnauroth Chris Nauroth added a comment - FileUtil#setPermission is a static method that's implemented entirely in terms of JDK classes like java.io.File . It doesn't interact with a Hadoop FileSystem , so I don't expect it to trigger the Hadoop group lookup machinery. I think using AclFileAttributeView implies that we'd need to reimplement the mapping of POSIX permissions onto NTFS ACLs. As per the following quote, the special OWNER, GROUP and EVERYONE users that would map directly to POSIX permissions are only applicable if the file system also supports PosixFileAttributeView , which Windows doesn't. When both the AclFileAttributeView and the PosixFileAttributeView are supported then these special user identities may be included in ACL entries that are read or written. This mapping is a non-trivial piece of logic. We already have that logic down in the JNI layer inside libwinutils.c, functions GetWindowsDACLs and ChangeFileModeByMask . I'm going to play around with this a bit more and come back with a recommendation for the simplest way this code can call into that logic.
          Hide
          cnauroth Chris Nauroth added a comment -

          It looks like our best option is adding a bit of special-case logic for Windows in LocalJavaKeyStoreProvider. In flush, we can check for Windows and call FileUtil#setPermission. The Set<PosixFilePermission> would need to get converted by calling FsPermission#valueOf. In stashOriginalFilePermissions, Windows would need to issue an external winutils call using Shell#getGetPermissionCommand. The returned string can be parsed back to a Set<PosixFilePermission>.

          After we implement HADOOP-11935 (native stat call), we can come back to some of this code and simplify.

          Show
          cnauroth Chris Nauroth added a comment - It looks like our best option is adding a bit of special-case logic for Windows in LocalJavaKeyStoreProvider . In flush , we can check for Windows and call FileUtil#setPermission . The Set<PosixFilePermission> would need to get converted by calling FsPermission#valueOf . In stashOriginalFilePermissions , Windows would need to issue an external winutils call using Shell#getGetPermissionCommand . The returned string can be parsed back to a Set<PosixFilePermission> . After we implement HADOOP-11935 (native stat call), we can come back to some of this code and simplify.
          Hide
          lmccay Larry McCay added a comment -

          Chris Nauroth - thanks for doing the leg work there!
          I'll see what I can do with that great insight.

          Show
          lmccay Larry McCay added a comment - Chris Nauroth - thanks for doing the leg work there! I'll see what I can do with that great insight.
          Hide
          lmccay Larry McCay added a comment -

          I've addressed each of the review items.
          I can see how HADOOP-11935 could help with the translation required in stashOriginalFilePermissions.

          I will try and follow up with a separate patch to simplify this once HADOOP-11935 is done.

          Show
          lmccay Larry McCay added a comment - I've addressed each of the review items. I can see how HADOOP-11935 could help with the translation required in stashOriginalFilePermissions. I will try and follow up with a separate patch to simplify this once HADOOP-11935 is done.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thanks, Larry. That will do it! +1 for patch v012, pending Jenkins run.

          Show
          cnauroth Chris Nauroth added a comment - Thanks, Larry. That will do it! +1 for patch v012, pending Jenkins run.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 41s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 36s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 4s The applied patch generated 1 new checkstyle issues (total was 15, now 2).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 8s Tests passed in hadoop-common.
              60m 24s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735679/HADOOP-11934.012.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4d8fb8c
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6848/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6848/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6848/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6848/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 41s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 4s The applied patch generated 1 new checkstyle issues (total was 15, now 2). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 8s Tests passed in hadoop-common.     60m 24s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735679/HADOOP-11934.012.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4d8fb8c checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6848/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6848/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6848/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6848/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          v13 addresses the one 81 char line...

          Show
          lmccay Larry McCay added a comment - v13 addresses the one 81 char line...
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 36s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 9m 22s There were no new javac warning messages.
          +1 javadoc 11m 40s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 24s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 2m 6s mvn install still works.
          +1 eclipse:eclipse 0m 41s The patch built with eclipse:eclipse.
          +1 findbugs 1m 58s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 24m 38s Tests failed in hadoop-common.
              68m 53s  



          Reason Tests
          Failed unit tests hadoop.ipc.TestIPC
            hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.security.token.delegation.web.TestWebDelegationToken



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735764/HADOOP-11934.013.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 5450413
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6851/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6851/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6851/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 9m 22s There were no new javac warning messages. +1 javadoc 11m 40s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 24s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 2m 6s mvn install still works. +1 eclipse:eclipse 0m 41s The patch built with eclipse:eclipse. +1 findbugs 1m 58s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 24m 38s Tests failed in hadoop-common.     68m 53s   Reason Tests Failed unit tests hadoop.ipc.TestIPC   hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.security.token.delegation.web.TestWebDelegationToken Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735764/HADOOP-11934.013.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5450413 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6851/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6851/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6851/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Those test failures are unrelated to this patch.
          I'll resubmit the patch to run again.

          Show
          lmccay Larry McCay added a comment - Those test failures are unrelated to this patch. I'll resubmit the patch to run again.
          Hide
          cnauroth Chris Nauroth added a comment -

          I agree that the test failures in the last Jenkins run are unrelated. I submitted a new run manually:

          https://builds.apache.org/job/PreCommit-HADOOP-Build/6856/

          Show
          cnauroth Chris Nauroth added a comment - I agree that the test failures in the last Jenkins run are unrelated. I submitted a new run manually: https://builds.apache.org/job/PreCommit-HADOOP-Build/6856/
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 41s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 3s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 22m 8s Tests failed in hadoop-common.
              59m 22s  



          Reason Tests
          Failed unit tests hadoop.security.token.delegation.web.TestWebDelegationToken



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735764/HADOOP-11934.013.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f1cea9c
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6856/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6856/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6856/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 41s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 3s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 8s Tests failed in hadoop-common.     59m 22s   Reason Tests Failed unit tests hadoop.security.token.delegation.web.TestWebDelegationToken Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735764/HADOOP-11934.013.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1cea9c hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6856/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6856/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6856/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          The last test failure was an unrelated socket bind error. This test relies on starting a server socket to discover an ephemeral port, immediately closing the server socket, and then saving the discovered ephemeral port for later use. There is an inherent race condition here between discovering the port and then binding it later in the main body of the test. It's unrelated to this patch.

          I am +1 for patch v013. I plan to commit it later today.

          Show
          cnauroth Chris Nauroth added a comment - The last test failure was an unrelated socket bind error. This test relies on starting a server socket to discover an ephemeral port, immediately closing the server socket, and then saving the discovered ephemeral port for later use. There is an inherent race condition here between discovering the port and then binding it later in the main body of the test. It's unrelated to this patch. I am +1 for patch v013. I plan to commit it later today.
          Hide
          lmccay Larry McCay added a comment -

          Great - thanks, Chris Nauroth!

          Show
          lmccay Larry McCay added a comment - Great - thanks, Chris Nauroth !
          Hide
          cnauroth Chris Nauroth added a comment -

          I have committed this to trunk, branch-2 and branch-2.7. Larry McCay, thank you for the contribution. This was a tricky one in multiple ways!

          Show
          cnauroth Chris Nauroth added a comment - I have committed this to trunk, branch-2 and branch-2.7. Larry McCay , thank you for the contribution. This was a tricky one in multiple ways!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7921 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7921/)
          HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. (cnauroth: rev 860b8373c3a851386b8cd2d4265dd35e5aabc941)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredentialProviderFactory.java
          • hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.alias.CredentialProviderFactory
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7921 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7921/ ) HADOOP-11934 . Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. (cnauroth: rev 860b8373c3a851386b8cd2d4265dd35e5aabc941) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredentialProviderFactory.java hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.alias.CredentialProviderFactory hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #212 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/212/)
          HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. (cnauroth: rev 860b8373c3a851386b8cd2d4265dd35e5aabc941)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.alias.CredentialProviderFactory
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredentialProviderFactory.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #212 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/212/ ) HADOOP-11934 . Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. (cnauroth: rev 860b8373c3a851386b8cd2d4265dd35e5aabc941) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.alias.CredentialProviderFactory hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredentialProviderFactory.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #942 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/942/)
          HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. (cnauroth: rev 860b8373c3a851386b8cd2d4265dd35e5aabc941)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.alias.CredentialProviderFactory
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredentialProviderFactory.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #942 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/942/ ) HADOOP-11934 . Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. (cnauroth: rev 860b8373c3a851386b8cd2d4265dd35e5aabc941) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.alias.CredentialProviderFactory hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredentialProviderFactory.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2140 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2140/)
          HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. (cnauroth: rev 860b8373c3a851386b8cd2d4265dd35e5aabc941)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.alias.CredentialProviderFactory
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredentialProviderFactory.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2140 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2140/ ) HADOOP-11934 . Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. (cnauroth: rev 860b8373c3a851386b8cd2d4265dd35e5aabc941) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.alias.CredentialProviderFactory hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredentialProviderFactory.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #201 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/201/)
          HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. (cnauroth: rev 860b8373c3a851386b8cd2d4265dd35e5aabc941)

          • hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.alias.CredentialProviderFactory
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredentialProviderFactory.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #201 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/201/ ) HADOOP-11934 . Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. (cnauroth: rev 860b8373c3a851386b8cd2d4265dd35e5aabc941) hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.alias.CredentialProviderFactory hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredentialProviderFactory.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #210 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/210/)
          HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. (cnauroth: rev 860b8373c3a851386b8cd2d4265dd35e5aabc941)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.alias.CredentialProviderFactory
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredentialProviderFactory.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #210 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/210/ ) HADOOP-11934 . Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. (cnauroth: rev 860b8373c3a851386b8cd2d4265dd35e5aabc941) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.alias.CredentialProviderFactory hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredentialProviderFactory.java
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Pulled this into 2.6.1 after fixing merge conflicts and removing java-7'isms.

          Ran compilation and TestCredentialProviderFactory before pushing the fix.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.6.1 after fixing merge conflicts and removing java-7'isms. Ran compilation and TestCredentialProviderFactory before pushing the fix.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Attaching patch that I committed to 2.6.1.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Attaching patch that I committed to 2.6.1.

            People

            • Assignee:
              lmccay Larry McCay
              Reporter:
              yoderme Mike Yoder
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development