Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12076

Incomplete Cache Mechanism in CredentialProvider API

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The AbstractJavaKeyStoreProvider class in the CredentialProvider API has a cache member variable and interrogation of it during access but does not populate it.

      1. HADOOP-12076-001.patch
        3 kB
        Larry McCay
      2. HADOOP-12076-002.patch
        2 kB
        Larry McCay
      3. HADOOP-12076-003.patch
        3 kB
        Larry McCay
      4. HADOOP-12076-004.patch
        2 kB
        Larry McCay

        Issue Links

          Activity

          Hide
          lmccay Larry McCay added a comment -

          Adds the population of the keystore provider cache and a test for verifying its functionality.

          Show
          lmccay Larry McCay added a comment - Adds the population of the keystore provider cache and a test for verifying its functionality.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 31s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 3s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 53s Tests passed in hadoop-common.
              62m 2s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738600/HADOOP-12076-001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8d0ef31
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6944/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6944/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6944/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 31s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 3s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 53s Tests passed in hadoop-common.     62m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738600/HADOOP-12076-001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8d0ef31 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6944/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6944/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6944/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          I'd like to request a review of this simple patch - Chris Nauroth, Sean Busbey, Owen O'Malley?
          Thanks!

          Show
          lmccay Larry McCay added a comment - I'd like to request a review of this simple patch - Chris Nauroth , Sean Busbey , Owen O'Malley ? Thanks!
          Hide
          busbey Sean Busbey added a comment -

          General comments

          • Are we worried about how the cache gets aged off?
          • Since AbstractJavaKeyStore isn't thread safe, do we know what happens if multiple instances are pointing at the same jks file?
          • Presuming the above works, how do we reconcile changes that happen to the underlying jks against the cache?
          +
          +    // delete the actual keystore
          +    file.delete();
          +    // make sure the password is cached
          +    assertArrayEquals(passwd, provider.getCredentialEntry("pass").getCredential());
          +  }
          

          You should do something to ensure that deleting the file means that non-cached entries won't be returned.

          Show
          busbey Sean Busbey added a comment - General comments Are we worried about how the cache gets aged off? Since AbstractJavaKeyStore isn't thread safe, do we know what happens if multiple instances are pointing at the same jks file? Presuming the above works, how do we reconcile changes that happen to the underlying jks against the cache? + + // delete the actual keystore + file.delete(); + // make sure the password is cached + assertArrayEquals(passwd, provider.getCredentialEntry( "pass" ).getCredential()); + } You should do something to ensure that deleting the file means that non-cached entries won't be returned.
          Hide
          lmccay Larry McCay added a comment -

          Hi Sean Busbey - thanks for the comments and review:

          Are we worried about how the cache gets aged off?

          I don't think so - at least not in this jira. The purpose of this patch is to complete the mechanism that should be on par with that of the JavaKeyStoreProvider in the KeyProvider API. If we want to add a TTL or the like then I think that should be a follow on jira as an enhancement.

          Since AbstractJavaKeyStore isn't thread safe, do we know what happens if multiple instances are pointing at the same jks file?

          I'm not entirely sure of the usecase that you have in mind. There are read/write locks in this class for providing thread safety. Perhaps there are state issues not covered properly? If so, we should file separate jiras for them. At any rate, I assume that you question is in the context of the cache. Multiple credential provider instances pointing at the same file should be fine assuming that these credentials are generally rather static. The initial usecases are primarily around the passwords that would otherwise be in config files such as those for SSL related keystores or LDAP bind passwords, etc. Therefore, multiple readers are largely not a problem. Now, if someone changes any of those stored passwords the consumers of the credential provider will either need to instantiate new providers (which the Configuration.getPassword call does on each call anyway) or restart.

          Presuming the above works, how do we reconcile changes that happen to the underlying jks against the cache?

          I think this is largely answered by in the previous question. I guess if passwords are added to the store that they will be picked up without any need for restart or new instances - due to them not being cached. If you used the CLI and deleted an SSL password it would still be returned by the cache until restart or new instances (again, Configuration.getPassword isn't a problem there). If you change the existing password to a keystore then it will probably continue to work until you restart and try to load the certs again. In which case, if the new password matches the keystore it will work otherwise it won't.

          You should do something to ensure that deleting the file means that non-cached entries won't be returned.

          That's a good idea - I'll add that additional check to ensure that the keystore is actually gone and whether we blow up - which we probably will. Will have to determine whether that is good or bad...

          Show
          lmccay Larry McCay added a comment - Hi Sean Busbey - thanks for the comments and review: Are we worried about how the cache gets aged off? I don't think so - at least not in this jira. The purpose of this patch is to complete the mechanism that should be on par with that of the JavaKeyStoreProvider in the KeyProvider API. If we want to add a TTL or the like then I think that should be a follow on jira as an enhancement. Since AbstractJavaKeyStore isn't thread safe, do we know what happens if multiple instances are pointing at the same jks file? I'm not entirely sure of the usecase that you have in mind. There are read/write locks in this class for providing thread safety. Perhaps there are state issues not covered properly? If so, we should file separate jiras for them. At any rate, I assume that you question is in the context of the cache. Multiple credential provider instances pointing at the same file should be fine assuming that these credentials are generally rather static. The initial usecases are primarily around the passwords that would otherwise be in config files such as those for SSL related keystores or LDAP bind passwords, etc. Therefore, multiple readers are largely not a problem. Now, if someone changes any of those stored passwords the consumers of the credential provider will either need to instantiate new providers (which the Configuration.getPassword call does on each call anyway) or restart. Presuming the above works, how do we reconcile changes that happen to the underlying jks against the cache? I think this is largely answered by in the previous question. I guess if passwords are added to the store that they will be picked up without any need for restart or new instances - due to them not being cached. If you used the CLI and deleted an SSL password it would still be returned by the cache until restart or new instances (again, Configuration.getPassword isn't a problem there). If you change the existing password to a keystore then it will probably continue to work until you restart and try to load the certs again. In which case, if the new password matches the keystore it will work otherwise it won't. You should do something to ensure that deleting the file means that non-cached entries won't be returned. That's a good idea - I'll add that additional check to ensure that the keystore is actually gone and whether we blow up - which we probably will. Will have to determine whether that is good or bad...
          Hide
          lmccay Larry McCay added a comment -

          Interestingly, based on trying to change the test to ensure that non-cached items are not returned when the underlying store is deleted, it seems that the in-memory keystore instance itself serves as a cache. I have found that when a credentialEntry is added to the in-memory it is always returned even if the underlying jks is deleted and the value wasn't queried prior. I even persisted the store with a flush() and instantiated a new provider. The act of loading the keystore reads everything into memory - so, even when I remove the file it is still returned by the getCredentialEntry since it is in the in-memory keystore. It doesn't even need to be in the cache.

          Not sure what value the additional cache adds here. There may be some overhead to pulling it out of the keystore and the KeyEntry but not sure.

          Show
          lmccay Larry McCay added a comment - Interestingly, based on trying to change the test to ensure that non-cached items are not returned when the underlying store is deleted, it seems that the in-memory keystore instance itself serves as a cache. I have found that when a credentialEntry is added to the in-memory it is always returned even if the underlying jks is deleted and the value wasn't queried prior. I even persisted the store with a flush() and instantiated a new provider. The act of loading the keystore reads everything into memory - so, even when I remove the file it is still returned by the getCredentialEntry since it is in the in-memory keystore. It doesn't even need to be in the cache. Not sure what value the additional cache adds here. There may be some overhead to pulling it out of the keystore and the KeyEntry but not sure.
          Hide
          busbey Sean Busbey added a comment -

          Since AbstractJavaKeyStore isn't thread safe, do we know what happens if multiple instances are pointing at the same jks file?

          I'm not entirely sure of the usecase that you have in mind. There are read/write locks in this class for providing thread safety. Perhaps there are state issues not covered properly? If so, we should file separate jiras for them.

          The getCache method means we have no control over how the HashMap backing the cache is accessed. A follow-on jira is fine by me.

          Presuming the above works, how do we reconcile changes that happen to the underlying jks against the cache?

          I think this is largely answered by in the previous question. I guess if passwords are added to the store that they will be picked up without any need for restart or new instances - due to them not being cached. If you used the CLI and deleted an SSL password it would still be returned by the cache until restart or new instances (again, Configuration.getPassword isn't a problem there). If you change the existing password to a keystore then it will probably continue to work until you restart and try to load the certs again. In which case, if the new password matches the keystore it will work otherwise it won't.

          Since JavaKeyStoreProvider has already been in released versions, any caching done here has to maintain the same behavior as found in 2.6.0 and 2.7.0 (or the change needs to be marked incompatible and release noted).

          Interestingly, based on trying to change the test to ensure that non-cached items are not returned when the underlying store is deleted, it seems that the in-memory keystore instance itself serves as a cache. I have found that when a credentialEntry is added to the in-memory it is always returned even if the underlying jks is deleted and the value wasn't queried prior. I even persisted the store with a flush() and instantiated a new provider. The act of loading the keystore reads everything into memory - so, even when I remove the file it is still returned by the getCredentialEntry since it is in the in-memory keystore. It doesn't even need to be in the cache.

          Not sure what value the additional cache adds here. There may be some overhead to pulling it out of the keystore and the KeyEntry but not sure.

          Does the loaded KeyStore recognize (non-deleted) changes to the underlying file after it's been loaded? It's possible that while the file still exists the keystore could use filesystem stat calls to find out if it has changed and update its cache as appropriate.

          Sounds like we should change the test to be backed by MiniDFS instead of a local file uri? That should give us a better idea of what's happening in ACCUMULO-3890.

          Show
          busbey Sean Busbey added a comment - Since AbstractJavaKeyStore isn't thread safe, do we know what happens if multiple instances are pointing at the same jks file? I'm not entirely sure of the usecase that you have in mind. There are read/write locks in this class for providing thread safety. Perhaps there are state issues not covered properly? If so, we should file separate jiras for them. The getCache method means we have no control over how the HashMap backing the cache is accessed. A follow-on jira is fine by me. Presuming the above works, how do we reconcile changes that happen to the underlying jks against the cache? I think this is largely answered by in the previous question. I guess if passwords are added to the store that they will be picked up without any need for restart or new instances - due to them not being cached. If you used the CLI and deleted an SSL password it would still be returned by the cache until restart or new instances (again, Configuration.getPassword isn't a problem there). If you change the existing password to a keystore then it will probably continue to work until you restart and try to load the certs again. In which case, if the new password matches the keystore it will work otherwise it won't. Since JavaKeyStoreProvider has already been in released versions, any caching done here has to maintain the same behavior as found in 2.6.0 and 2.7.0 (or the change needs to be marked incompatible and release noted). Interestingly, based on trying to change the test to ensure that non-cached items are not returned when the underlying store is deleted, it seems that the in-memory keystore instance itself serves as a cache. I have found that when a credentialEntry is added to the in-memory it is always returned even if the underlying jks is deleted and the value wasn't queried prior. I even persisted the store with a flush() and instantiated a new provider. The act of loading the keystore reads everything into memory - so, even when I remove the file it is still returned by the getCredentialEntry since it is in the in-memory keystore. It doesn't even need to be in the cache. Not sure what value the additional cache adds here. There may be some overhead to pulling it out of the keystore and the KeyEntry but not sure. Does the loaded KeyStore recognize (non-deleted) changes to the underlying file after it's been loaded? It's possible that while the file still exists the keystore could use filesystem stat calls to find out if it has changed and update its cache as appropriate. Sounds like we should change the test to be backed by MiniDFS instead of a local file uri? That should give us a better idea of what's happening in ACCUMULO-3890 .
          Hide
          lmccay Larry McCay added a comment -

          I just played around with more test code to check your last question.

          Does the loaded KeyStore recognize (non-deleted) changes to the underlying file after it's been loaded? It's possible that while the file still exists the keystore could use filesystem stat calls to find out if it has changed and update its cache as appropriate.

          I created a new provider and added a new password and persisted the keystore then attempted to get it from the older provider and it returns null as I expected.

          The values are loaded at provider creation - any added credentials that are needed by a running process requires a restart or a new provider instance.

          Show
          lmccay Larry McCay added a comment - I just played around with more test code to check your last question. Does the loaded KeyStore recognize (non-deleted) changes to the underlying file after it's been loaded? It's possible that while the file still exists the keystore could use filesystem stat calls to find out if it has changed and update its cache as appropriate. I created a new provider and added a new password and persisted the keystore then attempted to get it from the older provider and it returns null as I expected. The values are loaded at provider creation - any added credentials that are needed by a running process requires a restart or a new provider instance.
          Hide
          lmccay Larry McCay added a comment -

          I think that given the fact that the keystore is already caching the values as we expected to do in the caching within the keystore provider that I will remove the external cache rather than complete it.

          Can anyone think of any reason to have an additional cache beyond the keystore itself?

          Show
          lmccay Larry McCay added a comment - I think that given the fact that the keystore is already caching the values as we expected to do in the caching within the keystore provider that I will remove the external cache rather than complete it. Can anyone think of any reason to have an additional cache beyond the keystore itself?
          Hide
          busbey Sean Busbey added a comment -

          I think that given the fact that the keystore is already caching the values as we expected to do in the caching within the keystore provider that I will remove the external cache rather than complete it.

          +1 on this approach.

          Show
          busbey Sean Busbey added a comment - I think that given the fact that the keystore is already caching the values as we expected to do in the caching within the keystore provider that I will remove the external cache rather than complete it. +1 on this approach.
          Hide
          lmccay Larry McCay added a comment -

          Attaching a new patch revision that removes the cache altogether since it adds no real value over the keystore itself. There are no tests for this as it doesn't change any functionality.

          Show
          lmccay Larry McCay added a comment - Attaching a new patch revision that removes the cache altogether since it adds no real value over the keystore itself. There are no tests for this as it doesn't change any functionality.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 22s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 4s The applied patch generated 2 new checkstyle issues (total was 0, now 2).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 50s Tests passed in hadoop-common.
              62m 51s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738934/HADOOP-12076-002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a7a7768
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6952/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6952/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6952/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6952/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 22s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 4s The applied patch generated 2 new checkstyle issues (total was 0, now 2). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 50s Tests passed in hadoop-common.     62m 51s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738934/HADOOP-12076-002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a7a7768 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6952/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6952/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6952/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6952/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Removed unnecessary imports. They were already removed just not staged. :/
          Again, no tests are needed for this patch.

          Show
          lmccay Larry McCay added a comment - Removed unnecessary imports. They were already removed just not staged. :/ Again, no tests are needed for this patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 23s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 5s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 24m 4s Tests passed in hadoop-common.
              63m 8s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738960/HADOOP-12076-003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a7a7768
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6954/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6954/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6954/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 23s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 5s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 24m 4s Tests passed in hadoop-common.     63m 8s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738960/HADOOP-12076-003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a7a7768 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6954/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6954/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6954/console This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Okay - this is a clean patch according to jenkins - understanding that there are no new tests since there is no new functionality to test.

          Show
          lmccay Larry McCay added a comment - Okay - this is a clean patch according to jenkins - understanding that there are no new tests since there is no new functionality to test.
          Hide
          busbey Sean Busbey added a comment -
          @@ -209,13 +200,11 @@ protected void initFileSystem(URI keystoreUri, Configuration conf)
             @Override
             public CredentialEntry getCredentialEntry(String alias)
                 throws IOException {
          +    CredentialEntry entry = null;
               readLock.lock();
               try {
                 SecretKeySpec key = null;
                 try {
          -        if (cache.containsKey(alias)) {
          -          return cache.get(alias);
          -        }
                   if (!keyStore.containsAlias(alias)) {
                     return null;
                   }
          @@ -230,7 +219,8 @@ public CredentialEntry getCredentialEntry(String alias)
                   throw new IOException("Can't recover credential " + alias + " from "
                       + getPathAsString(), e);
                 }
          -      return new CredentialEntry(alias, bytesToChars(key.getEncoded()));
          +      entry = new CredentialEntry(alias, bytesToChars(key.getEncoded()));
          +      return entry;
               } finally {
                 readLock.unlock();
          

          nit: I don't see a reason to introduce the reference to CredentialEntry here, since there's nothing else to do with it we might as well just return as we did prior to this patch.

          Otherwise, lgtm.

          Show
          busbey Sean Busbey added a comment - @@ -209,13 +200,11 @@ protected void initFileSystem(URI keystoreUri, Configuration conf) @Override public CredentialEntry getCredentialEntry( String alias) throws IOException { + CredentialEntry entry = null ; readLock.lock(); try { SecretKeySpec key = null ; try { - if (cache.containsKey(alias)) { - return cache.get(alias); - } if (!keyStore.containsAlias(alias)) { return null ; } @@ -230,7 +219,8 @@ public CredentialEntry getCredentialEntry( String alias) throw new IOException( "Can't recover credential " + alias + " from " + getPathAsString(), e); } - return new CredentialEntry(alias, bytesToChars(key.getEncoded())); + entry = new CredentialEntry(alias, bytesToChars(key.getEncoded())); + return entry; } finally { readLock.unlock(); nit: I don't see a reason to introduce the reference to CredentialEntry here, since there's nothing else to do with it we might as well just return as we did prior to this patch. Otherwise, lgtm.
          Hide
          lmccay Larry McCay added a comment -

          I can buy that. New patch in the morning.

          Show
          lmccay Larry McCay added a comment - I can buy that. New patch in the morning.
          Hide
          lmccay Larry McCay added a comment -

          v4 addresses Sean Busbey's comment on the previous version.

          Show
          lmccay Larry McCay added a comment - v4 addresses Sean Busbey 's comment on the previous version.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 16s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 29s There were no new javac warning messages.
          +1 javadoc 9m 32s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 4s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 20s Tests passed in hadoop-common.
              62m 2s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739037/HADOOP-12076-004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 95c73d4
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6958/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6958/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6958/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 16s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 4s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 20s Tests passed in hadoop-common.     62m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739037/HADOOP-12076-004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 95c73d4 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6958/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6958/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6958/console This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          +1

          Show
          busbey Sean Busbey added a comment - +1
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for the patch. I committed this to trunk and branch-2.

          Larry, thank you for the patch and for the deep investigation of how the existing keystore code is working. Sean, thank you for your help with a very thorough code review.

          Show
          cnauroth Chris Nauroth added a comment - +1 for the patch. I committed this to trunk and branch-2. Larry, thank you for the patch and for the deep investigation of how the existing keystore code is working. Sean, thank you for your help with a very thorough code review.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8028 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8028/)
          HADOOP-12076. Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8028 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8028/ ) HADOOP-12076 . Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #961 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/961/)
          HADOOP-12076. Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #961 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/961/ ) HADOOP-12076 . Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #231 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/231/)
          HADOOP-12076. Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #231 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/231/ ) HADOOP-12076 . Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #220 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/220/)
          HADOOP-12076. Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #220 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/220/ ) HADOOP-12076 . Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #229 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/229/)
          HADOOP-12076. Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #229 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/229/ ) HADOOP-12076 . Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2177 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2177/)
          HADOOP-12076. Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2177 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2177/ ) HADOOP-12076 . Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2159/)
          HADOOP-12076. Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2159/ ) HADOOP-12076 . Incomplete Cache Mechanism in CredentialProvider API. Contributed by Larry McCay. (cnauroth: rev fbf55dcaf45285e1795cb107e7846799e4042b0b) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java hadoop-common-project/hadoop-common/CHANGES.txt

            People

            • Assignee:
              lmccay Larry McCay
              Reporter:
              lmccay Larry McCay
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development