Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13864

KMS should not require truststore password

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: kms, security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Trust store passwords are actually not required for read operations. They're only needed for writing to the trust store; in reads they serve as an integrity check. Normal hadoop sslclient.xml files don't require the truststore password, but when the KMS is used it's required.

      If I don't specify a hadoop trust store password I get:

      Failed to start namenode.
      java.io.IOException: java.security.GeneralSecurityException: The property 'ssl.client.truststore.password' has not been set in the ssl configuration file.
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.<init>(KMSClientProvider.java:428)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$Factory.createProvider(KMSClientProvider.java:333)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$Factory.createProvider(KMSClientProvider.java:324)
      	at org.apache.hadoop.crypto.key.KeyProviderFactory.get(KeyProviderFactory.java:95)
      	at org.apache.hadoop.util.KMSUtil.createKeyProvider(KMSUtil.java:65)
      	at org.apache.hadoop.hdfs.DFSUtil.createKeyProvider(DFSUtil.java:1920)
      	at org.apache.hadoop.hdfs.DFSUtil.createKeyProviderCryptoExtension(DFSUtil.java:1934)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:811)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:770)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:614)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:676)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:844)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:823)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1548)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1616)
      Caused by: java.security.GeneralSecurityException: The property 'ssl.client.truststore.password' has not been set in the ssl configuration file.
      	at org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:199)
      	at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:131)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.<init>(KMSClientProvider.java:426)
      	... 14 more
      

      Note that this does not happen to the namenode when the kms isn't in use.

        Issue Links

          Activity

          Hide
          yoderme Mike Yoder added a comment -

          To the question of "you're not requiring a password now, isn't that a bad (less secure) thing?" I reply:

          My first agrument is that of symmetry. For C/C++/Python programs (anything using openssl), a trust store is a plain-text file containing certificates. No password required, and indeed there is not even a way to password-protect it. So this "protection" in java has never been thought worthy of a feature in openssl. Note that since all our certificates need to be in both pem and jks format the passwordless trust stores will continue to exist in pem format regardless of what we do in java programs.

          My second argument is that the truststore password is worthless anyway. It could in theory be useful in the limited world of keytool generating a truststore, but when you actually go to use that truststore it all falls apart. The reason is that hadoop clients need the trust store in order to trust the server that they're talking to. Since the client needs it, the client has to be able to fully use the trust store. If the trust store password is given, then the client (anyone who connects to the hadoop cluster, that is) then knows the trust store password. There is no way around this: even if we try to encrypt that password, we would have to give the client the decryption key. Even if we tried to obfuscate that password, we'd have to unobfuscate the password before using it.

          The other thing to consider here is that customers frequently re-use the trust store password to be the same as the keystore password. This is dumb, but it happens, and now the password is spread far and wide. The "benefit" is that the integrity of the truststore is cryptographically verified. But since essentially anyone can learn that password, anyone could write to the truststore, so... who cares?

          My third argument is that the global trust store on the system has a well known password of "changeit" (even though changing it is pointless) and no software ever accesses the global trust store using this password - because it would provide no benefit.

          Show
          yoderme Mike Yoder added a comment - To the question of "you're not requiring a password now, isn't that a bad (less secure) thing?" I reply: My first agrument is that of symmetry. For C/C++/Python programs (anything using openssl), a trust store is a plain-text file containing certificates. No password required, and indeed there is not even a way to password-protect it. So this "protection" in java has never been thought worthy of a feature in openssl. Note that since all our certificates need to be in both pem and jks format the passwordless trust stores will continue to exist in pem format regardless of what we do in java programs. My second argument is that the truststore password is worthless anyway. It could in theory be useful in the limited world of keytool generating a truststore, but when you actually go to use that truststore it all falls apart. The reason is that hadoop clients need the trust store in order to trust the server that they're talking to. Since the client needs it, the client has to be able to fully use the trust store. If the trust store password is given, then the client (anyone who connects to the hadoop cluster, that is) then knows the trust store password. There is no way around this: even if we try to encrypt that password, we would have to give the client the decryption key. Even if we tried to obfuscate that password, we'd have to unobfuscate the password before using it. The other thing to consider here is that customers frequently re-use the trust store password to be the same as the keystore password. This is dumb, but it happens, and now the password is spread far and wide. The "benefit" is that the integrity of the truststore is cryptographically verified. But since essentially anyone can learn that password, anyone could write to the truststore, so... who cares? My third argument is that the global trust store on the system has a well known password of "changeit" (even though changing it is pointless) and no software ever accesses the global trust store using this password - because it would provide no benefit.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 39s trunk passed
          +1 compile 10m 18s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 11s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 37s trunk passed
          +1 javadoc 0m 51s trunk passed
          +1 mvninstall 0m 44s the patch passed
          +1 compile 10m 5s the patch passed
          +1 javac 10m 5s the patch passed
          +1 checkstyle 0m 29s the patch passed
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 40s the patch passed
          +1 javadoc 0m 48s the patch passed
          -1 unit 7m 37s hadoop-common in the patch failed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          47m 47s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13864
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841658/HADOOP-13864.000.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f060250aaaf1 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f885160
          Default Java 1.8.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11194/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11194/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11194/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 39s trunk passed +1 compile 10m 18s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 11s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 37s trunk passed +1 javadoc 0m 51s trunk passed +1 mvninstall 0m 44s the patch passed +1 compile 10m 5s the patch passed +1 javac 10m 5s the patch passed +1 checkstyle 0m 29s the patch passed +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 40s the patch passed +1 javadoc 0m 48s the patch passed -1 unit 7m 37s hadoop-common in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 47m 47s Reason Tests Failed junit tests hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13864 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841658/HADOOP-13864.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f060250aaaf1 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f885160 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11194/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11194/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11194/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Mike Yoder for reporting the issue and providing a patch!
          +1, will commit this end of today if no objections.

          Show
          xiaochen Xiao Chen added a comment - Thanks Mike Yoder for reporting the issue and providing a patch! +1, will commit this end of today if no objections.
          Hide
          xiaochen Xiao Chen added a comment -

          Failed test looks unrelated and passed locally. This commit results in a behavior change, but since as Mike explained here I think we should consider it a bug, hence not marking incompatible.

          Show
          xiaochen Xiao Chen added a comment - Failed test looks unrelated and passed locally. This commit results in a behavior change, but since as Mike explained here I think we should consider it a bug, hence not marking incompatible.
          Hide
          xiaochen Xiao Chen added a comment -

          Committed to trunk. Thanks Mike Yoder for the contribution, and Andrew Wang for offline reviews!

          Show
          xiaochen Xiao Chen added a comment - Committed to trunk. Thanks Mike Yoder for the contribution, and Andrew Wang for offline reviews!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10943 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10943/)
          HADOOP-13864. KMS should not require truststore password. Contributed by (xiao: rev a2b5d602201a4f619f6a68ec2168a884190d8de6)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/ReloadingX509TrustManager.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/FileBasedKeyStoresFactory.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/TestReloadingX509TrustManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10943 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10943/ ) HADOOP-13864 . KMS should not require truststore password. Contributed by (xiao: rev a2b5d602201a4f619f6a68ec2168a884190d8de6) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/ReloadingX509TrustManager.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/FileBasedKeyStoresFactory.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/TestReloadingX509TrustManager.java

            People

            • Assignee:
              yoderme Mike Yoder
              Reporter:
              yoderme Mike Yoder
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development