Hadoop Common
  1. Hadoop Common
  2. HADOOP-10847

Remove the usage of sun.security.x509.* in testing code

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As was told by Max (Oracle), JDK9 is likely to block all accesses to sun.* classes.

      Below is from email of Andrew Purtell:

      The use of sun.* APIs to create a certificate in Hadoop and HBase test code can be removed. Someone (Intel? Oracle?) can submit a JIRA that replaces the programmatic construction with a stringified binary cert for use in the relevant unit tests.

      In Hadoop, the calls in question are below:

      hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:24:import sun.security.x509.CertificateIssuerName;
      hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:25:import sun.security.x509.CertificateSerialNumber;
      hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:26:import sun.security.x509.CertificateSubjectName;
      hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:27:import sun.security.x509.CertificateValidity;
      hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:28:import sun.security.x509.CertificateVersion;
      hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:29:import sun.security.x509.CertificateX509Key;
      hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:30:import sun.security.x509.X500Name; 
      hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:31:import sun.security.x509.X509CertImpl; 
      hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:32:import sun.security.x509.X509CertInfo;
      
      1. HADOOP-10847-addendum.patch
        3 kB
        Robert Kanter
      2. HADOOP-10847-3.patch
        6 kB
        pascal oliva
      3. HADOOP-10847-2.patch
        5 kB
        pascal oliva
      4. HADOOP-10847-1.patch
        5 kB
        pascal oliva

        Issue Links

          Activity

          Hide
          pascal oliva added a comment -

          Here HADOOP-10847-1.patch
          to update generateCertificat function by using bouncycastle library
          <dependency>
          <groupId>org.bouncycastle</groupId>
          <artifactId>bcprov-jdk16</artifactId>
          <version>1.46</version>
          <scope>test</scope>
          </dependency>

          Show
          pascal oliva added a comment - Here HADOOP-10847 -1.patch to update generateCertificat function by using bouncycastle library <dependency> <groupId>org.bouncycastle</groupId> <artifactId>bcprov-jdk16</artifactId> <version>1.46</version> <scope>test</scope> </dependency>
          Hide
          pascal oliva added a comment -

          diff --git hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/pom.xml
          index c48bb8e..e633bce 100644
          — hadoop-common-project/hadoop-common/pom.xml
          +++ hadoop-common-project/hadoop-common/pom.xml
          @@ -250,6 +250,12 @@
          <groupId>org.apache.commons</groupId>
          <artifactId>commons-compress</artifactId>
          </dependency>
          + <dependency>
          + <groupId>org.bouncycastle</groupId>
          + <artifactId>bcprov-jdk16</artifactId>
          + <version>1.46</version>
          + <scope>test</scope>
          + </dependency>
          </dependencies>

          <build>
          diff --git hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          index a07faeb..9a68b30 100644
          — hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          +++ hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          @@ -19,17 +19,6 @@
          package org.apache.hadoop.security.ssl;

          import org.apache.hadoop.conf.Configuration;
          -import sun.security.x509.AlgorithmId;
          -import sun.security.x509.CertificateAlgorithmId;
          -import sun.security.x509.CertificateIssuerName;
          -import sun.security.x509.CertificateSerialNumber;
          -import sun.security.x509.CertificateSubjectName;
          -import sun.security.x509.CertificateValidity;
          -import sun.security.x509.CertificateVersion;
          -import sun.security.x509.CertificateX509Key;
          -import sun.security.x509.X500Name;
          -import sun.security.x509.X509CertImpl;
          -import sun.security.x509.X509CertInfo;

          import java.io.File;
          import java.io.FileOutputStream;
          @@ -52,6 +41,16 @@
          import java.util.HashMap;
          import java.util.Map;

          +import java.security.InvalidKeyException;
          +import java.security.NoSuchProviderException;
          +import java.security.SignatureException;
          +import java.security.cert.CertificateEncodingException;
          +import java.security.cert.CertificateException;
          +import java.security.cert.CertificateFactory;
          +import javax.security.auth.x500.X500Principal;
          +import org.bouncycastle.x509.X509V1CertificateGenerator;
          +
          +
          public class KeyStoreTestUtil {

          public static String getClasspathDir(Class klass) throws Exception {
          @@ -63,52 +62,40 @@ public static String getClasspathDir(Class klass) throws Exception

          { return baseDir; }

          +@SuppressWarnings("deprecation")
          /**

          • Create a self-signed X.509 Certificate.
          • @param dn the X.509 Distinguished Name, eg "CN=Test, L=London, C=GB"
          • @param pair the KeyPair
          • @param days how many days from now the Certificate is valid for
          • @param algorithm the signing algorithm, eg "SHA1withRSA"
          • @return the self-signed certificate
          • * @throws IOException thrown if an IO error ocurred.
          • * @throws GeneralSecurityException thrown if an Security error ocurred.
            */
          • public static X509Certificate generateCertificate(String dn, KeyPair pair,
          • int days, String algorithm)
          • throws GeneralSecurityException, IOException { - PrivateKey privkey = pair.getPrivate(); - X509CertInfo info = new X509CertInfo(); - Date from = new Date(); - Date to = new Date(from.getTime() + days * 86400000l); - CertificateValidity interval = new CertificateValidity(from, to); - BigInteger sn = new BigInteger(64, new SecureRandom()); - X500Name owner = new X500Name(dn); - - info.set(X509CertInfo.VALIDITY, interval); - info.set(X509CertInfo.SERIAL_NUMBER, new CertificateSerialNumber(sn)); - info.set(X509CertInfo.SUBJECT, new CertificateSubjectName(owner)); - info.set(X509CertInfo.ISSUER, new CertificateIssuerName(owner)); - info.set(X509CertInfo.KEY, new CertificateX509Key(pair.getPublic())); - info - .set(X509CertInfo.VERSION, new CertificateVersion(CertificateVersion.V3)); - AlgorithmId algo = new AlgorithmId(AlgorithmId.md5WithRSAEncryption_oid); - info.set(X509CertInfo.ALGORITHM_ID, new CertificateAlgorithmId(algo)); - - // Sign the cert to identify the algorithm that's used. - X509CertImpl cert = new X509CertImpl(info); - cert.sign(privkey, algorithm); - - // Update the algorith, and resign. - algo = (AlgorithmId) cert.get(X509CertImpl.SIG_ALG); - info - .set(CertificateAlgorithmId.NAME + "." + CertificateAlgorithmId.ALGORITHM, - algo); - cert = new X509CertImpl(info); - cert.sign(privkey, algorithm); - return cert; - }

            + public static X509Certificate generateCertificate(String dn, KeyPair pair, int days, String algorithm)
            + throws CertificateEncodingException,
            + InvalidKeyException,
            + IllegalStateException,
            + NoSuchProviderException, NoSuchAlgorithmException, SignatureException

            { + + Date from = new Date(); + Date to = new Date(from.getTime() + days * 86400000l); + BigInteger sn = new BigInteger(64, new SecureRandom()); + KeyPair keyPair = pair; + X509V1CertificateGenerator certGen = new X509V1CertificateGenerator(); + X500Principal dnName = new X500Principal(dn); + + certGen.setSerialNumber(sn); + certGen.setIssuerDN(dnName); + certGen.setNotBefore(from); + certGen.setNotAfter(to); + certGen.setSubjectDN(dnName); + certGen.setPublicKey(keyPair.getPublic()); + certGen.setSignatureAlgorithm(algorithm); + + X509Certificate cert = certGen.generate(pair.getPrivate()); + return cert; + }

          public static KeyPair generateKeyPair(String algorithm)
          throws NoSuchAlgorithmException {

          Show
          pascal oliva added a comment - diff --git hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/pom.xml index c48bb8e..e633bce 100644 — hadoop-common-project/hadoop-common/pom.xml +++ hadoop-common-project/hadoop-common/pom.xml @@ -250,6 +250,12 @@ <groupId>org.apache.commons</groupId> <artifactId>commons-compress</artifactId> </dependency> + <dependency> + <groupId>org.bouncycastle</groupId> + <artifactId>bcprov-jdk16</artifactId> + <version>1.46</version> + <scope>test</scope> + </dependency> </dependencies> <build> diff --git hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java index a07faeb..9a68b30 100644 — hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java +++ hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java @@ -19,17 +19,6 @@ package org.apache.hadoop.security.ssl; import org.apache.hadoop.conf.Configuration; -import sun.security.x509.AlgorithmId; -import sun.security.x509.CertificateAlgorithmId; -import sun.security.x509.CertificateIssuerName; -import sun.security.x509.CertificateSerialNumber; -import sun.security.x509.CertificateSubjectName; -import sun.security.x509.CertificateValidity; -import sun.security.x509.CertificateVersion; -import sun.security.x509.CertificateX509Key; -import sun.security.x509.X500Name; -import sun.security.x509.X509CertImpl; -import sun.security.x509.X509CertInfo; import java.io.File; import java.io.FileOutputStream; @@ -52,6 +41,16 @@ import java.util.HashMap; import java.util.Map; +import java.security.InvalidKeyException; +import java.security.NoSuchProviderException; +import java.security.SignatureException; +import java.security.cert.CertificateEncodingException; +import java.security.cert.CertificateException; +import java.security.cert.CertificateFactory; +import javax.security.auth.x500.X500Principal; +import org.bouncycastle.x509.X509V1CertificateGenerator; + + public class KeyStoreTestUtil { public static String getClasspathDir(Class klass) throws Exception { @@ -63,52 +62,40 @@ public static String getClasspathDir(Class klass) throws Exception { return baseDir; } +@SuppressWarnings("deprecation") /** Create a self-signed X.509 Certificate. * From http://bfo.com/blog/2011/03/08/odds_and_ends_creating_a_new_x_509_certificate.html . * @param dn the X.509 Distinguished Name, eg "CN=Test, L=London, C=GB" @param pair the KeyPair @param days how many days from now the Certificate is valid for @param algorithm the signing algorithm, eg "SHA1withRSA" @return the self-signed certificate * @throws IOException thrown if an IO error ocurred. * @throws GeneralSecurityException thrown if an Security error ocurred. */ public static X509Certificate generateCertificate(String dn, KeyPair pair, int days, String algorithm) throws GeneralSecurityException, IOException { - PrivateKey privkey = pair.getPrivate(); - X509CertInfo info = new X509CertInfo(); - Date from = new Date(); - Date to = new Date(from.getTime() + days * 86400000l); - CertificateValidity interval = new CertificateValidity(from, to); - BigInteger sn = new BigInteger(64, new SecureRandom()); - X500Name owner = new X500Name(dn); - - info.set(X509CertInfo.VALIDITY, interval); - info.set(X509CertInfo.SERIAL_NUMBER, new CertificateSerialNumber(sn)); - info.set(X509CertInfo.SUBJECT, new CertificateSubjectName(owner)); - info.set(X509CertInfo.ISSUER, new CertificateIssuerName(owner)); - info.set(X509CertInfo.KEY, new CertificateX509Key(pair.getPublic())); - info - .set(X509CertInfo.VERSION, new CertificateVersion(CertificateVersion.V3)); - AlgorithmId algo = new AlgorithmId(AlgorithmId.md5WithRSAEncryption_oid); - info.set(X509CertInfo.ALGORITHM_ID, new CertificateAlgorithmId(algo)); - - // Sign the cert to identify the algorithm that's used. - X509CertImpl cert = new X509CertImpl(info); - cert.sign(privkey, algorithm); - - // Update the algorith, and resign. - algo = (AlgorithmId) cert.get(X509CertImpl.SIG_ALG); - info - .set(CertificateAlgorithmId.NAME + "." + CertificateAlgorithmId.ALGORITHM, - algo); - cert = new X509CertImpl(info); - cert.sign(privkey, algorithm); - return cert; - } + public static X509Certificate generateCertificate(String dn, KeyPair pair, int days, String algorithm) + throws CertificateEncodingException, + InvalidKeyException, + IllegalStateException, + NoSuchProviderException, NoSuchAlgorithmException, SignatureException { + + Date from = new Date(); + Date to = new Date(from.getTime() + days * 86400000l); + BigInteger sn = new BigInteger(64, new SecureRandom()); + KeyPair keyPair = pair; + X509V1CertificateGenerator certGen = new X509V1CertificateGenerator(); + X500Principal dnName = new X500Principal(dn); + + certGen.setSerialNumber(sn); + certGen.setIssuerDN(dnName); + certGen.setNotBefore(from); + certGen.setNotAfter(to); + certGen.setSubjectDN(dnName); + certGen.setPublicKey(keyPair.getPublic()); + certGen.setSignatureAlgorithm(algorithm); + + X509Certificate cert = certGen.generate(pair.getPrivate()); + return cert; + } public static KeyPair generateKeyPair(String algorithm) throws NoSuchAlgorithmException {
          Hide
          pascal oliva added a comment -

          tested successfully with
          mvn -fn -Dtest=org.apache.hadoop.security.ssl.TestSSLFactory
          Running org.apache.hadoop.security.ssl.TestSSLFactory
          Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.318 sec - in org.apache.hadoop.security.ssl.TestSSLFactory

          and

          mvn -fn -Dtest=org.apache.hadoop.security.ssl.TestReloadingX509TrustManager
          Running org.apache.hadoop.security.ssl.TestReloadingX509TrustManager
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.829 sec - in org.apache.hadoop.security.ssl.TestReloadingX509TrustManager

          Show
          pascal oliva added a comment - tested successfully with mvn -fn -Dtest=org.apache.hadoop.security.ssl.TestSSLFactory Running org.apache.hadoop.security.ssl.TestSSLFactory Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.318 sec - in org.apache.hadoop.security.ssl.TestSSLFactory and mvn -fn -Dtest=org.apache.hadoop.security.ssl.TestReloadingX509TrustManager Running org.apache.hadoop.security.ssl.TestReloadingX509TrustManager Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.829 sec - in org.apache.hadoop.security.ssl.TestReloadingX509TrustManager
          Hide
          pascal oliva added a comment -

          command patch: git diff --no-prefix trunk > ../hadoop-patches/HADOOP-10847-1.patch

          Show
          pascal oliva added a comment - command patch: git diff --no-prefix trunk > ../hadoop-patches/ HADOOP-10847 -1.patch
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12659153/HADOOP-10847-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4398//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4398//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12659153/HADOOP-10847-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4398//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4398//console This message is automatically generated.
          Hide
          Luke Browning added a comment -

          Any idea when this will be included?

          Thanks, Luke

          Show
          Luke Browning added a comment - Any idea when this will be included? Thanks, Luke
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org
          against trunk revision e226b5b.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4969//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org against trunk revision e226b5b. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4969//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12659153/HADOOP-10847-1.patch
          against trunk revision 8984e9b.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4974//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12659153/HADOOP-10847-1.patch against trunk revision 8984e9b. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4974//console This message is automatically generated.
          Hide
          pascal oliva added a comment -

          I have rebased the patch with the trunk.

          Show
          pascal oliva added a comment - I have rebased the patch with the trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12678915/HADOOP-10847-2.patch
          against trunk revision 27715ec.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.ha.TestZKFailoverControllerStress

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5006//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5006//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678915/HADOOP-10847-2.patch against trunk revision 27715ec. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverControllerStress +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5006//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5006//console This message is automatically generated.
          Hide
          pascal oliva added a comment -

          From the trunck, I tested org.apache.hadoop.ha.TestZKFailoverControllerStress without the patch : HADOOP-10847-2.patch
          and i have got the same error :
          Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 77.234 sec <<< FAILURE! - in org.apache.hadoop.ha.TestZKFailoverControllerStress
          testExpireBackAndForth(org.apache.hadoop.ha.TestZKFailoverControllerStress) Time elapsed: 13.506 sec <<< ERROR!
          org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
          at org.apache.zookeeper.server.DataTree.getData(DataTree.java:648)

          So, This error is not related to my patch.

          Show
          pascal oliva added a comment - From the trunck, I tested org.apache.hadoop.ha.TestZKFailoverControllerStress without the patch : HADOOP-10847 -2.patch and i have got the same error : Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 77.234 sec <<< FAILURE! - in org.apache.hadoop.ha.TestZKFailoverControllerStress testExpireBackAndForth(org.apache.hadoop.ha.TestZKFailoverControllerStress) Time elapsed: 13.506 sec <<< ERROR! org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode at org.apache.zookeeper.server.DataTree.getData(DataTree.java:648) So, This error is not related to my patch.
          Hide
          Haohui Mai added a comment -

          The patch looks good to me. Nit: can you please fix the indent and trailing spaces?

          +1 after addressing it.

          Show
          Haohui Mai added a comment - The patch looks good to me. Nit: can you please fix the indent and trailing spaces? +1 after addressing it.
          Hide
          Haohui Mai added a comment -

          One more thing:

          +    <dependency>
          +	<groupId>org.bouncycastle</groupId>
          +	<artifactId>bcprov-jdk16</artifactId>
          +	<version>1.46</version>
          +    <scope>test</scope>
          +</dependency>
          

          The version of the dependency should go into hadoop-project/pom.xml, not directly into hadoop-common-project/hadoop-common/pom.xml

          Show
          Haohui Mai added a comment - One more thing: + <dependency> + <groupId>org.bouncycastle</groupId> + <artifactId>bcprov-jdk16</artifactId> + <version>1.46</version> + <scope>test</scope> +</dependency> The version of the dependency should go into hadoop-project/pom.xml , not directly into hadoop-common-project/hadoop-common/pom.xml
          Hide
          pascal oliva added a comment -

          Thx for the review, here a new patch (HADOOP-10847-3.patch) updated accordingly to your comments.

          Show
          pascal oliva added a comment - Thx for the review, here a new patch ( HADOOP-10847 -3.patch) updated accordingly to your comments.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12679188/HADOOP-10847-3.patch
          against trunk revision 2bb327e.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5013//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5013//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12679188/HADOOP-10847-3.patch against trunk revision 2bb327e. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5013//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5013//console This message is automatically generated.
          Hide
          Haohui Mai added a comment -

          +1. I'll commit it shortly.

          Show
          Haohui Mai added a comment - +1. I'll commit it shortly.
          Hide
          Haohui Mai added a comment -

          I've committed the patch to trunk and branch-2. Thanks pascal oliva for the contribution.

          Show
          Haohui Mai added a comment - I've committed the patch to trunk and branch-2. Thanks pascal oliva for the contribution.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6436 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6436/)
          HADOOP-10847. Remove the usage of sun.security.x509.* in testing code. Contributed by Pascal Oliva. (wheat9: rev 1eed1020234b8b5e5444bbc88299bc6689e6b015)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/pom.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6436 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6436/ ) HADOOP-10847 . Remove the usage of sun.security.x509.* in testing code. Contributed by Pascal Oliva. (wheat9: rev 1eed1020234b8b5e5444bbc88299bc6689e6b015) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/pom.xml
          Hide
          Robert Kanter added a comment -

          I was working on a similar patch for HADOOP-11230 (these sun classes actually don't work even in JDK 8), but looks like you beat me to it

          Anyway, this causes tests in other modules that use KeyStoreTestUtil to fail. For example, TestEncryptedShuffle

          Running org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle
          Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.794 sec <<< FAILURE! - in org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle
          encryptedShuffleWithClientCerts(org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle)  Time elapsed: 0.342 sec  <<< ERROR!
          java.lang.NoClassDefFoundError: org/bouncycastle/x509/X509V1CertificateGenerator
          	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
          	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
          	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
          	at org.apache.hadoop.security.ssl.KeyStoreTestUtil.generateCertificate(KeyStoreTestUtil.java:87)
          	at org.apache.hadoop.security.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:242)
          	at org.apache.hadoop.security.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:207)
          	at org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle.encryptedShuffleWithCerts(TestEncryptedShuffle.java:135)
          	at org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle.encryptedShuffleWithClientCerts(TestEncryptedShuffle.java:167)
          
          encryptedShuffleWithoutClientCerts(org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle)  Time elapsed: 0.133 sec  <<< ERROR!
          java.lang.NoClassDefFoundError: org/bouncycastle/x509/X509V1CertificateGenerator
          	at org.apache.hadoop.security.ssl.KeyStoreTestUtil.generateCertificate(KeyStoreTestUtil.java:87)
          	at org.apache.hadoop.security.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:251)
          	at org.apache.hadoop.security.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:207)
          	at org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle.encryptedShuffleWithCerts(TestEncryptedShuffle.java:135)
          	at org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle.encryptedShuffleWithoutClientCerts(TestEncryptedShuffle.java:172)
          

          This is because they're including hadoop-common's test jar, which does not pull in transitive dependencies. You have to explicitly add bouncycastle as a test dependency to each affected module: hadoop-kms, hadoop-hdfs-httpfs, hadoop-hdfs, hadoop-mapreduce-client-jobclient, and hadoop-yarn-server-applicationhistoryservice. I've attached an addendum patch (HADOOP-10847-addendum.patch) based on my HADOOP-11230 patch that fixes this. Haohui Mai, can you take a look? We can either put this in as an addendum here, or repurpose HADOOP-11230 for it.

          Show
          Robert Kanter added a comment - I was working on a similar patch for HADOOP-11230 (these sun classes actually don't work even in JDK 8), but looks like you beat me to it Anyway, this causes tests in other modules that use KeyStoreTestUtil to fail. For example, TestEncryptedShuffle Running org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.794 sec <<< FAILURE! - in org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle encryptedShuffleWithClientCerts(org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle) Time elapsed: 0.342 sec <<< ERROR! java.lang.NoClassDefFoundError: org/bouncycastle/x509/X509V1CertificateGenerator at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at org.apache.hadoop.security.ssl.KeyStoreTestUtil.generateCertificate(KeyStoreTestUtil.java:87) at org.apache.hadoop.security.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:242) at org.apache.hadoop.security.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:207) at org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle.encryptedShuffleWithCerts(TestEncryptedShuffle.java:135) at org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle.encryptedShuffleWithClientCerts(TestEncryptedShuffle.java:167) encryptedShuffleWithoutClientCerts(org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle) Time elapsed: 0.133 sec <<< ERROR! java.lang.NoClassDefFoundError: org/bouncycastle/x509/X509V1CertificateGenerator at org.apache.hadoop.security.ssl.KeyStoreTestUtil.generateCertificate(KeyStoreTestUtil.java:87) at org.apache.hadoop.security.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:251) at org.apache.hadoop.security.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:207) at org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle.encryptedShuffleWithCerts(TestEncryptedShuffle.java:135) at org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle.encryptedShuffleWithoutClientCerts(TestEncryptedShuffle.java:172) This is because they're including hadoop-common's test jar, which does not pull in transitive dependencies. You have to explicitly add bouncycastle as a test dependency to each affected module: hadoop-kms, hadoop-hdfs-httpfs, hadoop-hdfs, hadoop-mapreduce-client-jobclient, and hadoop-yarn-server-applicationhistoryservice. I've attached an addendum patch ( HADOOP-10847 -addendum.patch) based on my HADOOP-11230 patch that fixes this. Haohui Mai , can you take a look? We can either put this in as an addendum here, or repurpose HADOOP-11230 for it.
          Hide
          Haohui Mai added a comment -

          Robert Kanter, the patch looks good to me. Let's repurpose HADOOP-11230 and get the patch in.

          Show
          Haohui Mai added a comment - Robert Kanter , the patch looks good to me. Let's repurpose HADOOP-11230 and get the patch in.
          Hide
          Robert Kanter added a comment -

          Sure. I've updated HADOOP-11230 and uploaded the patch there. Haohui Mai, can you take a look at HADOOP-11230?

          Show
          Robert Kanter added a comment - Sure. I've updated HADOOP-11230 and uploaded the patch there. Haohui Mai , can you take a look at HADOOP-11230 ?
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #734 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/734/)
          HADOOP-10847. Remove the usage of sun.security.x509.* in testing code. Contributed by Pascal Oliva. (wheat9: rev 1eed1020234b8b5e5444bbc88299bc6689e6b015)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          • hadoop-common-project/hadoop-common/pom.xml
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #734 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/734/ ) HADOOP-10847 . Remove the usage of sun.security.x509.* in testing code. Contributed by Pascal Oliva. (wheat9: rev 1eed1020234b8b5e5444bbc88299bc6689e6b015) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-project/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java hadoop-common-project/hadoop-common/pom.xml
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1923 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1923/)
          HADOOP-10847. Remove the usage of sun.security.x509.* in testing code. Contributed by Pascal Oliva. (wheat9: rev 1eed1020234b8b5e5444bbc88299bc6689e6b015)

          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1923 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1923/ ) HADOOP-10847 . Remove the usage of sun.security.x509.* in testing code. Contributed by Pascal Oliva. (wheat9: rev 1eed1020234b8b5e5444bbc88299bc6689e6b015) hadoop-project/pom.xml hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1948 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1948/)
          HADOOP-10847. Remove the usage of sun.security.x509.* in testing code. Contributed by Pascal Oliva. (wheat9: rev 1eed1020234b8b5e5444bbc88299bc6689e6b015)

          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1948 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1948/ ) HADOOP-10847 . Remove the usage of sun.security.x509.* in testing code. Contributed by Pascal Oliva. (wheat9: rev 1eed1020234b8b5e5444bbc88299bc6689e6b015) hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt

            People

            • Assignee:
              pascal oliva
              Reporter:
              Kai Zheng
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development