Robert Kanter added a comment - 17/Feb/16 23:01 I verified the changes by checking the shuffle port after enabling encrypted shuffle. TLSv1.1 before [root@rkanter-z ~]# openssl s_client -connect rkanter-z.vpc.cloudera.com:13562 -tls1_1 CONNECTED(00000003) 139747317544776:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:339: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 5 bytes and written 7 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.1 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1455749132 Timeout : 7200 (sec) Verify return code: 0 (ok) --- TLSv1.1 after [root@rkanter-z jars]# openssl s_client -connect rkanter-z.vpc.cloudera.com:13562 -tls1_1 CONNECTED(00000003) depth=0 O = Hadoop, CN = rkanter-z.vpc.cloudera.com verify error:num=18:self signed certificate verify return:1 depth=0 O = Hadoop, CN = rkanter-z.vpc.cloudera.com verify return:1 --- Certificate chain 0 s:/O=Hadoop/CN=rkanter-z.vpc.cloudera.com i:/O=Hadoop/CN=rkanter-z.vpc.cloudera.com --- Server certificate -----BEGIN CERTIFICATE----- MIIDCzCCAfOgAwIBAgIEGPQogTANBgkqhkiG9w0BAQsFADA2MQ8wDQYDVQQKEwZI YWRvb3AxIzAhBgNVBAMTGnJrYW50ZXItei52cGMuY2xvdWRlcmEuY29tMB4XDTE2 MDIxNzIyMjU1MFoXDTIxMDIxNTIyMjU1MFowNjEPMA0GA1UEChMGSGFkb29wMSMw IQYDVQQDExpya2FudGVyLXoudnBjLmNsb3VkZXJhLmNvbTCCASIwDQYJKoZIhvcN AQEBBQADggEPADCCAQoCggEBAJ6xpEdLHS26AeVmris83Eqm1/roG9mCe2S9j/L9 cAiXrpiy5KsJROqzRwit76fPao4snY65uRcHSedOCUpPfWq7AqWLRoAfayTuYo43 xWVriD9RmLkdMNY4te4gw24rPCfgUQyQcFNuSWZDNT0UTDoye6h9TXDQwwWwN2pv Xp3W9/l4i0jwUKtB6KSrl2tOwPLwBnGRT33V2/S6W6JBCf3hpNwEm1swcnY2UwWr 8X/xGhtXobURe9iS2ZwWcLosloCwslRF/Cn1AdmyotcjjKIwHIgv3QxPpoKjNQm0 lGRCxTMlArX3jLH6JZfSOAKWMQlFbIbEqtk14BH3yyoPlhECAwEAAaMhMB8wHQYD VR0OBBYEFPia3A1galfIVf4hDU2XAk4nl4XxMA0GCSqGSIb3DQEBCwUAA4IBAQBk MQLg4SSsa/Ki3dqFy8aWiadw+nb+2caNvQvgk6PjOPVEWC5NB1kfWzErNF5uC9GP vsmo+NBfeTVLNz1S7+lTZaehC+mYoRpM7HKYOpq1wX+x5pDrMvgNojEo4xk95p/y oAWb/+olAQNWxK95JE2cv+yliWbPyIzAq2DlXmfoyg03wT5/vXT7tra3FwdhL+6U 66IdjfFKRHnkpbNGpca74Sur6UMRMWm4GiDBdK6PcQi+9yfW1ZagprVXUgTkFEvD Zb+YUExxT75jF8EgWW2RJCjGeXviZb/OkhpK4K0W28EP/MT3vtWTvDVbPTe6qEZQ kThiicNyhyVW44fv7Mv0 -----END CERTIFICATE----- subject=/O=Hadoop/CN=rkanter-z.vpc.cloudera.com issuer=/O=Hadoop/CN=rkanter-z.vpc.cloudera.com --- No client certificate CA names sent Server Temp Key: ECDH, secp521r1, 521 bits --- SSL handshake has read 1357 bytes and written 373 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.1 Cipher : ECDHE-RSA-AES256-SHA Session-ID: 56C4F9645E5BD9F826F4A77B1382BF4E4EFE93EDF81D030B27A45937A5E9447F Session-ID-ctx: Master-Key: 2DE6931DC740F4A3430A34FA28333BEAD19EAEC64F980FF598589A33D47B3620F99624901F2F5CF454FEDCF394A02C21 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1455749476 Timeout : 7200 (sec) Verify return code: 18 (self signed certificate) --- TLSv1.2 before [root@rkanter-z ~]# openssl s_client -connect rkanter-z.vpc.cloudera.com:13562 -tls1_2 CONNECTED(00000003) 140717584258888:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:339: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 5 bytes and written 7 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1455749158 Timeout : 7200 (sec) Verify return code: 0 (ok) --- TLSv1.2 after [root@rkanter-z jars]# openssl s_client -connect rkanter-z.vpc.cloudera.com:13562 -tls1_2 CONNECTED(00000003) depth=0 O = Hadoop, CN = rkanter-z.vpc.cloudera.com verify error:num=18:self signed certificate verify return:1 depth=0 O = Hadoop, CN = rkanter-z.vpc.cloudera.com verify return:1 --- Certificate chain 0 s:/O=Hadoop/CN=rkanter-z.vpc.cloudera.com i:/O=Hadoop/CN=rkanter-z.vpc.cloudera.com --- Server certificate -----BEGIN CERTIFICATE----- MIIDCzCCAfOgAwIBAgIEGPQogTANBgkqhkiG9w0BAQsFADA2MQ8wDQYDVQQKEwZI YWRvb3AxIzAhBgNVBAMTGnJrYW50ZXItei52cGMuY2xvdWRlcmEuY29tMB4XDTE2 MDIxNzIyMjU1MFoXDTIxMDIxNTIyMjU1MFowNjEPMA0GA1UEChMGSGFkb29wMSMw IQYDVQQDExpya2FudGVyLXoudnBjLmNsb3VkZXJhLmNvbTCCASIwDQYJKoZIhvcN AQEBBQADggEPADCCAQoCggEBAJ6xpEdLHS26AeVmris83Eqm1/roG9mCe2S9j/L9 cAiXrpiy5KsJROqzRwit76fPao4snY65uRcHSedOCUpPfWq7AqWLRoAfayTuYo43 xWVriD9RmLkdMNY4te4gw24rPCfgUQyQcFNuSWZDNT0UTDoye6h9TXDQwwWwN2pv Xp3W9/l4i0jwUKtB6KSrl2tOwPLwBnGRT33V2/S6W6JBCf3hpNwEm1swcnY2UwWr 8X/xGhtXobURe9iS2ZwWcLosloCwslRF/Cn1AdmyotcjjKIwHIgv3QxPpoKjNQm0 lGRCxTMlArX3jLH6JZfSOAKWMQlFbIbEqtk14BH3yyoPlhECAwEAAaMhMB8wHQYD VR0OBBYEFPia3A1galfIVf4hDU2XAk4nl4XxMA0GCSqGSIb3DQEBCwUAA4IBAQBk MQLg4SSsa/Ki3dqFy8aWiadw+nb+2caNvQvgk6PjOPVEWC5NB1kfWzErNF5uC9GP vsmo+NBfeTVLNz1S7+lTZaehC+mYoRpM7HKYOpq1wX+x5pDrMvgNojEo4xk95p/y oAWb/+olAQNWxK95JE2cv+yliWbPyIzAq2DlXmfoyg03wT5/vXT7tra3FwdhL+6U 66IdjfFKRHnkpbNGpca74Sur6UMRMWm4GiDBdK6PcQi+9yfW1ZagprVXUgTkFEvD Zb+YUExxT75jF8EgWW2RJCjGeXviZb/OkhpK4K0W28EP/MT3vtWTvDVbPTe6qEZQ kThiicNyhyVW44fv7Mv0 -----END CERTIFICATE----- subject=/O=Hadoop/CN=rkanter-z.vpc.cloudera.com issuer=/O=Hadoop/CN=rkanter-z.vpc.cloudera.com --- No client certificate CA names sent Server Temp Key: ECDH, secp521r1, 521 bits --- SSL handshake has read 1391 bytes and written 499 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-SHA384 Session-ID: 56C4F9800BF1838DC6196C712E31DEECBC1AF7411BCA9BCDDE0E2BEE5B7DC41C Session-ID-ctx: Master-Key: 1C6E5AC1951B8FDDFC39C17A152E212F957007D301EF26334EBA7DCB3F1AE0C8AF22B72ABCB4BFD06BB4A59F23AD7841 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1455749504 Timeout : 7200 (sec) Verify return code: 18 (self signed certificate) ---

Wei-Chiu Chuang added a comment - 17/Feb/16 23:50 Hi Robert Kanter thanks for working on this. Can you also update the default value of this property in EncryptedShuffle.md?

Hadoop QA added a comment - 18/Feb/16 00:17 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 12s trunk passed +1 compile 6m 39s trunk passed with JDK v1.8.0_72 +1 compile 7m 27s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 48s trunk passed +1 javadoc 1m 0s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 11s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 44s the patch passed +1 compile 6m 48s the patch passed with JDK v1.8.0_72 +1 javac 6m 48s the patch passed +1 compile 7m 19s the patch passed with JDK v1.7.0_95 +1 javac 7m 19s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 1m 9s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 0s the patch passed +1 javadoc 0m 58s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 9s the patch passed with JDK v1.7.0_95 +1 unit 8m 44s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 8m 35s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 66m 50s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788331/HADOOP-12817.001.patch JIRA Issue HADOOP-12817 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux f5d1b43665b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0fb14aa Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8652/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8652/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Robert Kanter added a comment - 18/Feb/16 00:29 Good catch. Updated the docs in the 002 patch.

Hadoop QA added a comment - 18/Feb/16 02:16 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 36s Maven dependency ordering for branch +1 mvninstall 7m 18s trunk passed +1 compile 9m 3s trunk passed with JDK v1.8.0_72 +1 compile 8m 7s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 14s trunk passed +1 mvnsite 1m 53s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 3m 11s trunk passed +1 javadoc 1m 55s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 52s trunk passed with JDK v1.7.0_95 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 10m 44s the patch passed with JDK v1.8.0_72 +1 javac 10m 44s the patch passed +1 compile 8m 56s the patch passed with JDK v1.7.0_95 +1 javac 8m 56s the patch passed +1 checkstyle 1m 13s the patch passed +1 mvnsite 1m 57s the patch passed +1 mvneclipse 0m 31s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 46s the patch passed +1 javadoc 1m 53s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_95 +1 unit 10m 22s hadoop-common in the patch passed with JDK v1.8.0_72. -1 unit 2m 15s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_72. +1 unit 9m 33s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 2m 21s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 94m 54s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.lib.input.TestFileInputFormat   hadoop.mapred.TestFileInputFormat JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.lib.input.TestFileInputFormat   hadoop.mapred.TestFileInputFormat Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788342/HADOOP-12817.002.patch JIRA Issue HADOOP-12817 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux d6c0ba7f1274 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0fb14aa Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8653/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8653/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8653/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8653/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8653/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8653/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Wei-Chiu Chuang added a comment - 18/Feb/16 02:55 002 patch looks good to me. +1 (non-binding)

Karthik Kambatla added a comment - 18/Feb/16 03:09 LGTM. +1.

Robert Kanter added a comment - 18/Feb/16 19:14 Thanks for the reviews Karthik Kambatla and Wei-Chiu Chuang . Committed to trunk and branch-2!

Hudson added a comment - 18/Feb/16 19:42 FAILURE: Integrated in Hadoop-trunk-Commit #9323 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9323/ ) HADOOP-12817 . Enable TLS v1.1 and 1.2 (rkanter) (rkanter: rev a365a3941cf96a31c289cb22678a602738880f74) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLFactory.java hadoop-common-project/hadoop-common/CHANGES.txt