Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14597

Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha4
    • Fix Version/s: 3.0.0-beta1
    • Component/s: None
    • Labels:
      None
    • Environment:

      openssl-1.1.0

      Description

      Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails with this error

      [WARNING] /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c: In function ‘check_update_max_output_len’:
      [WARNING] /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14: error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct evp_cipher_ctx_st}’
      [WARNING]    if (context->flags & EVP_CIPH_NO_PADDING) {
      [WARNING]               ^~
      

      https://github.com/openssl/openssl/issues/962 mattcaswell says

      One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2 version is that many types have been made opaque, i.e. applications are no longer allowed to look inside the internals of the structures

      1. HADOOP-14597.04.patch
        7 kB
        Ravi Prakash
      2. HADOOP-14597.03.patch
        3 kB
        Ravi Prakash
      3. HADOOP-14597.02.patch
        3 kB
        Ravi Prakash
      4. HADOOP-14597.01.patch
        3 kB
        Ravi Prakash
      5. HADOOP-14597.00.patch
        2 kB
        Ravi Prakash

        Activity

        Hide
        raviprak Ravi Prakash added a comment -

        I'm not sure this is the right patch, but it lets the compilation continue. I'm using functions defined here https://github.com/openssl/openssl/blob/master/crypto/evp/evp_lib.c#L196. I don't know what the intent of making the internals opaque is, if I can get to the same flags using this method. This leads me to believe this patch is probably not what the openssl devs intended. Anyone know anything about this?

        Show
        raviprak Ravi Prakash added a comment - I'm not sure this is the right patch, but it lets the compilation continue. I'm using functions defined here https://github.com/openssl/openssl/blob/master/crypto/evp/evp_lib.c#L196 . I don't know what the intent of making the internals opaque is, if I can get to the same flags using this method. This leads me to believe this patch is probably not what the openssl devs intended. Anyone know anything about this?
        Hide
        raviprak Ravi Prakash added a comment -

        This patch lets hadoop-pipes compile too

        Show
        raviprak Ravi Prakash added a comment - This patch lets hadoop-pipes compile too
        Hide
        raviprak Ravi Prakash added a comment - - edited

        HADOOP-14597.00.patch and HADOOP-14597.01.patch are bad. They are passing a EVP_CIPHER_CTX to EVP_CIPHER_flags which actually expects EVP_CIPHER . These are two different structs (none inheriting the other).

        I am now using the proper method and I'm guessing the openssl devs intended for us to use it this way.

        Can someone please review and commit?

        Show
        raviprak Ravi Prakash added a comment - - edited HADOOP-14597 .00.patch and HADOOP-14597 .01.patch are bad. They are passing a EVP_CIPHER_CTX to EVP_CIPHER_flags which actually expects EVP_CIPHER . These are two different structs (none inheriting the other). I am now using the proper method and I'm guessing the openssl devs intended for us to use it this way. Can someone please review and commit?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 33s Maven dependency ordering for branch
        +1 mvninstall 15m 48s trunk passed
        +1 compile 14m 6s trunk passed
        +1 mvnsite 1m 25s trunk passed
        0 mvndep 0m 42s Maven dependency ordering for patch
        +1 mvninstall 0m 47s the patch passed
        -1 compile 9m 25s root in the patch failed.
        -1 cc 9m 25s root in the patch failed.
        -1 javac 9m 25s root in the patch failed.
        +1 mvnsite 1m 19s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 unit 8m 2s hadoop-common in the patch passed.
        -1 unit 0m 17s hadoop-pipes in the patch failed.
        +1 asflicense 0m 35s The patch does not generate ASF License warnings.
        54m 6s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14597
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874774/HADOOP-14597.02.patch
        Optional Tests asflicense compile cc mvnsite javac unit
        uname Linux 9d3a0253e453 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 63ce159
        Default Java 1.8.0_131
        compile https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/artifact/patchprocess/patch-compile-root.txt
        cc https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/artifact/patchprocess/patch-compile-root.txt
        javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/artifact/patchprocess/patch-compile-root.txt
        whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-pipes.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-pipes U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 33s Maven dependency ordering for branch +1 mvninstall 15m 48s trunk passed +1 compile 14m 6s trunk passed +1 mvnsite 1m 25s trunk passed 0 mvndep 0m 42s Maven dependency ordering for patch +1 mvninstall 0m 47s the patch passed -1 compile 9m 25s root in the patch failed. -1 cc 9m 25s root in the patch failed. -1 javac 9m 25s root in the patch failed. +1 mvnsite 1m 19s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 unit 8m 2s hadoop-common in the patch passed. -1 unit 0m 17s hadoop-pipes in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 54m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14597 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874774/HADOOP-14597.02.patch Optional Tests asflicense compile cc mvnsite javac unit uname Linux 9d3a0253e453 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 63ce159 Default Java 1.8.0_131 compile https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/artifact/patchprocess/patch-compile-root.txt cc https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/artifact/patchprocess/patch-compile-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-pipes.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-pipes U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12641/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        aw Allen Wittenauer added a comment -

        cc root in the patch failed.

        That's sort of what I was expecting based upon https://wiki.openssl.org/index.php/1.1_API_Changes#Adding_forward-compatible_code_to_older_versions .

        It looks like we need to #ifdef old vs. new routines here.

        Show
        aw Allen Wittenauer added a comment - cc root in the patch failed. That's sort of what I was expecting based upon https://wiki.openssl.org/index.php/1.1_API_Changes#Adding_forward-compatible_code_to_older_versions . It looks like we need to #ifdef old vs. new routines here.
        Hide
        raviprak Ravi Prakash added a comment -

        Thanks Allen! I tested this one with OpenSSL-1.0.2. I had to pull in MAPREDUCE-6536 too.

        FWIW I built OpenSSL-1.0.2 from source using

         ./config --prefix=/some/directory; make; make install 

        To let common build using -Dopenssl.prefix=/some/directory, I had to

        diff --git a/hadoop-common-project/hadoop-common/src/CMakeLists.txt b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
        index 10b0f23ddf..01aadd546a 100644
        --- a/hadoop-common-project/hadoop-common/src/CMakeLists.txt
        +++ b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
        @@ -176,6 +176,7 @@ if(${CMAKE_SYSTEM_NAME} MATCHES "Windows")
             SET(OPENSSL_NAME "eay32")
         endif()
         message("CUSTOM_OPENSSL_PREFIX = ${CUSTOM_OPENSSL_PREFIX}")
        +SET(CMAKE_FIND_LIBRARY_SUFFIXES ".so" ".a")
         find_library(OPENSSL_LIBRARY
             NAMES ${OPENSSL_NAME}
             PATHS ${CUSTOM_OPENSSL_PREFIX} ${CUSTOM_OPENSSL_PREFIX}/lib
        

        Seems like we are playing games to be better at detecting OpenSSL than Cmake . I know I'm way off on a tangent now Hahaha

        Show
        raviprak Ravi Prakash added a comment - Thanks Allen! I tested this one with OpenSSL-1.0.2. I had to pull in MAPREDUCE-6536 too. FWIW I built OpenSSL-1.0.2 from source using ./config --prefix=/some/directory; make; make install To let common build using -Dopenssl.prefix=/some/directory , I had to diff --git a/hadoop-common-project/hadoop-common/src/CMakeLists.txt b/hadoop-common-project/hadoop-common/src/CMakeLists.txt index 10b0f23ddf..01aadd546a 100644 --- a/hadoop-common-project/hadoop-common/src/CMakeLists.txt +++ b/hadoop-common-project/hadoop-common/src/CMakeLists.txt @@ -176,6 +176,7 @@ if (${CMAKE_SYSTEM_NAME} MATCHES "Windows" ) SET(OPENSSL_NAME "eay32" ) endif() message( "CUSTOM_OPENSSL_PREFIX = ${CUSTOM_OPENSSL_PREFIX}" ) +SET(CMAKE_FIND_LIBRARY_SUFFIXES ".so" ".a" ) find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME} PATHS ${CUSTOM_OPENSSL_PREFIX} ${CUSTOM_OPENSSL_PREFIX}/lib Seems like we are playing games to be better at detecting OpenSSL than Cmake . I know I'm way off on a tangent now Hahaha
        Hide
        aw Allen Wittenauer added a comment -

        Yeah, the way most of this stuff is currently written it flat out rejects any static libraries. I don't know why it was written that way. I've got a 1.0.2 and 1.1.0 built on OS X with both static and dynamic. I'll try the patch out and see how it goes.

        Thanks!

        Show
        aw Allen Wittenauer added a comment - Yeah, the way most of this stuff is currently written it flat out rejects any static libraries. I don't know why it was written that way. I've got a 1.0.2 and 1.1.0 built on OS X with both static and dynamic. I'll try the patch out and see how it goes. Thanks!
        Hide
        aw Allen Wittenauer added a comment -

        OK, the pipes code is working for me just fine. The common code is not; compilation failures regardless of the library version. Let me dig into a bit. I think it might be an issue with how the command line is constructed on OS X.... again related to the funky things we do in cmake.

        Show
        aw Allen Wittenauer added a comment - OK, the pipes code is working for me just fine. The common code is not; compilation failures regardless of the library version. Let me dig into a bit. I think it might be an issue with how the command line is constructed on OS X.... again related to the funky things we do in cmake.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 1m 36s Maven dependency ordering for branch
        +1 mvninstall 15m 9s trunk passed
        +1 compile 13m 58s trunk passed
        +1 mvnsite 1m 28s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 0m 52s the patch passed
        +1 compile 11m 45s the patch passed
        -1 cc 11m 45s root generated 2 new + 8 unchanged - 0 fixed = 10 total (was 8)
        +1 javac 11m 45s the patch passed
        +1 mvnsite 1m 29s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 unit 8m 12s hadoop-common in the patch passed.
        +1 unit 0m 20s hadoop-pipes in the patch passed.
        +1 asflicense 0m 35s The patch does not generate ASF License warnings.
        56m 27s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14597
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874986/HADOOP-14597.03.patch
        Optional Tests asflicense compile cc mvnsite javac unit
        uname Linux e2c89293eb5f 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c1edca1
        Default Java 1.8.0_131
        cc https://builds.apache.org/job/PreCommit-HADOOP-Build/12657/artifact/patchprocess/diff-compile-cc-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12657/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-pipes U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12657/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 36s Maven dependency ordering for branch +1 mvninstall 15m 9s trunk passed +1 compile 13m 58s trunk passed +1 mvnsite 1m 28s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 0m 52s the patch passed +1 compile 11m 45s the patch passed -1 cc 11m 45s root generated 2 new + 8 unchanged - 0 fixed = 10 total (was 8) +1 javac 11m 45s the patch passed +1 mvnsite 1m 29s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 unit 8m 12s hadoop-common in the patch passed. +1 unit 0m 20s hadoop-pipes in the patch passed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 56m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14597 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874986/HADOOP-14597.03.patch Optional Tests asflicense compile cc mvnsite javac unit uname Linux e2c89293eb5f 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c1edca1 Default Java 1.8.0_131 cc https://builds.apache.org/job/PreCommit-HADOOP-Build/12657/artifact/patchprocess/diff-compile-cc-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12657/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-pipes U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12657/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        aw Allen Wittenauer added a comment -

        I think the problem might be that we're in JNI-land. Which means they need to be dynamically loaded. If you look at the top of the file, you'll see a bunch of dlsym definitions. We need to do the same thing with these routines, and then call them with the dlsym_ name.

        One concern I've got is that it doesn't look like _encrypting is meant to be a public API.

        Show
        aw Allen Wittenauer added a comment - I think the problem might be that we're in JNI-land. Which means they need to be dynamically loaded. If you look at the top of the file, you'll see a bunch of dlsym definitions. We need to do the same thing with these routines, and then call them with the dlsym_ name. One concern I've got is that it doesn't look like _encrypting is meant to be a public API.
        Hide
        raviprak Ravi Prakash added a comment -

        Aah! Good point. Thank you Allen! Done.

        Are you aware of any tests which I can run to check functionality? I don't see anything in the test directory for native encryption.

        Good point about EVP_CIPHER_CTX_encrypting. The function doesn't exist in OpenSSL-1.0.2 . I've put an ugly ifdef for that too now. I'm not sure what else I could do now, except dilute the semantics (really its just a one byte difference on checking buffer bounds depending on whether its encrypting or decrypting). Or we could set minimum version of OpenSSL to 1.1.0 which I'm sure will make a lot of people unhappy.

        Yi Liu Could you also please review this?

        Show
        raviprak Ravi Prakash added a comment - Aah! Good point. Thank you Allen! Done. Are you aware of any tests which I can run to check functionality? I don't see anything in the test directory for native encryption. Good point about EVP_CIPHER_CTX_encrypting . The function doesn't exist in OpenSSL-1.0.2 . I've put an ugly ifdef for that too now. I'm not sure what else I could do now, except dilute the semantics (really its just a one byte difference on checking buffer bounds depending on whether its encrypting or decrypting). Or we could set minimum version of OpenSSL to 1.1.0 which I'm sure will make a lot of people unhappy. Yi Liu Could you also please review this?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 22s Maven dependency ordering for branch
        +1 mvninstall 13m 48s trunk passed
        +1 compile 14m 17s trunk passed
        +1 mvnsite 1m 24s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 0m 49s the patch passed
        +1 compile 10m 41s the patch passed
        +1 cc 10m 41s the patch passed
        +1 javac 10m 41s the patch passed
        +1 mvnsite 1m 22s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 unit 7m 52s hadoop-common in the patch failed.
        +1 unit 0m 20s hadoop-pipes in the patch passed.
        +1 asflicense 0m 34s The patch does not generate ASF License warnings.
        52m 43s



        Reason Tests
        Failed junit tests hadoop.security.TestRaceWhenRelogin
          hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14597
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875105/HADOOP-14597.04.patch
        Optional Tests asflicense compile cc mvnsite javac unit
        uname Linux 5f6baed60a56 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5a75f73
        Default Java 1.8.0_131
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12662/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12662/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-pipes U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12662/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 22s Maven dependency ordering for branch +1 mvninstall 13m 48s trunk passed +1 compile 14m 17s trunk passed +1 mvnsite 1m 24s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 0m 49s the patch passed +1 compile 10m 41s the patch passed +1 cc 10m 41s the patch passed +1 javac 10m 41s the patch passed +1 mvnsite 1m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 unit 7m 52s hadoop-common in the patch failed. +1 unit 0m 20s hadoop-pipes in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 52m 43s Reason Tests Failed junit tests hadoop.security.TestRaceWhenRelogin   hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14597 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875105/HADOOP-14597.04.patch Optional Tests asflicense compile cc mvnsite javac unit uname Linux 5f6baed60a56 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5a75f73 Default Java 1.8.0_131 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12662/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12662/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-pipes U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12662/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        aw Allen Wittenauer added a comment -

        I think Yi is MIA. It'd be good to get some of the EC folks involved, since I think this is in their code path: Pinging Zhe Zhang, Kai Sasaki, Andrew Wang to help look this over.

        Show
        aw Allen Wittenauer added a comment - I think Yi is MIA. It'd be good to get some of the EC folks involved, since I think this is in their code path: Pinging Zhe Zhang , Kai Sasaki , Andrew Wang to help look this over.
        Hide
        aw Allen Wittenauer added a comment -

        I'm +1, but with some caveats that the unit tests in this part of the world are a bit broken. I was using:

        mvn test -Dtest='*ssl*' -Pnative -Drequire.openssl -Drequire.test.libhadoop
        

        to test on both OS X and Linux and with both OpenSSL 1.0x and 1.1.x. It revealed that if I pass in -Dopenssl.prefix=foo (to point to a different OpenSSL), the tests always fail. This appears to be more of a build problem than anything else. If the standard OpenSSL library is used (which in my case was 1.0x on Linux), it works fine.

        I think the patch is good enough to go in to unblock OpenSSL 1.1 users. We should probably do a follow-up to figure out why loading the libhadoop.so and openssl libraries during unit testing is failing.

        Show
        aw Allen Wittenauer added a comment - I'm +1, but with some caveats that the unit tests in this part of the world are a bit broken. I was using: mvn test -Dtest='*ssl*' -Pnative -Drequire.openssl -Drequire.test.libhadoop to test on both OS X and Linux and with both OpenSSL 1.0x and 1.1.x. It revealed that if I pass in -Dopenssl.prefix=foo (to point to a different OpenSSL), the tests always fail. This appears to be more of a build problem than anything else. If the standard OpenSSL library is used (which in my case was 1.0x on Linux), it works fine. I think the patch is good enough to go in to unblock OpenSSL 1.1 users. We should probably do a follow-up to figure out why loading the libhadoop.so and openssl libraries during unit testing is failing.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
              trunk Compile Tests
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 13m 12s trunk passed
        +1 compile 13m 47s trunk passed
        +1 mvnsite 1m 45s trunk passed
              Patch Compile Tests
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 0m 46s the patch passed
        +1 compile 10m 28s the patch passed
        +1 cc 10m 28s the patch passed
        +1 javac 10m 28s the patch passed
        +1 mvnsite 1m 43s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
              Other Tests
        -1 unit 8m 0s hadoop-common in the patch failed.
        +1 unit 0m 20s hadoop-pipes in the patch passed.
        +1 asflicense 0m 33s The patch does not generate ASF License warnings.
        51m 55s



        Reason Tests
        Failed junit tests hadoop.security.TestKDiag



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14597
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875105/HADOOP-14597.04.patch
        Optional Tests asflicense compile cc mvnsite javac unit
        uname Linux 1bf1c034fbb7 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f5f14a2
        Default Java 1.8.0_131
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12813/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12813/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-pipes U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12813/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 13m 12s trunk passed +1 compile 13m 47s trunk passed +1 mvnsite 1m 45s trunk passed       Patch Compile Tests 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 0m 46s the patch passed +1 compile 10m 28s the patch passed +1 cc 10m 28s the patch passed +1 javac 10m 28s the patch passed +1 mvnsite 1m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues.       Other Tests -1 unit 8m 0s hadoop-common in the patch failed. +1 unit 0m 20s hadoop-pipes in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 51m 55s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14597 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875105/HADOOP-14597.04.patch Optional Tests asflicense compile cc mvnsite javac unit uname Linux 1bf1c034fbb7 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f5f14a2 Default Java 1.8.0_131 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12813/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12813/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-pipes U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12813/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        raviprak Ravi Prakash added a comment -

        Thank you for the review and comments Allen. Committing shortly.

        Show
        raviprak Ravi Prakash added a comment - Thank you for the review and comments Allen. Committing shortly.
        Hide
        raviprak Ravi Prakash added a comment -

        I've filed https://issues.apache.org/jira/browse/HADOOP-14682 to document the issue in the Cmake files

        Show
        raviprak Ravi Prakash added a comment - I've filed https://issues.apache.org/jira/browse/HADOOP-14682 to document the issue in the Cmake files
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12051 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12051/)
        HADOOP-14597. Native compilation broken with OpenSSL-1.1.0. Contributed (raviprak: rev 94ca52ae9ec0ae04854d726bf2ac1bc457b96a9c)

        • (edit) hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c
        • (edit) hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12051 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12051/ ) HADOOP-14597 . Native compilation broken with OpenSSL-1.1.0. Contributed (raviprak: rev 94ca52ae9ec0ae04854d726bf2ac1bc457b96a9c) (edit) hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c (edit) hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc

          People

          • Assignee:
            raviprak Ravi Prakash
            Reporter:
            raviprak Ravi Prakash
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development