Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-3308

Openssl fails to load in hadoop checknative -a on Debian 9

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0, 1.5.0
    • Fix Version/s: 1.5.0, 1.4.1
    • Component/s: None
    • Labels:
      None

      Description

      Hi!

       

      I am trying to switch from CDH 5.16 to BigTop 1.4. I am currently trying to upgrade on a test cluster with Debian 9.

      We use Spark 2.4.4 with RPC encryption+auth enabled, relying on openssl's libcrypto capabilities. I noticed that with BigTop 1.4 hadoop checknative reports an error for openssl:

      openssl: false EVP_CIPHER_CTX_encrypting
      

      On Debian 9 we have both openssl 1.10 and 1.0.2 deployed, so we configured libcrypto in this way:

      0 lrwxrwxrwx 1 root root 44 Feb 17 08:59 /usr/lib/x86_64-linux-gnu/libcrypto.so -> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      

      We had to add this symlink on CDH since a versioned libcrypto.so is not picked up, and libssl-dev on Debian 9 creates a symlink to libcrypto1.1.0, not 1.0.2 (that was failing).

      I see that EVP_CIPHER_CTX_encrypting seems guarded by a ifdef after https://issues.apache.org/jira/browse/BIGTOP-2932, so I guess that BigTop on Debian 9 builds against libssl1.1.0? (Still a bit ignorant, don't know how to check how things are build )

      I tried to move the symlink to something like:

      0 lrwxrwxrwx 1 root root 42 Feb 17 10:31 /usr/lib/x86_64-linux-gnu/libcrypto.so -> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      

      And now checknative returns a different error:

      openssl: false EVP_CIPHER_CTX_cleanup
      

      The error in my opinion makes sense, since EVP_CIPHER_CTX_cleanup seems deprecated from openssl 1.1.0 onward. From the Hadoop mailing list I found this interesting thread, that shows the same problem:

      http://mail-archives.us.apache.org/mod_mbox/hadoop-user/201910.mbox/%3cCALh-6sAug29Ua2iX+aZnR_TjzjQpwVBcVV_macQDxNYYgoOLzA@mail.gmail.com%3e

      The main issue is that the reporter mentions libssl1.1.1, not 1.1.0, that should be working with the aforementioned patch. Looking at the code I can see that EVP_CIPHER_CTX_cleanup is not guarded by any ifdef (in https://github.com/apache/hadoop/blob/branch-2.8.5/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c), so the error kinda makes sense to me.

       

      Am I missing something obvious or is there a bug somewhere between Hadoop and BigTop?

       

      Thanks in advance!

       

        Attachments

        1. patch3-pom.diff
          0.4 kB
          Olaf Flebbe

          Issue Links

            Activity

              People

              • Assignee:
                elukey Luca Toscano
                Reporter:
                elukey Luca Toscano
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m