Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9481

Broken conditional logic with HADOOP_SNAPPY_LIBRARY

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 2.1.0-beta, 3.0.0-alpha1
    • 2.1.0-beta
    • None
    • None
    • Reviewed

    Description

      The problem is a regression introduced by recent fix https://issues.apache.org/jira/browse/HADOOP-8562 .
      That fix makes some improvements for Windows platform, but breaks native code work on Unix.
      Namely, let's see the diff HADOOP-8562 of the file hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c :

      --- hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
      +++ hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
      @@ -16,12 +16,18 @@
        * limitations under the License.
        */
      
      -#include <dlfcn.h>
      +
      +#if defined HADOOP_SNAPPY_LIBRARY
      +
       #include <stdio.h>
       #include <stdlib.h>
       #include <string.h>
      
      +#ifdef UNIX
      +#include <dlfcn.h>
       #include "config.h"
      +#endif // UNIX
      +
       #include "org_apache_hadoop_io_compress_snappy.h"
       #include "org_apache_hadoop_io_compress_snappy_SnappyCompressor.h"
      
      @@ -81,7 +87,7 @@ JNIEXPORT jint JNICALL Java_org_apache_hadoop_io_compress_snappy_SnappyCompresso
         UNLOCK_CLASS(env, clazz, "SnappyCompressor");
      
         if (uncompressed_bytes == 0) {
      -    return 0;
      +    return (jint)0;
         }
      
         // Get the output direct buffer
      @@ -90,7 +96,7 @@ JNIEXPORT jint JNICALL Java_org_apache_hadoop_io_compress_snappy_SnappyCompresso
         UNLOCK_CLASS(env, clazz, "SnappyCompressor");
      
         if (compressed_bytes == 0) {
      -    return 0;
      +    return (jint)0;
         }
      
         /* size_t should always be 4 bytes or larger. */
      @@ -109,3 +115,5 @@ JNIEXPORT jint JNICALL Java_org_apache_hadoop_io_compress_snappy_SnappyCompresso
         (*env)->SetIntField(env, thisj, SnappyCompressor_uncompressedDirectBufLen, 0);
         return (jint)buf_len;
       }
      +
      +#endif //define HADOOP_SNAPPY_LIBRARY
      

      Here we see that all the class implementation got enclosed into "if defined HADOOP_SNAPPY_LIBRARY" directive, and the point is that "HADOOP_SNAPPY_LIBRARY" is not defined.
      This causes the class implementation to be effectively empty, what, in turn, causes the UnsatisfiedLinkError to be thrown in the runtime upon any attempt to invoke the native methods implemented there.
      The actual intention of the authors of HADOOP-8562 was (as we suppose) to invoke "include config.h", where "HADOOP_SNAPPY_LIBRARY" is defined. But currently it is not included because it resides inside "if defined HADOOP_SNAPPY_LIBRARY" block.

      Similar situation with "ifdef UNIX", because UNIX or WINDOWS variables are defined in "org_apache_hadoop.h", which is indirectly included through "include "org_apache_hadoop_io_compress_snappy.h"", and in the current code this is done after code "ifdef UNIX", so in the current code the block "ifdef UNIX" is not executed on UNIX.

      The suggested patch fixes the described problems by reordering the "include" and "if" preprocessor directives accordingly, bringing the methods of class org.apache.hadoop.io.compress.snappy.SnappyCompressor back to work again.

      Of course, Snappy native libraries must be installed to build and invoke snappy native methods.

      (Note: there was a mistype in commit message: 8952 written in place of 8562:
      HADOOP-8952. Enhancements to support Hadoop on Windows Server and Windows Azure environments. Contributed by Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha, Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya Bharathi Nimmagadda.
      git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1453486 13f79535-47bb-0310-9956-ffa450edef68
      )

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vbondarev Vadim Bondarev Assign to me
            vbondarev Vadim Bondarev
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment