Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2841 Task level native optimization
  3. MAPREDUCE-5976

native-task should not fail to build if snappy is missing

        Details

        • Type: Sub-task
        • Status: Resolved
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: None
        • Component/s: task
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          Other native parts of Hadoop will automatically disable snappy support if snappy is not present and -Drequire.snappy is not passed. native-task should do the same. (right now, it fails to build if snappy is missing)

          1. Text File
            mapreduce-5976.txt
            9 kB
            Manu Zhang
          2. Text File
            mapreduce-5976-v2.txt
            34 kB
            Manu Zhang
          3. Text File
            mapreduce-5976-v3.txt
            36 kB
            Manu Zhang
          4. Text File
            mapreduce-5976-v4.txt
            35 kB
            Manu Zhang

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            clockfly Sean Zhong added a comment -

            Thanks, Manu,

            The patch looks good. +1

            Show
            clockfly Sean Zhong added a comment - Thanks, Manu, The patch looks good. +1
            Hide
            Permalink
            tlipcon Todd Lipcon added a comment -

            Do we also need some check in the java side that knows whether the Native code has snappy support built in?

            Show
            tlipcon Todd Lipcon added a comment - Do we also need some check in the java side that knows whether the Native code has snappy support built in?
            Hide
            Permalink
            clockfly Sean Zhong added a comment -

            Thanks, Manu. Looks good, +1

            Changes of the new patch:
            1. use system provided snappy header files, remove builtin snappy header
            2. Java side delegate the codec check to a native function, NativeRuntime.supportCompressionCodec(codecName : String)

            Show
            clockfly Sean Zhong added a comment - Thanks, Manu. Looks good, +1 Changes of the new patch: 1. use system provided snappy header files, remove builtin snappy header 2. Java side delegate the codec check to a native function, NativeRuntime.supportCompressionCodec(codecName : String)
            Hide
            Permalink
            mauzhang Manu Zhang added a comment -

            patch updated. Changes include

            1. checks for all compression codecs in the java side whether they have native support built in.
            2. remove snappy header files and include system snappy library in native code

            Show
            mauzhang Manu Zhang added a comment - patch updated. Changes include 1. checks for all compression codecs in the java side whether they have native support built in. 2. remove snappy header files and include system snappy library in native code
            Hide
            Permalink
            clockfly Sean Zhong added a comment -

            Todd Lipcon, can you take a look at the new patch?

            Show
            clockfly Sean Zhong added a comment - Todd Lipcon , can you take a look at the new patch?
            Hide
            Permalink
            tlipcon Todd Lipcon added a comment -

            Sorry for the delay here. The patch mostly looks good. Just a couple of nits:

            +      if (job.getBoolean(MRJobConfig.MAP_OUTPUT_COMPRESS, false) == true) {

            style: just use if (job.getBoolean(...)) instead of comparing {{ == true }}.

            +        String codec = job.get(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC);
+        if (NativeRuntime.supportCompressionCodec(codec.getBytes(Charsets.UTF_8)) == false) {

            Same – just use !NativeRuntime.supportCompressionCodec(...) instead of comparing {{ == false}}

            Also, English nit: should be named supportsCompressionCodec (note the 's')

            +          String message = "Native output collector don't support compression codec " + codec;

            Nit: should be "doesn't support" instead of "don't support"

            +   *
+   * @param codecName
+   * @return

            Remove these empty JavaDoc lines, since the main description of the function is sufficient to understand it.

            Show
            tlipcon Todd Lipcon added a comment - Sorry for the delay here. The patch mostly looks good. Just a couple of nits: + if (job.getBoolean(MRJobConfig.MAP_OUTPUT_COMPRESS, false ) == true ) { style: just use if (job.getBoolean(...)) instead of comparing {{ == true }}. + String codec = job.get(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC); + if (NativeRuntime.supportCompressionCodec(codec.getBytes(Charsets.UTF_8)) == false ) { Same – just use !NativeRuntime.supportCompressionCodec(...) instead of comparing {{ == false}} Also, English nit: should be named supportsCompressionCodec (note the 's') + String message = "Native output collector don't support compression codec " + codec; Nit: should be "doesn't support" instead of "don't support" + * + * @param codecName + * @ return Remove these empty JavaDoc lines, since the main description of the function is sufficient to understand it.
            Hide
            Permalink
            mauzhang Manu Zhang added a comment -

            thanks, Todd. I've updated the patch accordingly and also fixed "don't support" in other places.

            Show
            mauzhang Manu Zhang added a comment - thanks, Todd. I've updated the patch accordingly and also fixed "don't support" in other places.
            Hide
            Permalink
            tlipcon Todd Lipcon added a comment -

            +1, looks good to me. I'll commit this momentarily

            Show
            tlipcon Todd Lipcon added a comment - +1, looks good to me. I'll commit this momentarily
            Hide
            Permalink
            tlipcon Todd Lipcon added a comment -

            Actually it looks like this doesn't apply against tip of the branch. Is this patch based on another in-flight patch, Manu?

            Show
            tlipcon Todd Lipcon added a comment - Actually it looks like this doesn't apply against tip of the branch. Is this patch based on another in-flight patch, Manu?
            Hide
            Permalink
            mauzhang Manu Zhang added a comment -

            sorry, my local branch was not updated. Recreated a patch against latest branch.

            Show
            mauzhang Manu Zhang added a comment - sorry, my local branch was not updated. Recreated a patch against latest branch.
            Hide
            Permalink
            clockfly Sean Zhong added a comment -

            Committed to branch at r1616115. Thanks!

            Show
            clockfly Sean Zhong added a comment - Committed to branch at r1616115. Thanks!
            Hide
            Permalink
            clockfly Sean Zhong added a comment -

            commited to branch MR-2841 at r1616115

            Show
            clockfly Sean Zhong added a comment - commited to branch MR-2841 at r1616115
            Hide
            Permalink
            hudson Hudson added a comment -

            SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/)
            MAPREDUCE-5976. native-task: should not fail to build if snappy is missing (Manu Zhang) (seanzhong: rev 432f641bc2f6c7fdb67da578233cbe07fdfbf1ea)

            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestReadBuffer.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Compressions.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/org_apache_hadoop_mapred_nativetask_NativeRuntime.h
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestCompressions.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/snappy-c.h
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/NativeRuntimeJniImpl.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/snappy.h
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeRuntime.java
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/SnappyCodec.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/util/SnappyUtil.java
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt
            • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
            Show
            hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/ ) MAPREDUCE-5976 . native-task: should not fail to build if snappy is missing (Manu Zhang) (seanzhong: rev 432f641bc2f6c7fdb67da578233cbe07fdfbf1ea) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestReadBuffer.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Compressions.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/org_apache_hadoop_mapred_nativetask_NativeRuntime.h hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestCompressions.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/snappy-c.h hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/NativeRuntimeJniImpl.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/snappy.h hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeRuntime.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/SnappyCodec.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/util/SnappyUtil.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt
            Hide
            Permalink
            hudson Hudson added a comment -

            FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/)
            MAPREDUCE-5976. native-task: should not fail to build if snappy is missing (Manu Zhang) (seanzhong: rev 432f641bc2f6c7fdb67da578233cbe07fdfbf1ea)

            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/snappy.h
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Compressions.cc
            • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/util/SnappyUtil.java
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/org_apache_hadoop_mapred_nativetask_NativeRuntime.h
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/NativeRuntimeJniImpl.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestCompressions.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/snappy-c.h
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeRuntime.java
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/SnappyCodec.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestReadBuffer.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc
            Show
            hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/ ) MAPREDUCE-5976 . native-task: should not fail to build if snappy is missing (Manu Zhang) (seanzhong: rev 432f641bc2f6c7fdb67da578233cbe07fdfbf1ea) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/snappy.h hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Compressions.cc hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/util/SnappyUtil.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/org_apache_hadoop_mapred_nativetask_NativeRuntime.h hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/NativeRuntimeJniImpl.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestCompressions.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/snappy-c.h hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeRuntime.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/SnappyCodec.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestReadBuffer.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc
            Hide
            Permalink
            hudson Hudson added a comment -

            SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/)
            MAPREDUCE-5976. native-task: should not fail to build if snappy is missing (Manu Zhang) (seanzhong: rev 432f641bc2f6c7fdb67da578233cbe07fdfbf1ea)

            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/NativeRuntimeJniImpl.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Compressions.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/snappy-c.h
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestCompressions.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeRuntime.java
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/org_apache_hadoop_mapred_nativetask_NativeRuntime.h
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/snappy.h
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/util/SnappyUtil.java
            • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/SnappyCodec.cc
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestReadBuffer.cc
            Show
            hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/ ) MAPREDUCE-5976 . native-task: should not fail to build if snappy is missing (Manu Zhang) (seanzhong: rev 432f641bc2f6c7fdb67da578233cbe07fdfbf1ea) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/NativeRuntimeJniImpl.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Compressions.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/snappy-c.h hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestCompressions.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeRuntime.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/org_apache_hadoop_mapred_nativetask_NativeRuntime.h hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/snappy.h hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/util/SnappyUtil.java hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/SnappyCodec.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestReadBuffer.cc

              People

              • Assignee:
                clockfly Sean Zhong
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development