Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2841 Task level native optimization
  3. MAPREDUCE-5978

native-task CompressTest failure on Ubuntu

        Details

        • Type: Sub-task Sub-task
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: None
        • Component/s: task
        • Labels:
          None

          Description

          The MR-2841 branch fails the following unit tests on my box:
          CompressTest.testBzip2Compress:84 file compare result: if they are the same ,then return true expected:<true> but was:<false>
          CompressTest.testDefaultCompress:116 file compare result: if they are the same ,then return true expected:<true> but was:<false>

          We need to fix these before merging.

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            Todd Lipcon added a comment -

            I see the following in the logs:

            load nativetask lib failed, Native-Task Delegation is disabled
            org.apache.hadoop.mapred.InvalidJobConfException: Native output collector don't support compression codec org.apache.hadoop.io.compress.BZip2Codec, We support Gzip, Lz4, snappy
            at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:94)
            at org.apache.hadoop.mapred.nativetask.testutil.EnforceNativeOutputCollectorDelegator.init(EnforceNativeOutputCollectorDelegator.java:31)
            at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:392)
            at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
            at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:683)
            at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755)
            at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
            at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:724)
            create file ./compress/input.txt
            org.apache.hadoop.io.Text org.apache.hadoop.io.Text
            create file ./compress/input.txt
            org.apache.hadoop.io.Text org.apache.hadoop.io.Text
            load nativetask lib failed, Native-Task Delegation is disabled
            org.apache.hadoop.mapred.InvalidJobConfException: Native output collector don't support compression codec org.apache.hadoop.io.compress.DefaultCodec, We support Gzip, Lz4, snappy
            at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:94)
            at org.apache.hadoop.mapred.nativetask.testutil.EnforceNativeOutputCollectorDelegator.init(EnforceNativeOutputCollectorDelegator.java:31)
            at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:392)
            at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
            at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:683)
            at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755)
            at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
            at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:724)

            Show
            Todd Lipcon added a comment - I see the following in the logs: load nativetask lib failed, Native-Task Delegation is disabled org.apache.hadoop.mapred.InvalidJobConfException: Native output collector don't support compression codec org.apache.hadoop.io.compress.BZip2Codec, We support Gzip, Lz4, snappy at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:94) at org.apache.hadoop.mapred.nativetask.testutil.EnforceNativeOutputCollectorDelegator.init(EnforceNativeOutputCollectorDelegator.java:31) at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:392) at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:683) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) create file ./compress/input.txt org.apache.hadoop.io.Text org.apache.hadoop.io.Text create file ./compress/input.txt org.apache.hadoop.io.Text org.apache.hadoop.io.Text load nativetask lib failed, Native-Task Delegation is disabled org.apache.hadoop.mapred.InvalidJobConfException: Native output collector don't support compression codec org.apache.hadoop.io.compress.DefaultCodec, We support Gzip, Lz4, snappy at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:94) at org.apache.hadoop.mapred.nativetask.testutil.EnforceNativeOutputCollectorDelegator.init(EnforceNativeOutputCollectorDelegator.java:31) at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:392) at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:683) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724)
            Hide
            Permalink
            Sean Zhong added a comment -

            Hi Todd,

            This is expected. We currently not support BZip2Codec and DefaultCodec. These two UT test are placeholder to track which codec we don't support yet.
            Maybe, we can change the expected result to false, and add a warning like this "Support for BZip2Codec is not implemented yet, please switch the flag to true once this codec implemented"

            Make sense?

            Show
            Sean Zhong added a comment - Hi Todd, This is expected. We currently not support BZip2Codec and DefaultCodec. These two UT test are placeholder to track which codec we don't support yet. Maybe, we can change the expected result to false, and add a warning like this "Support for BZip2Codec is not implemented yet, please switch the flag to true once this codec implemented" Make sense?
            Hide
            Permalink
            Todd Lipcon added a comment -

            Got it. We typically don't allow failing tests in Hadoop - though they act as a placeholder, they will fail for anyone running the build.

            Let's remove them for now and file a lower priority JIRA to later implement bzip support if we need it. However, I've never seen anyone use bzip for intermediate compression - it's just too CPU intensive to be useful. I would prefer to keep the code small and only support a small number of intermediate codecs (eg just lz4 and gzip) since I think that covers 99% of the real world use cases.

            Show
            Todd Lipcon added a comment - Got it. We typically don't allow failing tests in Hadoop - though they act as a placeholder, they will fail for anyone running the build. Let's remove them for now and file a lower priority JIRA to later implement bzip support if we need it. However, I've never seen anyone use bzip for intermediate compression - it's just too CPU intensive to be useful. I would prefer to keep the code small and only support a small number of intermediate codecs (eg just lz4 and gzip) since I think that covers 99% of the real world use cases.
            Hide
            Permalink
            Manu Zhang added a comment -

            Hi Todd, I've attached a patch that simply removes those two tests and their configuration files.

            Show
            Manu Zhang added a comment - Hi Todd, I've attached a patch that simply removes those two tests and their configuration files.
            Hide
            Permalink
            Sean Zhong added a comment -

            Ok, patch looks good, +1

            Show
            Sean Zhong added a comment - Ok, patch looks good, +1
            Hide
            Permalink
            Todd Lipcon added a comment -

            Sean Zhong – since you're a branch committer, feel free to commit Manu's patch. Feel free to ping me if you run into any issues

            Show
            Todd Lipcon added a comment - Sean Zhong – since you're a branch committer, feel free to commit Manu's patch. Feel free to ping me if you run into any issues
            Hide
            Permalink
            Sean Zhong added a comment -

            commited to branch MR-2841 at r1616116. Thanks, Manu.

            Show
            Sean Zhong added a comment - commited to branch MR-2841 at r1616116. Thanks, Manu.
            Hide
            Permalink
            Hudson added a comment -

            SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/)
            MAPREDUCE-5978. native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a)

            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml
            • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
            Show
            Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/ ) MAPREDUCE-5978 . native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt
            Hide
            Permalink
            Hudson added a comment -

            FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/)
            MAPREDUCE-5978. native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a)

            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
            • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
            Show
            Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/ ) MAPREDUCE-5978 . native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt
            Hide
            Permalink
            Hudson added a comment -

            SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/)
            MAPREDUCE-5978. native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a)

            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml
            • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
            • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml
            Show
            Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/ ) MAPREDUCE-5978 . native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml

              People

              • Assignee:
                Manu Zhang
                Reporter:
                Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development