Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: task
    • Labels:
      None

      Description

      The MR-2841 branch fails the following unit tests on my box:
      CompressTest.testBzip2Compress:84 file compare result: if they are the same ,then return true expected:<true> but was:<false>
      CompressTest.testDefaultCompress:116 file compare result: if they are the same ,then return true expected:<true> but was:<false>

      We need to fix these before merging.

        Activity

        Hide
        Todd Lipcon added a comment -

        I see the following in the logs:

        load nativetask lib failed, Native-Task Delegation is disabled
        org.apache.hadoop.mapred.InvalidJobConfException: Native output collector don't support compression codec org.apache.hadoop.io.compress.BZip2Codec, We support Gzip, Lz4, snappy
        at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:94)
        at org.apache.hadoop.mapred.nativetask.testutil.EnforceNativeOutputCollectorDelegator.init(EnforceNativeOutputCollectorDelegator.java:31)
        at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:392)
        at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:683)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
        create file ./compress/input.txt
        org.apache.hadoop.io.Text org.apache.hadoop.io.Text
        create file ./compress/input.txt
        org.apache.hadoop.io.Text org.apache.hadoop.io.Text
        load nativetask lib failed, Native-Task Delegation is disabled
        org.apache.hadoop.mapred.InvalidJobConfException: Native output collector don't support compression codec org.apache.hadoop.io.compress.DefaultCodec, We support Gzip, Lz4, snappy
        at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:94)
        at org.apache.hadoop.mapred.nativetask.testutil.EnforceNativeOutputCollectorDelegator.init(EnforceNativeOutputCollectorDelegator.java:31)
        at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:392)
        at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:683)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)

        Show
        Todd Lipcon added a comment - I see the following in the logs: load nativetask lib failed, Native-Task Delegation is disabled org.apache.hadoop.mapred.InvalidJobConfException: Native output collector don't support compression codec org.apache.hadoop.io.compress.BZip2Codec, We support Gzip, Lz4, snappy at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:94) at org.apache.hadoop.mapred.nativetask.testutil.EnforceNativeOutputCollectorDelegator.init(EnforceNativeOutputCollectorDelegator.java:31) at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:392) at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:683) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) create file ./compress/input.txt org.apache.hadoop.io.Text org.apache.hadoop.io.Text create file ./compress/input.txt org.apache.hadoop.io.Text org.apache.hadoop.io.Text load nativetask lib failed, Native-Task Delegation is disabled org.apache.hadoop.mapred.InvalidJobConfException: Native output collector don't support compression codec org.apache.hadoop.io.compress.DefaultCodec, We support Gzip, Lz4, snappy at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:94) at org.apache.hadoop.mapred.nativetask.testutil.EnforceNativeOutputCollectorDelegator.init(EnforceNativeOutputCollectorDelegator.java:31) at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:392) at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:683) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724)
        Hide
        Sean Zhong added a comment -

        Hi Todd,

        This is expected. We currently not support BZip2Codec and DefaultCodec. These two UT test are placeholder to track which codec we don't support yet.
        Maybe, we can change the expected result to false, and add a warning like this "Support for BZip2Codec is not implemented yet, please switch the flag to true once this codec implemented"

        Make sense?

        Show
        Sean Zhong added a comment - Hi Todd, This is expected. We currently not support BZip2Codec and DefaultCodec. These two UT test are placeholder to track which codec we don't support yet. Maybe, we can change the expected result to false, and add a warning like this "Support for BZip2Codec is not implemented yet, please switch the flag to true once this codec implemented" Make sense?
        Hide
        Todd Lipcon added a comment -

        Got it. We typically don't allow failing tests in Hadoop - though they act as a placeholder, they will fail for anyone running the build.

        Let's remove them for now and file a lower priority JIRA to later implement bzip support if we need it. However, I've never seen anyone use bzip for intermediate compression - it's just too CPU intensive to be useful. I would prefer to keep the code small and only support a small number of intermediate codecs (eg just lz4 and gzip) since I think that covers 99% of the real world use cases.

        Show
        Todd Lipcon added a comment - Got it. We typically don't allow failing tests in Hadoop - though they act as a placeholder, they will fail for anyone running the build. Let's remove them for now and file a lower priority JIRA to later implement bzip support if we need it. However, I've never seen anyone use bzip for intermediate compression - it's just too CPU intensive to be useful. I would prefer to keep the code small and only support a small number of intermediate codecs (eg just lz4 and gzip) since I think that covers 99% of the real world use cases.
        Hide
        Manu Zhang added a comment -

        Hi Todd, I've attached a patch that simply removes those two tests and their configuration files.

        Show
        Manu Zhang added a comment - Hi Todd, I've attached a patch that simply removes those two tests and their configuration files.
        Hide
        Sean Zhong added a comment -

        Ok, patch looks good, +1

        Show
        Sean Zhong added a comment - Ok, patch looks good, +1
        Hide
        Todd Lipcon added a comment -

        Sean Zhong – since you're a branch committer, feel free to commit Manu's patch. Feel free to ping me if you run into any issues

        Show
        Todd Lipcon added a comment - Sean Zhong – since you're a branch committer, feel free to commit Manu's patch. Feel free to ping me if you run into any issues
        Hide
        Sean Zhong added a comment -

        commited to branch MR-2841 at r1616116. Thanks, Manu.

        Show
        Sean Zhong added a comment - commited to branch MR-2841 at r1616116. Thanks, Manu.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/)
        MAPREDUCE-5978. native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml
        • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/ ) MAPREDUCE-5978 . native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/)
        MAPREDUCE-5978. native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
        • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/ ) MAPREDUCE-5978 . native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/)
        MAPREDUCE-5978. native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml
        • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/ ) MAPREDUCE-5978 . native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml

          People

          • Assignee:
            Manu Zhang
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development