Details
-
Type:
Sub-task
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: task
-
Labels:None
Description
The MR-2841 branch fails the following unit tests on my box:
CompressTest.testBzip2Compress:84 file compare result: if they are the same ,then return true expected:<true> but was:<false>
CompressTest.testDefaultCompress:116 file compare result: if they are the same ,then return true expected:<true> but was:<false>
We need to fix these before merging.
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Hi Todd,
This is expected. We currently not support BZip2Codec and DefaultCodec. These two UT test are placeholder to track which codec we don't support yet.
Maybe, we can change the expected result to false, and add a warning like this "Support for BZip2Codec is not implemented yet, please switch the flag to true once this codec implemented"
Make sense?
Got it. We typically don't allow failing tests in Hadoop - though they act as a placeholder, they will fail for anyone running the build.
Let's remove them for now and file a lower priority JIRA to later implement bzip support if we need it. However, I've never seen anyone use bzip for intermediate compression - it's just too CPU intensive to be useful. I would prefer to keep the code small and only support a small number of intermediate codecs (eg just lz4 and gzip) since I think that covers 99% of the real world use cases.
Hi Todd, I've attached a patch that simply removes those two tests and their configuration files.
Ok, patch looks good, +1
Sean Zhong – since you're a branch committer, feel free to commit Manu's patch. Feel free to ping me if you run into any issues
commited to branch MR-2841 at r1616116. Thanks, Manu.
SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/)
MAPREDUCE-5978. native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a)
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml
- hadoop-mapreduce-project/CHANGES.
MAPREDUCE-2841.txt
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/)
MAPREDUCE-5978. native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a)
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
- hadoop-mapreduce-project/CHANGES.
MAPREDUCE-2841.txt
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/)
MAPREDUCE-5978. native-task: remove test case for not supported codec Bzip2Codec and DefaultCodec (Manu Zhang) (seanzhong: rev 4d4fb1723fd825df4ad2488e4ecde0c69359e83a)
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-bzip2-compress-conf.xml
- hadoop-mapreduce-project/CHANGES.
MAPREDUCE-2841.txt - hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/test-default-compress-conf.xml
I see the following in the logs:
load nativetask lib failed, Native-Task Delegation is disabled
org.apache.hadoop.mapred.InvalidJobConfException: Native output collector don't support compression codec org.apache.hadoop.io.compress.BZip2Codec, We support Gzip, Lz4, snappy
at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:94)
at org.apache.hadoop.mapred.nativetask.testutil.EnforceNativeOutputCollectorDelegator.init(EnforceNativeOutputCollectorDelegator.java:31)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:392)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:683)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
create file ./compress/input.txt
org.apache.hadoop.io.Text org.apache.hadoop.io.Text
create file ./compress/input.txt
org.apache.hadoop.io.Text org.apache.hadoop.io.Text
load nativetask lib failed, Native-Task Delegation is disabled
org.apache.hadoop.mapred.InvalidJobConfException: Native output collector don't support compression codec org.apache.hadoop.io.compress.DefaultCodec, We support Gzip, Lz4, snappy
at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:94)
at org.apache.hadoop.mapred.nativetask.testutil.EnforceNativeOutputCollectorDelegator.init(EnforceNativeOutputCollectorDelegator.java:31)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:392)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:683)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)