Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3104

Tez fails on Bzip2 intermediate output format on hadoop 2.7.1 and earlier

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.1, 0.8.3
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      HADOOP_CLASSPATH="$TEZ_CONF_DIR:$TEZ_HOME/:$TEZ_HOME/lib/" yarn jar /home/gs/tez/current/tez-tests-*.jar mrrsleep -Dmapreduce.reduce.log.level=TRACE -Dtez.task.log.level=TRACE -Dtez.runtime.compress=true -Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.BZip2Codec -Dmapred.map.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec -Dmapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec -Dmapreduce.reduce.shuffle.parallelcopies=30 -m 100 -ir 10 -r 100

      2016-02-09 02:31:36,605 [ERROR] [ShuffleAndMergeRunner {map}] |orderedgrouped.Shuffle|: map: ShuffleRunner failed with error
      org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in fetcher {map} #16
      	at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:360)
      	at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:337)
      	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.UnsupportedOperationException
      	at org.apache.hadoop.io.compress.bzip2.BZip2DummyDecompressor.decompress(BZip2DummyDecompressor.java:32)
      	at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
      	at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
      	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
      	at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readToMemory(IFile.java:626)
      	at org.apache.tez.runtime.library.common.shuffle.ShuffleUtils.shuffleToMemory(ShuffleUtils.java:113)
      	at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyMapOutput(FetcherOrderedGrouped.java:502)
      	at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:279)
      	at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.fetchNext(FetcherOrderedGrouped.java:169)
      	at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.run(FetcherOrderedGrouped.java:184)
      

        Attachments

        1. TEZ-3104.1.patch
          0.9 kB
          Jonathan Turner Eagles

          Issue Links

            Activity

              People

              • Assignee:
                jeagles Jonathan Turner Eagles
                Reporter:
                jeagles Jonathan Turner Eagles
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: