Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.1, 2.2.0
    • Fix Version/s: 1.3.0, 2.5.0
    • Component/s: None
    • Labels:
      None

      Description

      Hadoop uses CBZip2InputStream to decode bzip2 files. However, the implementation is not threadsafe. This is not a really problem for Hadoop MapReduce because Hadoop runs each task in a separate JVM. But for other libraries that utilize multithreading and use Hadoop's InputFormat, e.g., Spark, it will cause exceptions like the following:

      java.lang.ArrayIndexOutOfBoundsException: 6 org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.recvDecodingTables(CBZip2InputStream.java:729) org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.getAndMoveToFrontDecode(CBZip2InputStream.java:795) org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock(CBZip2InputStream.java:499) org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.changeStateToProcessABlock(CBZip2InputStream.java:330) org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.read(CBZip2InputStream.java:394) org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionInputStream.read(BZip2Codec.java:428) java.io.InputStream.read(InputStream.java:101) org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205) org.apache.hadoop.util.LineReader.readLine(LineReader.java:169) org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:176) org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:43) org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:198) org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:181) org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:35) scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1000) org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847) org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847) org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077) org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077) org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111) org.apache.spark.scheduler.Task.run(Task.scala:51) org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:724)
      

        Attachments

        1. bzip2-2.diff
          2 kB
          Xiangrui Meng
        2. bzip2.diff
          2 kB
          Xiangrui Meng

          Issue Links

            Activity

              People

              • Assignee:
                mengxr Xiangrui Meng
                Reporter:
                mengxr Xiangrui Meng
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: