[SPARK-1861] ArrayIndexOutOfBoundsException when reading bzip2 files

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 0.9.0, 1.0.0
• Component/s: Spark Core
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Created: 16/May/14 02:25
• Updated: 15/Aug/14 11:37
• Resolved: 20/May/14 03:54

Description

Hadoop uses CBZip2InputStream to decode bzip2 files. However, the implementation is not threadsafe and Spark may run multiple tasks in the same JVM, which leads to this error. This is not a problem for Hadoop MapReduce because Hadoop runs each task in a separate JVM.

A workaround is to set `SPARK_WORKER_CORES=1` in spark-env.sh for a standalone cluster.

Issue Links

• requires: HADOOP-10614 CBZip2InputStream is not threadsafe