[HIVE-7431] When run on spark cluster, some spark tasks may fail

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Assignee: Rui Li
• Reporter: Rui Li
• Created: 17/Jul/14 02:25
• Updated: 29/May/15 02:31
• Resolved: 21/Jul/14 05:22

Description

When running queries on spark, some spark tasks fail (usually the first couple of tasks) with the following stack trace:

org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:154)
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunction.call(HiveMapFunction.java:60)
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunction.call(HiveMapFunction.java:35)
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:161)
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:161)
org.apache.spark.rdd.RDD$$anonfun$12.apply(RDD.scala:559)
org.apache.spark.rdd.RDD$$anonfun$12.apply(RDD.scala:559)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158)
...

Observed for spark standalone cluster. Not verified for spark on yarn or mesos.
NO PRECOMMIT TESTS. This is for spark branch only.

Issue Links

• is part of: HIVE-7292 Hive on Spark
• is related to: HIVE-9127 Improve CombineHiveInputFormat.getSplit performance