This issue occurs when hive.merge.sparkfiles is set to true. And can be workaround by setting hive.merge.sparkfiles to false.
BTW, we did a local experiment to run the case with MR engine (set hive.merge.mapfiles=true; set hive.merge.mapredfiles=true, it can pass.
– Hive Spark Branch 70eeadd2f019dcb2e301690290c8807731eab7a1 + Hive-11473 patch (
HIVE-11473.3-spark.patch) ---> This is to support Spark 1.5 for Hive on Spark
– Spark 1.5.1
– Big-Bench Data Load (load data from HDFS to Hive warehouse, scored as ORC format). The related HiveQL: