[SPARK-6067] Spark sql hive dynamic partitions job will fail if task fails

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.2.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Jason Hubbard
• Created: 27/Feb/15 21:12
• Updated: 19/Sep/15 13:58
• Resolved: 19/Sep/15 13:58

Description

When inserting into a hive table from spark sql while using dynamic partitioning, if a task fails it will cause the task to continue to fail and eventually fail the job.

/mytable/.hive-staging_hive_2015-02-27_11-53-19_573_222-3/-ext-10000/partition=2015-02-04/part-00001 for client <ip> already exists

The task may need to clean up after a failed task to write to the location of the previously failed task.

Issue Links

• duplicates: SPARK-8379 LeaseExpiredException when using dynamic partition with speculative execution
• is related to: SPARK-6369 InsertIntoHiveTable and Parquet Relation should use logic from SparkHadoopWriter
• is related to: SPARK-3007 Add "Dynamic Partition" support to Spark Sql hive
• links to: [Github] Pull Request #4980 (baishuo)