[SPARK-9899] JSON/Parquet writing on retry or speculation broken with direct output committer

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Sprint: Spark 1.5 doc/QA sprint
• Assignee: Cheng Lian
• Reporter: Michael Armbrust
• Created: 12/Aug/15 21:03
• Updated: 14/Sep/15 18:53
• Resolved: 19/Aug/15 21:27

Description

If the first task fails all subsequent tasks will. We probably need to set a different boolean when calling create.

java.io.IOException: File already exists: ...
...
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:564)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:545)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:452)
	at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:128)
	at org.apache.spark.sql.execution.datasources.json.JsonOutputWriter.<init>(JSONRelation.scala:185)
	at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anon$1.newInstance(JSONRelation.scala:160)
	at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:217)
	at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
	at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:88)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

The reason behind this issue is that speculation shouldn't be used together with direct output committer. As there are multiple corner cases that this combination may cause data corruption and/or data loss. Please refer to this GitHub comment for more details about these corner cases.

Issue Links

• links to: [Github] Pull Request #8191 (liancheng)
• links to: [Github] Pull Request #8317 (liancheng)
• links to: [Github] Pull Request #8687 (cloud-fan)