[SPARK-6352] Supporting non-default OutputCommitter when using saveAsParquetFile

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.1, 1.2.1, 1.3.0
• Fix Version/s: 1.4.0
• Component/s: SQL
• Target Version/s: 1.4.0
• Assignee: Pei-Lun Lee
• Reporter: Pei-Lun Lee
• Created: 16/Mar/15 09:01
• Updated: 05/Jan/18 20:29
• Resolved: 28/Apr/15 09:07

Description

SPARK-3595 only handles custom OutputCommitter for saveAsHadoopFile, it can be nice to have similar behavior in saveAsParquetFile. It maybe difficult to have a fully customizable OutputCommitter solution, at least adding something like a DirectParquetOutputCommitter and letting users choose between this and the default should be enough.

Issue Links

• is related to: SPARK-10063 Remove DirectParquetOutputCommitter
• is related to: SPARK-3595 Spark should respect configured OutputCommitter when using saveAsHadoopFile
• links to: [Github] Pull Request #5042 (ypcat)
• links to: [Github] Pull Request #5525 (ypcat)