[SPARK-2867] saveAsHadoopFile() in PairRDDFunction.scala should allow use other OutputCommiter class

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.0.0, 1.1.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Joseph Su
• Created: 05/Aug/14 19:27
• Updated: 06/Jan/16 09:56
• Resolved: 06/Jan/16 09:56

Description

The saveAsHadoopFile() in PairRDDFunction.scala hard-coded the OutputCommitter class as FileOutputCommitter because of the following code in the source:

hadoopConf.setOutputCommitter(classOf[FileOutputCommitter])

However, OutputCommitter is a changeable option in regular Hadoop MapReduce program. Users can specify "mapred.output.committer.class" to change the committer class used by other Hadoop programs.

The saveAsHadoopFile() function should remove this hard-coded assignment and provide a way to specify the OutputCommitte used here.