[SPARK-7155] SparkContext's newAPIHadoopFile does not support comma-separated list of files, but the other API hadoopFile does.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.1
• Fix Version/s: 1.3.2, 1.4.0
• Component/s: Spark Core

Environment

Ubuntu 14.04

• Assignee: Yong Tang
• Reporter: Yong Tang
• Created: 26/Apr/15 18:02
• Updated: 18/Aug/15 12:26
• Resolved: 29/Apr/15 22:57

Description

SparkContext's newAPIHadoopFile() does not support comma-separated list of files. For example, the following:

sc.newAPIHadoopFile("/root/file1.txt,/root/file2.txt", classOf[TextInputFormat], classOf[LongWritable], classOf[Text])

will throw

org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/root/file1.txt,/root/file2.txt

However, the other API hadoopFile() is able to process comma-separated list of files correctly.

In addition, since sc.textFile() uses hadoopFile(), it is also able to process comma-separated list of files correctly.

The problem is that newAPIHadoopFile() use addInputPath() to add the file path into NewHadoopRDD. See Ln 928-931, master branch:
val job = new NewHadoopJob(conf)
NewFileInputFormat.addInputPath(job, new Path(path))
val updatedConf = job.getConfiguration
new NewHadoopRDD(this, fClass, kClass, vClass, updatedConf).setName(path)

Change addInputPath(job, new Path(path)) to addInputPaths(job, path) will resolve this issue.

Issue Links

• is duplicated by: SPARK-5544 wholeTextFiles should recognize multiple input paths delimited by ,
• links to: [Github] Pull Request #5708 (yongtang)