[SPARK-12146] SparkR jsonFile should support multiple input files

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0, 2.0.0
• Component/s: SparkR
• Assignee: Yanbo Liang
• Reporter: Yanbo Liang
• Created: 04/Dec/15 08:59
• Updated: 24/Dec/15 13:29
• Resolved: 11/Dec/15 19:48

Description

This bug is easy to reproduce, jsonFile did not support character vector as arguments.

> path <- c("/path/to/dir1","/path/to/dir2")
> raw.terror<-jsonFile(sqlContext,path)
15/12/03 15:59:55 ERROR RBackendHandler: jsonFile on 1 failed
Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :
  java.io.IOException: No input paths specified in job
        at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:201)
        at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
        at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:207)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
        at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
        at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2

Issue Links

• is related to: SPARK-10185 Spark SQL does not handle comma separates paths on Hadoop FileSystem
• links to: [Github] Pull Request #10145 (yanboliang)