[SPARK-17665] SparkR does not support options in other types consistently other APIs

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.0
• Component/s: SparkR
• Target Version/s: 2.1.0
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 26/Sep/16 05:34
• Updated: 12/Dec/22 18:10
• Resolved: 07/Oct/16 18:35

Description

Currently, SparkR only supports a string as option in some APIs such as `read.df`/`write.df` and etc.

It'd be great if they support other types consistently with Python/Scala/Java/SQL APIs.

• Python supports all types but converts it to string
• Scala/Java/SQL - Long/Boolean/String/Double.

Currently,

> read.df("text.json", "csv", inferSchema=FALSE)

throws an exception as below:

Error in value[[3L]](cond) :
  Error in invokeJava(isStatic = TRUE, className, methodName, ...): java.lang.ClassCastException: java.lang.Boolean cannot be cast to java.lang.String
	at org.apache.spark.sql.internal.SessionState$$anonfun$newHadoopConfWithOptions$1.apply(SessionState.scala:59)
	at org.apache.spark.sql.internal.SessionState$$anonfun$newHadoopConfWithOptions$1.apply(SessionState.scala:59)
	at scala.collection.immutable.Map$Map3.foreach(Map.scala:161)
	at org.apache.spark.sql.internal.SessionState.newHadoopConfWithOptions(SessionState.scala:59)
	at org.apache.spark.sql.execution.datasources.PartitioningAwareFileCatalog.<init>(PartitioningAwareFileCatalog.scala:45)
	at org.apache.spark.sql.execution.datasources.ListingFileCatalog.<init>(ListingFileCatalog.scala:45)
	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:401)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149)
	at org.apache.spark.sql.DataFrameReader.lo

Issue Links

• links to: [Github] Pull Request #15239 (HyukjinKwon)