[SPARK-24190] lineSep shouldn't be required in JSON write

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 05/May/18 16:52
• Updated: 24/Jun/18 00:41
• Resolved: 24/Jun/18 00:41

Description

Currently, the lineSep option is required by JSON datasource in write if encoding is different from UTF-8. For example, the code:

df.write.option("encoding", "UTF-32BE").json(file)

throws the exception:

requirement failed: The lineSep option must be specified for the UTF-32BE encoding
java.lang.IllegalArgumentException: requirement failed: The lineSep option must be specified for the UTF-32BE encoding
	at scala.Predef$.require(Predef.scala:224)
	at org.apache.spark.sql.catalyst.json.JSONOptions$$anonfun$32.apply(JSONOptions.scala:118)
	at org.apache.spark.sql.catalyst.json.JSONOptions$$anonfun$32.apply(JSONOptions.scala:103)
	at scala.Option.map(Option.scala:146)

The restriction should NOT be applied to writing.

Issue Links

• links to: [Github] Pull Request #21247 (MaxGekk)