[SPARK-38955] Disable lineSep option in 'from_csv' and 'schema_of_csv'

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0, 3.2.2
• Component/s: SQL
• Labels: correctness
• Assignee: Hyukjin Kwon
• Reporter: Robert Joseph Evans
• Created: 19/Apr/22 21:55
• Updated: 12/Dec/22 18:10
• Resolved: 22/Apr/22 02:44

Description

I don't know how critical this is. I was doing some general testing to understand from_csv and found that if I happen to have a lineSep in the input data and I noticed that the next row appears to be corrupted. multiLine does not appear to fix it. Because this is data corruption I am inclined to mark this as CRITICAL or BLOCKER, but it is an odd corner case so I m not going to set it myself.

Seq[String]("1,\n2,3,4,5","6,7,8,9,10", "11,12,13,14,15", null).toDF.select(col("value"), from_csv(col("value"), StructType(Seq(StructField("a", LongType), StructField("b", StringType))), Map[String,String]())).show()
+--------------+---------------+
|         value|from_csv(value)|
+--------------+---------------+
|   1,\n2,3,4,5|      {1, null}|
|    6,7,8,9,10|      {null, 8}|
|11,12,13,14,15|       {11, 12}|
|          null|           null|
+--------------+---------------+

Seq[String]("1,:2,3,4,5","6,7,8,9,10", "11,12,13,14,15", null).toDF.select(col("value"), from_csv(col("value"), StructType(Seq(StructField("a", LongType), StructField("b", StringType))), Map[String,String]("lineSep" -> ":"))).show()
+--------------+---------------+
|         value|from_csv(value)|
+--------------+---------------+
|    1,:2,3,4,5|      {1, null}|
|    6,7,8,9,10|      {null, 8}|
|11,12,13,14,15|       {11, 12}|
|          null|           null|
+--------------+---------------+

Seq[String]("1,\n2,3,4,5","6,7,8,9,10", "11,12,13,14,15", null).toDF.select(col("value"), from_csv(col("value"), StructType(Seq(StructField("a", LongType), StructField("b", StringType))), Map[String,String]("lineSep" -> ":"))).show()
+--------------+---------------+
|         value|from_csv(value)|
+--------------+---------------+
|   1,\n2,3,4,5|       {1, \n2}|
|    6,7,8,9,10|         {6, 7}|
|11,12,13,14,15|       {11, 12}|
|          null|           null|
+--------------+---------------+

Issue Links

• links to: [Github] Pull Request #36294 (HyukjinKwon)