[SPARK-48852] String trim function returning incorrect results in Spark connect

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Connect
• Labels: correctness pull-request-available
• Assignee: Biruk Tesfaye
• Reporter: Biruk Tesfaye
• Created: 10/Jul/24 08:47
• Updated: 11/Jul/24 23:56
• Resolved: 11/Jul/24 23:55

Description

With spark connect, the snippet: 

 

import org.apache.spark.sql.functions.trim 
import spark.implicits._ 

val df = Seq((" john"), ("jane "), (" ann ")).toDF("name") 
val trimmedDF = df.withColumn("name", trim($"name", " ")) 
trimmedDF.collect()

returns an incorrect result

Array[Row] = Array([], [], [])

We get the same result when using `rtrim` and `ltrim` as well.  
 

Issue Links

• links to: [Github] Pull Request #47277 (biruktesf-db)
• links to: GitHub Pull Request #47277