[SPARK-39107] Silent change in regexp_replace's handling of empty strings

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.2
• Fix Version/s: 3.3.0, 3.1.4, 3.2.2
• Component/s: SQL
• Labels: correctness release-notes
• Assignee: Lorenzo Martini
• Reporter: Willi Raschkowski
• Created: 05/May/22 15:24
• Updated: 03/Jan/23 15:21
• Resolved: 10/May/22 00:46

Description

Hi, we just upgraded from 3.0.2 to 3.1.2 and noticed a silent behavior change that a) seems incorrect, and b) is undocumented in the migration guide:

3.0.2

scala> val df = spark.sql("SELECT '' AS col")
df: org.apache.spark.sql.DataFrame = [col: string]

scala> df.withColumn("replaced", regexp_replace(col("col"), "^$", "<empty>")).show
+---+--------+
|col|replaced|
+---+--------+
|   | <empty>|
+---+--------+

3.1.2

scala> val df = spark.sql("SELECT '' AS col")
df: org.apache.spark.sql.DataFrame = [col: string]

scala> df.withColumn("replaced", regexp_replace(col("col"), "^$", "<empty>")).show
+---+--------+
|col|replaced|
+---+--------+
|   |        |
+---+--------+

Note, the regular expression ^$ should match the empty string, but doesn't in version 3.1. E.g. this is the Java behavior:

scala> "".replaceAll("^$", "<empty>");
res1: String = <empty>

Issue Links

• links to: [Github] Pull Request #36457 (LorenzoMartini)
• links to: [Github] Pull Request #36457 (LorenzoMartini)