[SPARK-47845] Support column type in split function in scala and python

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.1
• Fix Version/s: 4.0.0
• Component/s: Connect, Spark Core
• Labels: pull-request-available

Language

Python Scala

• Assignee: Liu Cao
• Reporter: Liu Cao
• Created: 14/Apr/24 23:42
• Updated: 22/Apr/24 01:43
• Resolved: 22/Apr/24 01:42

Description

I have a use case to split a String typed column with different delimiters defined in other columns of the dataframe. SQL already supports this, but scala / python functions currently don't.

 

A hypothetical example to illustrate:

import org.apache.spark.sql.functions.{col, split}

val example = spark.createDataFrame(
    Seq(
      ("Doe, John", ", ", 2),
      ("Smith,Jane", ",", 2),
      ("Johnson", ",", 1)
    )
  )
  .toDF("name", "delim", "expected_parts_count")

example.createOrReplaceTempView("test_data")

// works for SQL
spark.sql("SELECT split(name, delim, expected_parts_count) AS name_parts FROM test_data").show()

// currently doesn't compile for scala, but easy to support
example.withColumn("name_parts", split(col("name"), col("delim"), col("expected_parts_count"))).show() 

 

Pretty simple patch that I can make a PR soon

Issue Links

• links to: GitHub Pull Request #46045