[SPARK-42556] Dataset.colregex should link a plan_id when it only matches a single column.

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect
• Epic Link: Spark Connect Scala Client - Basic
• Assignee: Unassigned
• Reporter: Herman van Hövell
• Created: 24/Feb/23 18:20
• Updated: 04/Mar/23 05:46
• Resolved: 04/Mar/23 05:46

Description

When colregex returns a single column it should link the plans plan_id. For reference here is the non-connect Dataset code that does this:

https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L1512

This also needs to be fixed for the Python client.

Issue Links

• links to: [Github] Pull Request #40265 (beliefer)