[SPARK-42691] Implement Dataset.semanticHash

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.1
• Component/s: Connect
• Epic Link: Spark Connect Scala Client - Basic
• Assignee: Unassigned
• Reporter: Herman van Hövell
• Created: 06/Mar/23 19:33
• Updated: 17/Mar/23 18:00
• Resolved: 11/Mar/23 09:04

Description

Implement Dataset.semanticHash:

/**
* Returns a `hashCode` of the logical query plan against this [[Dataset]].
*
* @note Unlike the standard `hashCode`, the hash is calculated against the query plan
* simplified by tolerating the cosmetic differences such as attribute names.
* @since 3.4.0
*/
@DeveloperApi
def semanticHash(): Int

This has to be computed on the spark connect server to do this. Please extend the 

AnalyzePlanRequest and AnalyzePlanResponse messages for this.

Also make sure this works in PySpark.

Issue Links

• fixes: SPARK-41922 Implement DataFrame `semanticHash`
• links to: [Github] Pull Request #40366 (beliefer)