[SPARK-30791] Dataframe add sameResult and sementicHash method

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: ML, SQL
• Target Version/s: 3.1.0
• Assignee: Liang Zhang
• Reporter: Weichen Xu
• Created: 11/Feb/20 14:05
• Updated: 18/Feb/20 02:47
• Resolved: 18/Feb/20 01:29

Description

Sometimes, we want to check whether two dataframes are the same.

There is already an internal API like:

df1.queryExecution.analyzed.sameResult(...)
df1.queryExecution.analyzed.semanticHash()

We can make a public API for this:

Like:

df1.sameSemantic(df2) // return true if dataframe has the same semantic
df1.semanticHash() // return a semantic hashcode

CC cloud_fan mengxr liangz

 

Issue Links

• links to: GitHub Pull Request #27565