[SPARK-6459] Warn when Column API is constructing trivially true equality

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.3.1, 1.4.0
• Component/s: SQL
• Target Version/s: 1.3.1
• Assignee: Michael Armbrust
• Reporter: Michael Armbrust
• Created: 22/Mar/15 19:34
• Updated: 17/Aug/18 18:08
• Resolved: 24/Mar/15 19:10

Description

Right now its pretty confusing when a user constructs and equality predicate that is going to be use in a self join, where the optimizer cannot distinguish between the attributes in question (e.g., SPARK-6231). Since there is really no good reason to do this, lets print a warning.

Issue Links

• is related to: SPARK-25150 Joining DataFrames derived from the same source yields confusing/incorrect results
• relates to: SPARK-20073 Unexpected Cartesian product when using eqNullSafe in join with a derived table
• links to: [Github] Pull Request #5163 (marmbrus)