[SPARK-7990] Add methods to facilitate equi-join on multiple join keys

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Sprint: Spark 1.5 doc/QA sprint
• Assignee: L. C. Hsieh
• Reporter: Reynold Xin
• Created: 01/Jun/15 04:26
• Updated: 09/Jun/15 06:27
• Resolved: 09/Jun/15 06:27

Description

We have a variant of the join function that facilitates equi-join on a single join key, but we don't have one to do it for multiple join keys.

This is the existing Python API:

def join(self, other, joinExprs=None, joinType=None):

I think we should rename joinExprs to "on", and joinType to "how" to match Pandas. And then the "on" column should support either a string, a join condition, a list of string, or a list of join condition ("and" together).

In order to support the Python API, we'd need to add a variant for Scala as well. I think we can add another join method that looks like

def join(other: DataFrame, on: Seq[String], joinType: String): DataFrame

and update the existing Scala one to call this one.

Issue Links

• relates to: SPARK-7197 Join with DataFrame Python API not working properly with more than 1 column
• links to: [Github] Pull Request #6616 (viirya)