[SPARK-19044] PySpark dropna() can fail with AnalysisException

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Component/s: PySpark, SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Josh Rosen
• Created: 31/Dec/16 23:27
• Updated: 21/May/19 04:13
• Resolved: 21/May/19 04:13

Description

In PySpark, the following fails with an AnalysisException:

v1 = spark.range(10)
v2 = v1.crossJoin(v1)
v2.dropna()

AnalysisException: u"Reference 'id' is ambiguous, could be: id#66L, id#69L.;"

However, the equivalent Scala code works fine:

val v1 = spark.range(10)
val v2 = v1.crossJoin(v1)
v1.na.drop()