[SPARK-34319] Self-join after cogroup applyInPandas fails due to unresolved conflicting attributes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.0.1, 3.1.0, 3.2.0
• Fix Version/s: 3.0.2, 3.1.1
• Component/s: SQL
• Assignee: wuyi
• Reporter: wuyi
• Created: 02/Feb/21 02:53
• Updated: 12/Dec/22 18:10
• Resolved: 02/Feb/21 07:27

Description

 

df = spark.createDataFrame([(1, 1)], ("column", "value"))row = df.groupby("ColUmn").cogroup(
    df.groupby("COLUMN")
).applyInPandas(lambda r, l: r + l, "column long, value long")
row.join(row).show()

Conflicting attributes: column#163321L,value#163322L
;;
’Join Inner
:- FlatMapCoGroupsInPandas [ColUmn#163312L], [COLUMN#163312L], <lambda>(column#163312L, value#163313L, column#163312L, value#163313L), [column#163321L, value#163322L]
:  :- Project [ColUmn#163312L, column#163312L, value#163313L]
:  :  +- LogicalRDD [column#163312L, value#163313L], false
:  +- Project [COLUMN#163312L, column#163312L, value#163313L]
:     +- LogicalRDD [column#163312L, value#163313L], false
+- FlatMapCoGroupsInPandas [ColUmn#163312L], [COLUMN#163312L], <lambda>(column#163312L, value#163313L, column#163312L, value#163313L), [column#163321L, value#163322L]
   :- Project [ColUmn#163312L, column#163312L, value#163313L]
   :  +- LogicalRDD [column#163312L, value#163313L], false
   +- Project [COLUMN#163312L, column#163312L, value#163313L]
      +- LogicalRDD [column#163312L, value#163313L], false

 

 

Issue Links

• is related to: SPARK-27463 Support Dataframe Cogroup via Pandas UDFs
• links to: [Github] Pull Request #31429 (Ngone51)
• links to: [Github] Pull Request #31429 (Ngone51)