[SPARK-41824] Implement DataFrame.explain format to be similar to PySpark

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect
• Assignee: Jiaan Geng
• Reporter: Sandeep Singh
• Created: 02/Jan/23 17:08
• Updated: 07/Jan/23 04:52
• Resolved: 07/Jan/23 04:52

Description

df = spark.createDataFrame([(14, "Tom"), (23, "Alice"), (16, "Bob")], ["age", "name"]) 
df.explain()
df.explain(True)
df.explain(mode="formatted")
df.explain("cost")

File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1296, in pyspark.sql.connect.dataframe.DataFrame.explain
Failed example:
    df.explain()
Expected:
    == Physical Plan ==
    *(1) Scan ExistingRDD[age...,name...]
Got:
    == Physical Plan ==
    LocalTableScan [age#1148L, name#1149]
    <BLANKLINE>
    <BLANKLINE>
**********************************************************************
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1314, in pyspark.sql.connect.dataframe.DataFrame.explain
Failed example:
    df.explain(mode="formatted")
Expected:
    == Physical Plan ==
    * Scan ExistingRDD (...)
    (1) Scan ExistingRDD [codegen id : ...]
    Output [2]: [age..., name...]
    ...
Got:
    == Physical Plan ==
    LocalTableScan (1)
    <BLANKLINE>
    <BLANKLINE>
    (1) LocalTableScan
    Output [2]: [age#1170L, name#1171]
    Arguments: [age#1170L, name#1171]
    <BLANKLINE>
    <BLANKLINE>

Issue Links

• is a clone of: SPARK-41823 DataFrame.join creating ambiguous column names
• links to: [Github] Pull Request #39436 (beliefer)