Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31895

df.explain("mode") should work in PySpark side as well for consistency

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: PySpark
    • Labels:
      None

      Description

      Scala:

      scala> spark.range(10).explain("cost")
      == Optimized Logical Plan ==
      Range (0, 10, step=1, splits=Some(12)), Statistics(sizeInBytes=80.0 B)
      
      == Physical Plan ==
      *(1) Range (0, 10, step=1, splits=12)
      

      PySpark:

      >>> spark.range(10).explain("cost")
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "/.../spark/python/pyspark/sql/dataframe.py", line 333, in explain
          raise TypeError(err_msg)
      TypeError: extended (optional) should be provided as bool, got <class 'str'>
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hyukjin.kwon Hyukjin Kwon
                Reporter:
                hyukjin.kwon Hyukjin Kwon
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: