Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31895

df.explain("mode") should work in PySpark side as well for consistency

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • PySpark
    • None

    Description

      Scala:

      scala> spark.range(10).explain("cost")
      == Optimized Logical Plan ==
      Range (0, 10, step=1, splits=Some(12)), Statistics(sizeInBytes=80.0 B)
      
      == Physical Plan ==
      *(1) Range (0, 10, step=1, splits=12)
      

      PySpark:

      >>> spark.range(10).explain("cost")
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "/.../spark/python/pyspark/sql/dataframe.py", line 333, in explain
          raise TypeError(err_msg)
      TypeError: extended (optional) should be provided as bool, got <class 'str'>
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gurwls223 Hyukjin Kwon
            gurwls223 Hyukjin Kwon
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment