Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40117

Convert condition to java in DataFrameWriterV2.overwrite

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.3, 3.3.0, 3.2.2
    • 3.1.4, 3.3.1, 3.2.3, 3.4.0
    • PySpark, SQL
    • None

    Description

      DataFrameWriterV2.overwrite() fails to convert the condition parameter to java. This prevents the function from being called.

      It is caused by the following commit that deleted the `_to_java_column` call instead of fixing it: https://github.com/apache/spark/commit/a1e459ed9f6777fb8d5a2d09fda666402f9230b9

      Attachments

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            wenlilooi Wenli Looi
            wenlilooi Wenli Looi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment