Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28153

Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.3, 2.4.3, 3.0.0
    • 2.4.4, 3.0.0
    • PySpark
    • None

    Description

      from pyspark.sql.functions import udf, input_file_name
      spark.range(10).write.mode("overwrite").parquet("/tmp/foo")
      spark.read.parquet("/tmp/foo").select(udf(lambda x: x, "long")("id"), input_file_name()).show()
      
      +------------+-----------------+
      |<lambda>(id)|input_file_name()|
      +------------+-----------------+
      |           8|                 |
      |           5|                 |
      |           0|                 |
      |           9|                 |
      |           6|                 |
      |           2|                 |
      |           3|                 |
      |           4|                 |
      |           7|                 |
      |           1|                 |
      +------------+-----------------+
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            hyukjin.kwon Hyukjin Kwon
            hyukjin.kwon Hyukjin Kwon
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment